jq -e said the health check passed. Prometheus was down.

My monitoring host runs an hourly health-check script, and its alerts go to my phone via ntfy. There was no real outage here: I found this bug during a quarterly test of the alerting chain.

What broke

Its reachability pre-check for Prometheus had this form:

curl -s "$PROM_URL/api/v1/query?query=up" | jq -e '.status=="success"'

It passed while Prometheus was refusing connections. The next check then reported "0 of 2 log shippers up". That check failed safe, but it named the wrong incident.

What I tried

I trusted jq -e's exit code as the success test for the whole pipeline.

What worked

On that host's jq, -e exits 0 on empty input. So a dead endpoint that returns nothing looks like success.

The fix is to capture the body first, test for empty separately, then hand it to jq:

BODY=$(curl -s "$PROM_URL/api/v1/query?query=up")
[ -z "$BODY" ] && { echo "prometheus unreachable"; exit 1; }
echo "$BODY" | jq -e '.status=="success"'

I retested both branches.

Worth knowing

• Check your own jq version. The current jq manual documents exit status 4 when no result is produced, so this behaviour depends on the build. The empty-body test works either way.
• Test the alerting chain on a schedule. A probe that can't fail looks exactly like a healthy one until you try it.
• A related trap from the same week: in a sed replacement, an unescaped & means "the whole match". It duplicated an entire crontab line into itself. Rewrite crontabs with a grep-out / append / crontab file cycle instead of sed.