Beacon needs to check customers’ endpoints from several regions at once, and Workers is a natural fit: deploy once, get nodes everywhere. But a natural fit isn’t the same as a smooth one.
Gotcha one: probes get blocked as bots
Many customers have a WAF in front of their API. We now send probes with a fixed User-Agent and link to a public page listing our IP ranges, so customers can allowlist us in one click.
Gotcha two: CPU time is not wall-clock time
A probe may wait 8 seconds for a response, but that wait doesn’t count as CPU time. Parsing a large response body does. So we only read the headers and the first 4KB, which is plenty to judge the status.
Gotcha three: don’t trust local clocks
When results from several regions are merged, node clocks drift slightly. We sort by the time the central node received each result, and use node timestamps only to compute durations.
Gotcha four: retries need a cap and a delay
const delays = [0, 1500, 4000];
for (const delay of delays) {
await sleep(delay);
const res = await probe(target);
if (res.ok) return res;
}Three failures in a row count as a real outage. Otherwise a single network blip wakes a customer at 3 a.m.
Gotcha five: alert fatigue
If you notify about the same incident every minute, people mute you quickly. Now it’s one message when an incident starts, one reminder after 30 minutes, and one when it’s resolved. That’s all.
The biggest lesson from building a monitoring product: bothering users less is harder than detecting more problems.