A service can look alive and be dead. The process is running, the health check is green, the dashboard is up — and the thing the system exists to do stopped hours ago. Nothing pages anyone, because by every signal being watched, everything is fine.
We set monitoring up accurately, so you don't have a service that looks alive but really isn't.
What's Included
What the audit covers
- What is monitored, what isn't, and which gaps actually matter
- Alerts that fire constantly and are therefore ignored — alert fatigue is a monitoring failure, not a people problem
- Alerts that have silently stopped firing, which nobody notices by definition
- Rules still firing for hosts and services that no longer exist — the usual reason a team stops reading its own alerts
- Thresholds set for a system that has since changed shape, so they trigger on normal behaviour or miss the condition entirely
- Checks that watch a proxy instead of the thing: process up rather than work done, port open rather than request served
- Whether a host going down floods you with fifty service alerts or inhibits them and sends one
- Who gets woken up, by what, and whether that routing still matches the team
- Coverage of the boring failures that cause real outages: disks filling, certificates expiring, backups that stopped running
What we work with
Our default build is Prometheus, Grafana and Alertmanager, wired into PagerDuty for on-call routing. That is the stack we run ourselves, which is why it is the one we reach for first.
If you already have monitoring, we would rather improve it than replace it. A migration nobody asked for is its own outage risk, and most stacks are closer to working than they look. That applies to commercial platforms as much as open-source ones — we work with Splunk and New Relic, and with CloudWatch where AWS-native is the right answer rather than another tool to run.
The tool is rarely the problem. What is monitored, what alerts, and who gets woken up are the problem, and those decisions survive a change of vendor.
What we build
Alert tuning and rule pruning, which is usually the fastest improvement available. Retiring rules that watch decommissioned infrastructure, correcting thresholds that no longer match the system, and cutting the noise that trained your team to ignore the dashboard. This is unglamorous and it is often the single highest-value day of the engagement.
Routing and escalation, so the right person is woken for the right reason, and a host going down sends one alert instead of fifty.
Checks that catch the failures which actually cause outages — disks filling, certificates expiring, backups that quietly stopped, and output-level tests that fire when a service stops doing its job rather than when its process exits.
SLOs and error budgets where they help, and nowhere they don't.
A runbook per alert class, so being paged at 3am comes with instructions instead of archaeology.
We build and tune the monitoring; the alerts route to your team and your on-call rotation.
What you get
A written assessment of current coverage with the gaps ranked by what they would cost you, a remediation plan, and documentation for whatever we build. If your monitoring is already sound we will tell you that, and the audit is cheaper than the outage that proves otherwise.
A worked example, from our own infrastructure
One of our own streaming services sat reporting healthy for an extended period while the output it existed to produce had stopped. The process was up the whole time. The unit file said active. Nothing alerted, because the check watched the process and not the output.
That is the failure this service exists to prevent, and it is far more common than an outage that announces itself. Monitoring that watches processes tells you the server is on. Monitoring that watches outputs tells you the business is working.
Monitoring is also the easiest thing to install and the hardest thing to get right. Any tool gives you a dashboard within an hour; knowing which of four hundred possible signals actually predicts an outage for your systems is the work. That is why this starts as an audit rather than a build — and why we are willing to tell you about one of ours.