Why monitor at all
A system that nobody is watching tells you about its problems through its users. The first sign of trouble is a complaint, often after the problem has already caused damage. Monitoring is the practice of collecting information about how systems are behaving so that you can see trouble coming, notice it quickly when it arrives and understand it afterward.
Monitoring and alerting are related but different. Monitoring is the collecting and displaying of information. Alerting is the part that interrupts a person because something needs attention. Much confusion, and many burned-out teams, come from mixing these up. Plenty of information is worth collecting and looking at but not worth waking anyone for.
What kinds of information there are
Most monitoring draws on three kinds of data.
Metrics. Numbers measured over time, such as processor use, memory in use, free disk space, the number of requests per minute or how long a page takes to load. They are compact, easy to chart and good for spotting trends.
Logs. Written records of events, such as a user signing in, an error occurring or a service restarting. They answer the question “what exactly happened?” and are covered further in the discussion of reading system logs.
Checks. Simple tests that ask a direct question: Does the website respond? Does the certificate expire soon? Can the backup job report success? A check gives a clear pass or fail.
Many setups add traces, which follow a single request through several systems, but those are mostly relevant to larger software systems.
What to watch
It is easy to collect everything and understand nothing. A useful way to decide is to start from what users care about and work backward.
Is the service working? Can people reach it and do what they need to do? This is the most important question, and a check that mimics a user action, such as loading a page or signing in, answers it directly.
Is it fast enough? A service that responds slowly may be effectively broken. Watch response times, and be aware of the effect of network latency.
Is it producing errors? The share of requests that fail is often more informative than any hardware number.
Is it running out of something? Disk space, memory, network capacity, database connections and licenses all run out. Watching usage trends gives you a warning before the limit is reached, which is far less stressful than discovering it during an outage.
Is something about to expire? Certificates, domain registrations and service accounts have expiry dates that regularly cause preventable outages.
Did the routine jobs run? Backups are the classic case. A backup job that quietly stopped working weeks ago is only discovered when the data is needed, which is why restore testing matters, and why monitoring should confirm that the job ran and finished.
Is it secure? Repeated failed sign-ins, new administrator accounts and unexpected changes to important settings are worth recording and, in some cases, alerting on.
Prefer measures of what users experience over measures of internal causes. High processor use may be perfectly healthy if the service is responding fast. A slow response is a problem whatever the processor says.
Designing alerts that matter
An alert is a request for a person’s attention, and attention is limited. If people receive many alerts that need no action, they start to ignore all of them, including the important ones. This is often called alert fatigue, and it is one of the most common monitoring failures.
A few principles help.
Alert on symptoms, not every cause. “Users cannot sign in” is worth an immediate alert. “One of six servers has high memory” may only deserve a note in a dashboard, if the service is working normally.
Make every alert actionable. Before creating an alert, ask: if this fires at 3 a.m., what will the person do? If the answer is “nothing” or “wait and see,” it should not wake anyone.
Set sensible thresholds. A threshold that is too tight fires constantly. One that is too loose misses real problems. Base thresholds on how the system normally behaves, and adjust after observing results.
Require persistence. A brief spike lasting seconds is often harmless. Alerting only if a condition holds for several minutes reduces false alarms.
Use severity levels. For example, urgent issues notify a person immediately, important issues create a ticket for working hours and minor issues are only recorded. Each level should have a clear meaning.
Group related alerts. If a network fault makes forty services unreachable, one alert pointing to the fault is more useful than forty separate ones.
Write helpful messages. An alert should say what is wrong, where, how bad it is and where to look next. A link to a short procedure saves precious time.
Keeping the system healthy
Monitoring needs maintenance. Review alerts regularly: which fired most, which led to action, and which were ignored? Remove or retune the ones that never need a response. After an incident, ask whether monitoring caught the problem early enough, and if not, what single addition would have helped. This can be part of a post-incident review.
Remember to monitor the monitoring. If the tool that sends alerts fails, silence looks like good news. A regular test message, or a check from a separate system, can confirm the alerting path still works.
Think also about who gets alerted. For on-call work, spread the load fairly, write clear handover notes and make sure the person receiving an alert has the access and the instructions to act. Poorly designed alerting is a common reason people dislike being on call.
Starting small
On a small environment, a short list goes a long way: check that key services respond, watch free disk space, check that backups complete, track certificate expiry dates and record failed sign-ins. Add more only when a real need appears. Good monitoring is not about the number of graphs; it is about knowing, quickly and with little noise, whether the things people depend on are working.