Why every screen stayed green
The worst automation failures do not raise errors. They present as a system that is working, and the only defence is to monitor freshness instead of existence.
A trading system is down for most of a session. The health check is green the whole time. There are no errors in the log. Memory and processor usage look normal. The deploy that preceded it reported success.
This is not a contrived scenario, it is the characteristic shape of a serious automation failure, and it happens because of a mismatch between what monitoring usually asks and what you actually need to know.
Existence is not liveness
Most monitoring asks existence questions. Is the process running? Is the port accepting connections? Does the health endpoint return a success? Are there errors in the log?
What you need are freshness questions. When did the last tick arrive? When did this node last consume a signal? When was the last order acknowledged? How many requests completed in the last minute?
Three reasons nothing looked wrong
A health endpoint that touches nothing cannot detect anything. If it returns a success without exercising the database, the market feed or the broker connection, then it is reporting that the web server can serve a response. That is genuinely all it means. A deployment that reports "live and healthy" on the strength of such a check is reporting that the container started.
A hanging request produces no log line. Most servers write an access log entry when a request completes. Requests that are stuck have not completed, so they are absent from the log. The result is the most misleading signal in the whole picture: as the system gets worse, the log gets quieter, and a silent access log looks exactly like an idle one.
Resource exhaustion is not resource shortage. When a shared pool of connections is used up, the symptom is that everything waits. Processor usage is low — nothing is computing, everything is blocked. Memory is fine. The database, asked what is wrong, shows connections sitting idle inside open transactions with no lock contention, so it does not look like a database problem either. Every individual signal says things are fine, and the system is unusable.
That last pattern is worth recognising by its signature, because it is the most common cause of "the whole thing is stuck". Low processor use, a silent access log, a green health check and pages that never load is not an ambiguous picture once you have seen it once. It almost always means a shared resource has been exhausted and every request is queued behind it — and the usual cause is something holding that resource while waiting on a slow external call.
Feeds fail in their own particular way
Market data deserves its own warning, because a dead feed is uniquely good at looking alive.
An open socket is not a delivering socket. A connection can remain established while nothing comes down it. Checking that the socket exists tells you nothing about whether data is flowing.
A stale price looks exactly like a quiet market. When a feed stops, the last value you received stays on screen. It is a plausible number. It was true at some point. And a strategy reading it has no way to know it is forty minutes old unless something tracks the age.
One instrument can die while the rest are fine. A subscription for a single contract can fail while the connection and every other subscription are healthy. Aggregate feed health is green; the one instrument you are trading is frozen.
Three defences follow directly, and they are the shape of what good feed handling looks like:
A heartbeat — periodic traffic in both directions, so silence can be distinguished from an idle market. A watchdog on the age of the last message, which declares the feed dead after a defined interval rather than waiting for an error that will not come. And per-instrument freshness, so a single frozen contract is visible rather than averaged away.
The principle underneath all three: the absence of data is not data. A quiet feed must be treated as suspect until something proves the market is quiet, and the thing that proves it is a heartbeat that kept arriving.
What to put on your own screen
If you run automated strategies, these are worth having visible, and none of them requires building anything elaborate:
The age of the last market data update, per instrument you actually trade. Not whether the feed is connected. The age.
The age of the last signal your strategy consumed, alongside the latest signal the source has produced. Two numbers that should be close together; the gap between them is the delivery health from the previous chapters.
The time of the last order acknowledgement, and whether anything was rejected today. "No orders today" is a fact you want to notice deliberately rather than discover at the close.
Your position reconciliation, from the previous chapter, run at the open and the close.
And the habit that ties them together: when something looks wrong, check freshness before you check correctness. Most of the time the question is not "is this number right" but "is this number from today".
The last thing
This module has been about failures that do not announce themselves, and it is worth saying plainly what they have in common, because it is also the answer.
A risk limit set to zero. A signal delivered but never consumed. The same instruction acted on twice. A square-off sent to the wrong book. A position on a contract nothing can name. A price that stopped updating forty minutes ago. In every one of those cases the software did exactly what it was told, and every screen was green.
Nothing broke. Which means no error message was ever going to arrive, and no amount of watching for one would have helped. The only thing that finds these is a deliberate question asked on purpose: has the thing I care about actually happened, recently?
Automation does not reduce the attention trading requires. It moves where the attention has to go — away from the decision, which the machine now handles reliably, and towards the plumbing, which the machine handles exactly as instructed and never questions. Knowing that, and knowing the specific list of seams, is what separates automation that quietly compounds from automation that quietly stops.
What to take from this chapter
- Monitoring usually asks existence questions. You need freshness questions: when did the thing I care about last happen?
- A health check that touches no dependency reports only that the web server can serve a response.
- Access logs are written on completion, so a system full of hanging requests gets quieter as it gets worse.
- Low processor use, a silent log, a green health check and pages that never load is the signature of an exhausted shared resource, not of an idle system.
- An open socket is not a delivering socket, and a stale price looks exactly like a quiet market.
- Feeds need a heartbeat, a watchdog on message age, and per-instrument freshness so one frozen contract is not averaged away.
- The absence of data is not data. Treat a quiet feed as suspect until a heartbeat proves otherwise.
- Watch four things: tick age per instrument, the gap between signals produced and consumed, the last order acknowledgement, and a reconciliation at open and close.
- Check freshness before correctness. The question is usually not whether the number is right, but whether it is from today.