The same signal, twice
Delivery systems retry. That is not a bug, it is how they guarantee anything arrives at all — and it means your strategy must be built to receive the same instruction more than once.
Your strategy received one signal and placed two orders. Nothing malfunctioned.
This surprises people, so it is worth stating the underlying fact plainly: almost every delivery mechanism in existence offers at-least-once delivery, not exactly-once. If a sender cannot confirm that a message arrived, it sends it again. That is the only way to guarantee delivery over an unreliable link, and the unavoidable consequence is that duplicates happen.
Exactly-once delivery is extraordinarily hard to build and essentially nobody offers it to you. So the duplicate is not an anomaly to be prevented upstream. It is a normal event that your side has to handle.
Where duplicates come from
A lost response. Your endpoint received the signal, processed it, placed the order — and the acknowledgement back to the sender was lost. From the sender's point of view the request failed, so it retries. Both attempts are legitimate; the second one is indistinguishable from a genuine new signal unless you make it distinguishable.
A sender's own retry policy. Webhook senders typically retry on timeout or on any non-success response. If your endpoint is slow enough to time out while still completing the work, you will reliably get duplicates — and the slower your processing, the more often.
A reconnect replaying a backlog. A consumer that drops its connection and reconnects often re-reads from its last confirmed position, which may be behind where it actually got to. Everything between is delivered a second time.
A replayed tick. This is the subtle one. Market data is sometimes re-sent — on reconnection, on a snapshot refresh, on a gap-fill. A replayed tick is a copy of a past event, not a new event, and a strategy that treats it as new will act on something that already happened. In a system with several layers that can each replay, this can arrive more than once from more than one direction.
That last case deserves the rule it implies: a copy must be marked as a copy. If a tick, bar or signal can be re-sent, it needs to carry a flag saying so, and every consumer needs to respect the flag. A replay that looks identical to a live event is a design that guarantees this class of bug, and no amount of care in the consumers will fix it.
The fix is a key, and a memory
Three requirements make this work, and each one is a place it commonly goes wrong.
The id must be stable across retries. It has to identify the event, not the delivery attempt. An id generated when the message is sent will be different on the retry and the deduplication will never fire. Derive it from the signal's own content and timing, or have the sender generate it once and reuse it on every attempt.
The record must outlive a restart. A memory of seen ids held only in process is lost when the process restarts — and a restart is exactly when a backlog is most likely to be replayed. The record needs to be somewhere durable.
The window must be long enough. You cannot remember every id forever, so you keep them for a period. Make it comfortably longer than the longest retry window of any sender you accept from. Too short and a late retry sails through as new.
Make the action idempotent too, where you can
Deduplication by id is the main defence. A second, independent one is to make the action safe to repeat, so that a duplicate that slips through does less damage.
The useful distinction is between instructions that are naturally repeatable and instructions that accumulate.
"Be long 300 units" is repeatable. Received twice, the second one finds the position already correct and does nothing. The instruction describes a desired state.
"Buy 300 units" accumulates. Received twice, you are long 600. The instruction describes a change.
Where you have the choice, state the desired position rather than the change. Many strategies can be expressed either way, and the state form is strictly safer under duplication — and, as a bonus, self-correcting if a previous order was partially filled. It also makes a position mismatch repairable: reasserting the intended state fixes a drift that a stream of deltas can never recover from.
One more trap: restored events keep their original times
When a system restarts and recovers its state, the rows it restores describe things that happened earlier. Their timestamps are the original exchange times — not the recovery time, which is correct and occasionally surprising.
The surprise arrives when something downstream treats "the time I received this" as "the time this happened". After a restart, you can receive a batch of events whose timestamps are hours old, and a rule reading them as current will act on a market that has moved on. Equally, a monitor that sorts by receipt time rather than event time will show a restored history in the wrong order.
The rule is the same one that governs the whole module: an event's time is a property of the event, not of when you learned about it. Carry it, store it, and use it for every decision that depends on when something happened.
What to take from this chapter
- Delivery is at-least-once, not exactly-once. Duplicates are normal operation, not an anomaly to prevent upstream.
- They come from lost acknowledgements, sender retries, reconnects replaying a backlog, and replayed market data.
- A replayed tick is a copy of a past event. A copy must be marked as a copy, and every consumer must respect the mark.
- Deduplicate on an id that identifies the event and is stable across retries — never generated at send time.
- Keep the record of seen ids somewhere durable and for longer than the longest sender retry window.
- Record duplicates rather than dropping them silently. The record is how you learn your endpoint is timing out.
- Prefer "be long 300" to "buy 300". State instructions are safe to repeat and repair drift; delta instructions accumulate.
- Restored events keep their original times. An event's time is a property of the event, never of when you received it.