Payments are the moment a digital promise becomes tangible. That is why reliability matters more than cleverness.

Over three integrations we learned the same lesson three times: a payment system is only as good as its failure handling. Here is what that actually looks like.

Lesson 1 — Make the state machine explicit.

Every payment has a life cycle: initiated, submitted, confirmed, completed, failed, and reversed. We modelled these as an enum with strict transitions. A payment in "confirmed" cannot jump to "failed" — it must pass through "reversed". This sounds obvious until you realize most bugs come from implicit states.

Lesson 2 — Retry like a human would.

Network timeouts are rarely permanent. We added exponential backoff with jitter, capped at three attempts, and a circuit breaker that pauses retries if the failure rate crosses 50% in a rolling minute. The key rule: never retry a customer-initiated action more than the user can see.

Lesson 3 — Tell the user what happened.

When a transaction fails, the message must name the cause. "Payment failed" is useless. "M-Pesa timed out — please check your phone and confirm, or try again" is actionable. We instrumented every failure with a code and surfaced it in the UI.

Lesson 4 — Reconciliation is not optional.

Safaricom callbacks can arrive out of order, duplicate, or not at all. We built a nightly reconciliation job that compares our ledger against Safaricom's settlement reports. Mismatches trigger alerts and manual review. Without this, you will lose money silently.

Lesson 5 — Idempotency keys everywhere.

Every STK Push request carries a client-generated idempotency key (UUID v4). Safaricom echoes it back in the callback. Our API layer deduplicates on this key before any state change. This prevents double-charges when users hammer the "pay" button or network retries fire.

Lesson 6 — Observability pays for itself.

We log every state transition, every callback payload, every retry attempt, every reconciliation mismatch. Dashboards show: success rate by hour, callback latency p50/p95/p99, retry distribution, circuit breaker state. When Safaricom has an incident, we know before support tickets arrive.

Lesson 7 — Test with production-like chaos.

Our CI pipeline includes chaos tests: simulated Safaricom downtime, callback delays (0–120s), duplicate callbacks, malformed payloads, network partitions. Tests must pass before merge. This catches regressions that unit tests miss.

Our approach pairs clear transaction states with safe retries, useful observability, and copy that helps people understand what happened. These patterns travel well beyond one provider. They are the foundation of every dependable commerce experience.

Reference implementation:

See our open-source mpesa-resilient-client for a production-ready TypeScript client with all these patterns baked in.

Build what comes next

Want to explore what these ideas could mean for your business? Start a conversation with our team.