Parsing a million mobile money SMS at 99% accuracy

Mobile money operators in Burkina Faso don’t offer a transaction API. What they offer is a text message:

Cher client, vous avez transfere 50,500.00 FCFA au numero
00000000,NOM PRENOM. Votre nouveau solde est de 12,300.00 FCFA.
ID Trans: XX000000.0000.X00000

(Names, numbers and IDs above are placeholders, not real data.)

That message is the entire integration surface. If you want to build financial software here, SMS is the API: undocumented, unversioned, and changed whenever the operator edits a template. I lead Yembi at DOKAL Technologies and was its only engineer through launch. This is how I built its parser to survive that, and what it taught me about building on hostile interfaces.

The hard part isn’t the regex

Pulling an amount out of one message is easy. Doing it reliably across every operator, every transaction type, and every template variant they’ve shipped and quietly retired over the years is where the real work lives. A message has to be routed to the right account on a dual-SIM phone, recognised as the right kind of transaction before anything is extracted, and then read for amount, counterparty, ID and balance.

The piece that matters most is what happens after extraction. Every parse is checked for internal consistency before it touches the ledger, and anything that doesn’t add up is quarantined for review rather than written silently. In a financial app, a confident wrong answer is worse than no answer.

The decision that saved the project: formats as data, not code

Early on, format rules lived in code. Then an operator changed a template, and fixing it meant shipping an APK and waiting for users to update. Unacceptable for a financial ledger.

The decision that fixed it was treating formats as data the app consumes, not logic baked into a release. A template change became something I could push out without a new build. Just as important: accuracy isn’t asserted, it’s measured. The 99%+ figure comes from evaluating against a body of real anonymized messages before anything ships, so a change that quietly breaks an old format gets caught instead of reaching users. How that evaluation works is the part I’ll keep in-house.

Idempotency, or: 2G will retry you into corruption

Parsing is half the system. The other half is syncing parsed transactions to the backend over networks that drop mid-request. The failure mode that matters: the server commits, the ACK never arrives, the client retries. Your user now has a duplicate 150,000 FCFA withdrawal in their history.

The fix is boring, which is what you want in a ledger: every transaction carries an idempotency key derived from its own content, and the server is built so that replaying the same transaction can’t create a second row. A persistent client-side queue with exponential backoff can then retry as aggressively as it wants. Duplicates are structurally impossible rather than carefully avoided.

Performance: indexes are a product feature

A year of active use generates a serious local database, and Yembi’s whole pitch is “ask questions about your money.” Category breakdowns and date-range queries run against composite indexes designed for exactly those questions, which keeps them under 50ms on million-row tables, on mid-range Android hardware. The dashboard feels instant because the schema was designed backwards from the questions.

What I’d tell anyone building on an unofficial interface

• Treat the format as adversarial. Assume it will change without notice, and make a format change cheap to absorb instead of a fire drill.
• Measure, don’t assert. “It’s accurate” means nothing without something to measure it against.
• Quarantine, don’t guess. A financial app that’s wrong is worse than one that asks.
• Make duplicates impossible, not unlikely.
• Design indexes from the user’s questions, not the entity model.

For most of this project I was the only engineer on Yembi: app, backend, admin portal, parser tooling, Play Store listing. We’ve since hired a second developer, and treating formats as data is the main reason onboarding them was painless.

Yembi launched in March. Next post: the launch itself, what worked, what flopped, and the numbers.