ESR

Why Are You Still Working with CSV Files in 2025? (An NHS Digital Leader’s Pain Story)

A composite NHS digital leader story about the hidden operational pain of ESR CSV drops—late errors, duplicated feeds, and endless workarounds—and the practical path to modernise with a governed ESR-to-API layer.

WeHub

Reading time: ~8–10 min

A familiar meeting, a familiar problem

It’s a Tuesday. You’re in the kind of meeting the diary auto-accepts because declining it feels irresponsible. The title is something polite like: “Workforce Data Alignment – ESR Feed Review”.But everyone knows what it really means: “Why don’t the numbers match again?”The digital lead opens with the usual calm: “We just need one source of truth.”Someone from operations replies, equally calm: “We thought ESR was the source of truth.”The vendor rep is also calm: “We consume the ESR extract you provide.”And then the quiet moment happens—the one where you realise you’re not arguing about data. You’re arguing about interpretations of a CSV file.

“It works… until it doesn’t”

On paper, it’s simple:

ESR exports a CSV
systems ingest it
everyone stays updated

In practice, “it works” means:

it worked for one system
until another system needed a slightly different mapping
until a column changed
until a department code appeared unexpectedly
until leaver dates didn’t line up
until positions conflicted
until the nightly drop arrived late
until someone asked for “just one extra field”

Now you don’t have one ESR integration. You have five. Then ten.Each consumer has its own:

parsing logic
mapping rules
filters
schedules
and error handling (often: none)

And it’s fine… until you need to change anything.

The hidden costs nobody budgets for

CSV-based integration doesn’t always show up as a line item. That’s the trick.The cost hides in:

manual reconciliation (“just update this spreadsheet so payroll doesn’t explode”)
late error discovery (issues found after data spreads into systems)
mini projects (small mapping updates that require multi-owner coordination)
shadow processes (side databases, scripts, ad-hoc exports)
confidence loss (teams stop trusting dashboards and go back to email chains)

The digital lead describes it perfectly: “We’re paying for integration with human attention.”That’s the most expensive currency you have.

The moment it becomes operational (not technical)

Then it happens—the week you stop talking about architecture and start talking about impact. A starter doesn’t appear in time for a downstream workflow. A leaver’s access removal lags. A position change triggers a mismatch somewhere critical.The outcome isn’t a bug report. It’s an incident.And once workforce data becomes operational, the tolerance for “CSV quirks” drops fast. Because the problem isn’t the file. It’s the fact that the file is being used as a contract.

Why CSV is a bad contract, not a bad format

CSV is fine as a transport. It’s even fine as an ingestion method.What’s not fine is this pattern: Raw ESR file → every system parses it → everyone invents their own truth.That’s how you get:

schema drift
inconsistent validation
batch-only truth
duplicated integration logic
and change that never ends

The frustration isn’t about CSV itself. It’s about what CSV enables when it’s left unmanaged: distributed responsibility and ambiguous ownership.

The turning point: stop feeding files to downstream systems

The turning point was not “we need a new platform.” It was one decision:Downstream systems should stop consuming raw ESR files.Not immediately. Not everywhere. But intentionally, step by step. Instead, they created a governed middle layer: ESR exports in → normalise + validate → publish APIs/events out.This did two things:

It centralised rules and validation
It created a stable contract downstream systems could trust

Even if ESR remained file-driven, the organisation stopped living as file-driven.

What changed after shifting to ESR-to-API

No miracles. Just fewer fires.1) One version of workforce logicInstead of every consumer defining ESR differently, the organisation defined it once.2) Errors moved upstreamBad records were quarantined with reasons. Clean outputs flowed forward.3) Operations got faster (without ripping everything out)APIs and events made starter/leaver/position changes more responsive—even when ingestion remained batch.4) Change became manageableMappings updated centrally. Contracts versioned. Rollouts staged.5) Audit became realIt was suddenly possible to answer:

what changed?
where did it come from?
who consumed it?
why did it fail?

Without detective work.

Lessons learned (for any NHS organisation)

Don’t modernise the file. Modernise the contract.
Validation is not a phase. It’s a feature.
Start with a slice (starters/leavers/position integrity).
Run shadow mode first. Prove trust before switching consumers.
Make multi-tenancy a first-class concept if you operate regionally.
Keep the rollout calm: one consumer, then templates, then scale.

The bottom line

If you’re still working with CSV files in 2025, it’s not because you’re behind. It’s because the system evolved around the easiest transport—until transport became the contract.The realistic way out isn’t a big-bang rebuild. It’s a governed layer: ESR exports in → rules + validation → clean APIs/events out → audit + exceptions.That’s how you stop paying the integration tax with human attention.

Keywords

ESRNHS DigitalWorkforceIntegrationAPIsData GovernanceAutomation