← All services Data aggregation & cleansing

Data aggregation & cleansing

Pull from messy sources, normalize, dedupe, and turn it into something your team can actually act on.

When you need this

  • You have data — but it's spread across vendor exports, third-party feeds, and PDF reports.
  • Your records have the same customer listed under three slightly-different names.
  • An external system updates faster than your manual import process can keep up.
  • Your sales team is making decisions on numbers nobody fully trusts.
  • You're paying for data you can't actually use because it doesn't line up with anything else.

What we do

  • ETL pipelines from any combination of sources (CSV, XLSX, JSON, APIs, databases)
  • Structured extraction from PDFs and other unstructured formats
  • Public-records and third-party data enrichment (people, properties, companies)
  • Deduplication, fuzzy matching, normalization rules
  • Validation and confidence scoring so you know what's reliable
  • Scheduled refreshes that land cleaned data where your tools expect it
How this gets built now

This is where AI helped me least

Writing a fuzzy matcher is a solved problem. Deciding what counts as a match is not. Whether a score auto-merges or goes to a person, whether two sources become one canonical record or get reconciled at query time — those are business calls whose consequences surface months later, as a number nobody trusts. The mechanical parts got faster. The deciding never moved, and of everything I do this is the bucket where that gap matters most.

How I use AI coding agents on client work →
Examples from real engagements

What this looks like

Title Pro → ACC matching

Two large datasets that each described the same properties using different identifiers — fuzzy matched, scored, and reconciled into a single canonical record set the client could query against.

Multi-source auction aggregation

Hundreds of auction listings each week from a half-dozen different bidding platforms — pulled, normalized to a single schema, deduplicated, and made queryable through one portal.

Person vs trust classification

Property records often list a trust as the legal owner, hiding the underlying person. Built a classifier that flags trust ownership and links it back to the real beneficial owner where possible.

Common questions

Questions people ask before starting

Can you pull structured data out of PDFs?

Yes — structured extraction from PDFs and other unstructured formats is routine work here. The honest caveat is that extraction accuracy depends heavily on how consistent the documents are. A thousand PDFs from one system is a solved problem; a thousand from forty different senders is a project, and it is worth sampling before anyone quotes it.

How do you handle duplicates across sources?

Fuzzy matching where exact matching quietly misses people, confidence scoring on every match, and a review queue where a person decides the uncertain ones. On a nationwide recruiting firm's participant database — tens of thousands of people matched on names, phones and emails that were never linked — the review queue was the part that made the result trustworthy, not the matching algorithm.

What sources can you pull from?

Any combination of CSV, XLSX, JSON, APIs and databases, plus public-records and third-party enrichment for people, properties and companies. Sources that disagree with each other are the normal case rather than the exception, so the pipeline has to record which source said what.

How do I know which records I can trust?

Validation and confidence scoring are part of the output, not a separate report. A cleaned dataset that cannot tell you which rows were certain and which were a judgment call pushes the risk downstream to whoever acts on it.

Does the cleaned data stay clean?

Only if the refresh is scheduled, which is why it is. Scheduled refreshes land cleaned data where your tools already expect it. A one-time cleanup is worth doing and is usually stale within a quarter, so the pipeline matters more than the first pass.

What does data work cost?

I bill new work at $95 to $125 an hour, against $150 to $300 for a typical agency blended rate. For this kind of work the estimate depends almost entirely on how much the sources disagree with each other: pulling ten thousand rows out of one clean export is an afternoon, and reconciling three systems that each spell the same company differently is the actual project. A sample of the real data produces a real number, which is why I ask for one before quoting.

Should I use an off-the-shelf tool like Fivetran or Airbyte instead?

Yes, if your sources are on their connector list. A managed pipeline from a vendor who already supports your systems will be cheaper and steadier than anything written by hand, and I will tell you so. Custom earns its place on the sources nobody builds connectors for — a vendor PDF, a public records page, a supplier portal you have an account for, a feed whose columns change without notice — and on the reconciliation afterwards, which is the part those tools deliberately leave to you.

Got messy data that needs to be useful?

Tell me what you're trying to fix — I'll reply the same day.