E-commerce / data
Product data normalization system
An automated pipeline for collecting, deduplicating and enriching product data — turning a 90-minute manual entry job into 10 minutes, for 150+ back-office users.
Overview
Every catalogue business has the same quiet tax: someone, somewhere, is typing product data into a form. Titles, descriptions, specs, images — pulled from suppliers, cleaned up, deduplicated, and made consistent enough to sell. At scale that tax is enormous, and it’s paid in the most error-prone currency there is: human attention on repetitive work.
Problem
Manual entry took around 90 minutes per product. Worse than the time was the drift: the same product would enter the system twice from two suppliers with slightly different wording, and now you had duplicates to reconcile forever. The content wasn’t just slow to produce — it was fighting itself.
Solution
An automated system for collecting, processing and storing product data, built around one non-negotiable idea: content uniqueness. It ingests from third-party sources, normalizes everything into a consistent shape, detects and collapses duplicates, and hands the back office a clean record to review rather than a blank form to fill. It served 150+ back-office users and integrated with the surrounding third-party software they already worked in.
The headline result: entry dropped from 90 minutes to 10, and product updates got roughly 10× faster. But the number I care about more is the one that’s hard to put on a slide — the duplicates stopped multiplying.
Architecture
A Laravel backend owns the ingestion and normalization pipeline; a Vue.js interface gives the back office a review-and-correct workflow rather than a data-entry one. MySQL holds the canonical catalogue.
The pipeline is deliberately staged — collect → normalize → deduplicate → enrich → publish — so each concern is isolated and independently fixable. Deduplication is the heart of it: matching “the same product” across sources that disagree on formatting, units and naming is a fuzzy problem, and keeping that logic in one well-tested stage (instead of smeared across the importers) is what made it maintainable.
Technical challenges
Deduplication is easy to get 80% right and genuinely hard to get to 99%. The long tail is full of near-misses: two listings that are the same product in different packaging, or different products that a naive match would merge. Get it wrong in the merging direction and you silently destroy data, which is worse than a duplicate. So the system is biased toward surfacing likely matches to a human rather than auto-merging aggressively — automation for the obvious cases, a person for the ambiguous ones.
Throughput mattered too. Normalizing and matching against a growing catalogue has to stay fast enough that the back office isn’t waiting on it, which meant being careful about how matching queries hit the database as the catalogue grew.
Lessons learned
The win wasn’t “replace the humans” — it was change what the humans do. Moving them from typing to reviewing kept their judgement in the loop exactly where it was valuable (the ambiguous matches) and took it out of the loop where it was just expensive (retyping a spec sheet). That framing — automate the boring certainty, escalate the interesting doubt — is one I’ve reached for on every data pipeline since.