Built an internal price-intelligence platform for an online retailer that was losing sales to competitors on price — it monitors competitor prices, imports supplier price-lists, and updates prices in the live shop automatically.
The problem
The retailer's catalog held over 21,000 products, and competitors were undercutting on price. Manually tracking rivals across 10 competitor web-stores at that scale wasn't feasible — pricing decisions lagged the market.
My role
Designed and built the platform end-to-end (2018–2019).
What I built
- Competitor price crawler covering 10 stores — parallelized with worker-farm, parsing with Cheerio, MongoDB as the crawl cache, MySQL as the catalog store. Each competitor site needed its own parser for its HTML, product, and catalog structure.
- Polite, resilient scraping — progressive-cooldown rate limiting so crawlers never hammered target sites, and workarounds for anti-scraping countermeasures.
- Product matching — mapping competitor SKUs and product pages onto our 21,000+ item catalog. The genuinely hard part: naming, units, and structure differ on every site.
- Supplier price-list import — each supplier ships their own custom-formatted XLS price list. Built a web UI for mapping supplier columns onto catalog fields with a live preview of matched values, plus pluggable per-field transform handlers to normalize supplier quirks: whitespace trimming, extracting UPC codes embedded in product names, routing a price into the right currency by its sign with conversion at the current exchange rate, summing multi-warehouse stock columns, and interpreting "empty = in stock" conventions.
- Safe automated repricing — prices are recalculated and updated on the live shop on a schedule, with no manual work. Added an anomaly check: if a price jumps too sharply, it isn't published right away — it goes to a manager for approval, so a data error never reaches the storefront.
- Infrastructure & operations — owned integration and deployment end-to-end: set up the Linux server, deployed the application with Docker (docker-compose), and configured backups of the dockerized app and its data.
Results
- In production use for over a year — I personally maintained and supported it for the first six months.
- Automatic price adjustment kept the shop competitive and attracted significantly more buyers to the store.
- Whole-assortment price monitoring became a single-admin task; supplier price-list imports got dramatically faster, with live currency-rate conversion and protection against erroneous over- or under-pricing.
Stack
Node.js · Express · Sequelize / MySQL · MongoDB · Cheerio · worker-farm · OpenCart · XLSX · Docker · Linux
