How we built an intelligent product categorisation system

Picture the situation: you have a catalogue of tens of thousands of metalworking products – from HSS milling cutters and machine taps through cobalt drills to turning tools and measuring instruments. Every product needs a unique SEO description with precise technical specifications, has to sit in the right place in a hierarchical category structure (Milling → HSS cutters → Slab mills, say), and the data comes from a range of different supplier sources. Processing that volume by hand means months of work.

That was exactly our challenge. In this post I want to talk about the obstacles we hit along the way and how we managed to build a system that automates the process.

Project context

Our client – a shop specialising in metalworking tools and metal parts – needed a solution that would:

• generate SEO-optimised product descriptions automatically
• assign products to a 3-level category structure
• aggregate data from a range of supplier sources
• process tens of thousands of products efficiently

The main challenges and obstacles

1. Batch processing performance

The challenge: in the first tests with a catalogue of more than 80,000 products, processing took unacceptably long. The system tried to process everything at once, which led to:

• excessive memory use (the system hung)
• long waits (hours of processing)
• inefficient queries to external data sources

The solution: we introduced an intelligent batch processing system:

• Chunking the data – products are processed in controlled batches of an optimal size
• Controlled concurrency – a configurable number of products processed at once
• Query optimisation – limiting parallel queries to supplier databases
• Metrics and monitoring – detailed logging of processing time, memory use and errors

The result: processing time fell significantly and memory use was brought down to a safe level.

2. Integrating multiple data sources

The challenge: products came from different suppliers’ systems, each with its own data format. Some products (specialist taps or cutters with non-standard parameters, for instance) were simply not available in the sources, which caused:

• repeated queries for the same non-existent products
• unnecessary API costs (Google searches for products with no technical data)
• no way of tracking which products we had already checked – particularly awkward with products that have many variants (taps in different diameters and thread pitches, say)

The solution: we introduced a flagging system:

• Persistent “not found” flags – when a product cannot be found in a data source, the system flags it and skips it on subsequent runs
• A reset mechanism – flags can be cleared manually when the data sources are updated
• Intelligent skipping – the system automatically passes over flagged products, saving time and resources

3. Interface performance

The challenge: the product page loaded slowly, and frequent automatic refreshing caused:

• excessive load on the server
• a sluggish interface
• performance problems with large data sets

The solution: we carried out a thorough optimisation:

• Removing auto-refresh – replaced with a manual refresh button
• Server-side pagination – reducing the default number of products shown at once
• Debounced search – a delay before the query runs, cutting the number of unnecessary requests
• Lazy loading – loading data gradually, as needed
• Component memoisation – fewer unnecessary re-renders

4. Accuracy of category assignment

The challenge: assigning products to categories automatically had to be highly accurate, particularly in a technical sector where getting the category wrong causes problems with search. For example, an M8 metric machine tap has to end up in “Threading → Machine taps → Metric machine taps”, not among hand taps or dies. The standard approach gave satisfactory but not perfect results.

The solution: we deployed an advanced categorisation system:

• Iterative reasoning – the system analyses a product repeatedly, refining the assignment with each pass
• Confidence scoring – each category receives a confidence score
• Automatic evaluation – the system assesses the accuracy of the assignment and suggests corrections
• Decision history – a record of the decision process for audit and learning

5. Managing worker configuration

The challenge: the background processing system (the worker) needed flexible configuration, but the settings had to survive restarts.

The solution:

• Configuration in the database – worker settings are saved in a NoSQL database and survive a server restart
• Environment variables as defaults – initial values can be set through environment variables
• A user interface – a control panel allows parameters to be changed without restarting the system
• Real-time metrics – monitoring of processing progress and statistics

How it works (at a high level)

System architecture

The system consists of several key components:

1. The data layer

• the main database holding products, categories and configuration
• integration with supplier systems as a source of product data
• integration with a web search engine to enrich the data

1. The processing layer

• a worker system for background processing
• integration with AI for generating descriptions and categorising
• intelligent scoring of search results

1. The presentation layer

• a modern web interface
• a dashboard with real-time metrics
• tools for importing and managing products

How a product gets processed

An example: an M8 metric machine tap

1. Data import – the product is imported from the supplier’s system or entered manually through the interface. The system receives the basics: SKU, name, technical parameters (diameter, thread pitch, material)
2. Enriching the data – the system searches Google for information about the product, scoring the relevance of the results. For technical tools, manufacturer specifications, standards (DIN, ISO) and range of applications matter most
3. Generating the description – AI analyses all the available data and generates a unique SEO description with precise technical information, materials, applications and standards
4. Assigning a category – the system places the product in the right category in the hierarchy (Threading → Machine taps → Metric machine taps, for instance) using an iterative decision process that accounts for the product’s technical specifics
5. Evaluation – automatic assessment of the accuracy of the assignment and suggested corrections, which matters particularly with technical products, where getting the category wrong causes problems with search

Intelligent features

• Dynamic keywords – the system automatically pulls phrases from category names (“machine taps”, “HSS cutters”, “cobalt drills”) and uses them to score search results. In a technical sector, multi-word phrases matter especially, because they identify specialist products more reliably
• Hybrid scoring – a combination of different factors for scoring search results. The system rewards results containing technical standards (DIN, ISO), material specifications (HSS, carbide) and technical parameters
• Lock management – a mechanism preventing the same product from being processed twice at once, which matters particularly on large imports of tools with many variants (taps in different diameters, for instance)
• Audit and history – a full history of changes and decisions for every product, which allows category assignments for specialist tools to be verified

Conclusions and lessons

What worked well

• Modular architecture – splitting the system into layers made it easier to develop and maintain
• Configurability – being able to adjust parameters without changing code
• Monitoring and metrics – detailed logging helped us identify problems
• An iterative approach – gradual improvement rather than revolution

The most important lessons

1. Performance from the start – with large volumes of data you cannot leave optimisation until later
2. Intelligent skipping – sometimes the best solution is not to try again
3. The user at the centre – even the best backend will not help if the UI is slow
4. Flexible configuration – needs change, and the system has to account for that
5. Monitoring is essential – without metrics you do not know where the problems are

Summary

Building a system to automate product categorisation was a journey full of technical and organisational challenges. From performance problems through integrating multiple data sources to optimising the interface, every problem called for a considered solution.

What proved key to success was:

• A systematic approach – identifying problems, planning solutions, implementing and verifying
• Flexibility – a willingness to change approach when the first solution does not work
• Focus on the user – both internal (the people operating the system) and end users (the platform’s customers)
• Continuous improvement – the system is not static; it evolves along with what is needed

Today the system efficiently processes tens of thousands of metalworking products – cutters, taps, turning tools – generating unique technical descriptions and placing them precisely in the right categories. That is not only a saving in time and cost but a better experience for the end customers: metalworking professionals who can find the tool they need, with the right technical parameters, faster.