Clustering Free-Text Return Comments With LLMs

Every returns portal has a dropdown and a comment box. The dropdown produces clean data that is frequently wrong, because customers pick the first plausible option. The comment box produces messy data that is frequently right, because people describe what actually happened. Most merchants analyse the dropdown and ignore the box.

Language models make the second source tractable for the first time, but only if you resist the temptation to ask the model for an answer instead of a structure.

Why Dropdowns Lie

Reason codes are chosen under friction. The customer wants a label and a refund, not a taxonomy exercise. Given six options they select the one nearest the top that is not obviously false, which systematically inflates whichever generic category sits first in your list.

The comment beside it often says something quite different: not "wrong size" but "the waist was fine but the leg was far too long", which is a length problem, not a sizing problem, and points at a completely different fix. That distinction never survives a dropdown, and it is exactly the kind of detail a good return reason taxonomy is trying to capture.

Ask for Structure, Not Conclusions

The failure mode with language models here is asking "why are returns up?" and accepting the narrative that comes back. Models are fluent and will produce a confident explanation from thin evidence. The reliable pattern is to use the model as a classifier and an extractor, then do the counting yourself in SQL.

Concretely: give the model a fixed label set plus an explicit "does not fit any label" option, ask it to assign one label per comment with a confidence, and require it to quote the span of text that justified the label. The quote is the control. It lets a human spot-check fifty rows in ten minutes and see immediately whether the labels are grounded in the text or invented.

Let the Taxonomy Emerge, Then Freeze It

Start with open clustering on a sample of a few thousand comments to discover what customers actually talk about. Read the clusters. Turn them into a fixed label set. Then switch to classification against that fixed set, because a taxonomy that changes every run produces trends you cannot compare over time.

Reserve a permanent "unclassified" bucket and watch its share. When it grows, something new is happening in the catalogue or the market, and that growth is often the earliest signal you have of a developing problem. That is the same early-warning logic as return reason code drift, applied to text rather than codes.

• Use the model to label and extract; do the aggregation in your warehouse.
• Require a quoted justification span for every label assigned.
• Keep an explicit unclassified bucket and alert on its growth.
• Freeze the label set before you start reporting trends.
• Spot-check a random sample every run; never trust the accuracy you measured once.

A model that cannot show you the sentence it based the label on is not classifying. It is guessing eloquently.

Privacy Is Not Optional Here

Free-text return comments contain personal data far more often than teams expect. Customers write names, phone numbers, addresses, health details explaining why something did not fit, and occasionally complaints about a named employee. Sending that text to an external model is a processing decision with real obligations attached under frameworks such as the GDPR.

Practical mitigations are straightforward: strip obvious identifiers before processing, use a provider with a data processing agreement and no training on your data, keep the retention window on raw comments short, and store the derived labels rather than the raw text where you can. This sits squarely inside your existing returns data privacy compliance scope rather than beside it.

Turning Labels Into Action

Classification is worthless until it is joined to something. The join that produces the most value is to the product: comment labels grouped by SKU turn a vague quality worry into a named list of products with a specific defect pattern.

The second most valuable join is to time. A label that was two percent of comments for six months and is now nine percent for one product line is a signal worth a same-week investigation, and it usually precedes the return rate itself moving. Merchants already running product-level return rate analysis get this join almost for free, because the SKU dimension is already there.

Finally, close the loop back to the dropdown. If the comments consistently reveal a category your reason codes do not offer, add it. The point of mining free text is not to replace structured data permanently but to find out what the structure is missing.