Per-merchant machine learning on small data: how 50 refunds a month becomes a working model

Most ML practitioners would tell you that running a per-merchant predictive model on Shopify-DTC volume is a bad idea. The argument is structural: a typical Shopify shop with $500K in annual sales sees 8 to 30 fraud chargebacks per year. That is too few labeled positives to train a model. The conventional answer is to pool data across merchants and train one global model, accepting that the global model is worse on every individual merchant than a hypothetical per-merchant model would have been if the data existed.

We did not want the global pooled model. We wanted a per-merchant model. The blocker was the label volume, and the fix was to change the label.

The volume objection

Fraud chargebacks are sparse. On a typical small-DTC merchant the rate runs between 0.05% and 0.5% of orders. A merchant with 5,000 orders per year and a 0.2% chargeback rate produces 10 chargeback labels per year. Ten positives is not enough to train any meaningful model. Even the largest merchants in the small-DTC tier produce maybe 100 positives per year, and that volume only barely supports a small per-merchant model with heavy regularization.

That is the volume objection. It is correct. A model trained on chargebacks per merchant will not work for the long tail of small Shopify merchants.

The reframe

The volume objection assumes the label is fraud chargebacks. There is no rule that says it has to be.

The label we picked is refunds, specifically the binary "did this customer's order produce a refund within 60 days." A typical small-DTC merchant produces between 50 and 500 refunds per month. That is 600 to 6,000 positive labels per year per merchant. The volume problem disappears the moment you change the label.

The catch is that "produced a refund" is not the same as "was fraudulent." Most refunds are normal customer behavior: wrong size, wrong color, didn't fit the use case, customer changed their mind. A model trained to predict refund-likelihood does not directly predict fraud-likelihood.

That is fine. The fraud-likelihood prediction was never the point. The point of the per-merchant ML model is to produce one strong feature that the calibrated risk engine then consumes alongside its other signals. The risk engine is the thing that decides what a refund-likelihood score means in the context of all the other evidence. The model contributes one number; the engine does the policy work.

The reframe has a second benefit: refund data is universal across return-management apps. The previous post in this series covered why we use refund records as the label rather than return records. The same reasoning applies to ML training. A model trained on refund records works the same way on a Shopify-Returns-only merchant and a Loop merchant and a ReturnGO merchant. A model trained on returns produces three different label distributions for the same merchant population.

Per-merchant or global

Once the volume problem is solved, the per-merchant-versus-global question becomes a choice rather than a forced move.

A global pooled model has the advantage of more training data; the disadvantage is that the model's predictions are biased toward the median Shopify merchant's distribution. A merchant whose customer base is far from the median (high-AOV streetwear, low-AOV beauty, multi-currency Plus stores) gets predictions that are correctly trained but wrong-distribution for them.

A per-merchant model has the advantage of fitting that specific merchant's distribution; the disadvantage is that the model has less data and is at higher risk of overfitting.

The tradeoff inverts at different merchant volumes. A merchant with 50 refunds per month does not have enough data to train a per-merchant model from scratch. A merchant with 500 refunds per month does. The middle of the merchant distribution is the awkward zone.

Our compromise is the standard one: per-merchant models with a global pooled fallback. The fallback is a global model trained on the union of refund records across all RefundSentry merchants. New merchants get the global model from day one. As the merchant accumulates 90 days of data and 500+ refund labels, the per-merchant model takes over. The handoff is automatic.

The cold-start prior

For merchants whose data is below the per-merchant threshold but above zero, we use a five-feature cold-start cohort prior. The cohort is defined by industry vertical, AOV bracket, and refund-rate bracket. The prior pools the merchant's available data with the cohort's pooled distribution, weighted toward the cohort during low-volume months and toward the merchant's own data as volume accumulates.

The cohort definitions are deliberately coarse. We do not have enough merchants in any single sub-cohort to produce a stable cohort prior; coarse cohorts are the only stable cohorts. Five features is the smallest set that captures the variance we care about.

Weekly retraining

The model retrains weekly. Weekly is the right cadence for two reasons: (a) a week is short enough that distribution drift in the merchant's customer base does not get baked into the model, (b) a week is long enough that retraining noise (small label-set changes between runs) does not cause score instability that the merchant would notice.

The retraining pipeline is the standard one. Pull the merchant's last 365 days of refund records, join to order features from OrderFeatureSnapshot, train an XGBoost classifier, evaluate on a held-out test set, write the model artifact, register it in a model-registry table that the live scoring path reads.

The XGBoost choice is unromantic and load-bearing. XGBoost handles small training sets well, runs fast enough to retrain weekly per merchant on a single Cloud Run instance, requires minimal feature engineering, and produces calibrated probability outputs without a separate calibration step. We tried logistic regression first; it was fine. XGBoost was meaningfully better on the same training set. Anything more sophisticated (deep nets, transformers) would not have produced enough lift to justify the operational cost on small training sets.

Feature schema versioning

The feature schema can change between model versions. Old OrderFeatureSnapshot rows were built under feature schema 1.4. New rows are built under feature schema 1.5. A model trained on the new schema cannot read old rows directly.

The fix is to version the schema and rebuild snapshots from the raw event store (covered in Post 1 of this series) when the schema bumps. The rebuild is per-merchant per-bump; it runs as a backfill stage during the bump rollout. The model registry tracks which schema version each model artifact expects. The live scoring path verifies the schema match at every prediction call.

This is the place where the raw event store earns its retention budget. Without 365 days of raw payloads, a feature-schema bump would either freeze training data for six months (waiting for forward-from-bump events to accumulate) or fork the model permanently between old-schema and new-schema training cohorts. Both are bad. The rebuild path is the cheap option.

In-process Node training, not Vertex

The model training runs inside the existing application's Node.js process, not on a separate ML platform. The reason is operational simplicity. The training time per merchant is under 5 minutes for the small-merchant tier and under 20 minutes for the largest merchants. Cloud Run scales to zero, so the training compute cost is amortized across the rest of the application's idle capacity. We are not paying for a Vertex AI quota. We are not paying for a SageMaker notebook. We are not paying for a Modal job.

The threshold for moving to a Cloud Run Job is when training time crosses 5 minutes consistently for the median merchant. We are not there yet. The current largest merchants train in under 4 minutes. When we hit the threshold, the move is straightforward (the Node training script becomes a Cloud Run Job; the application enqueues training jobs to it; the artifact-write path stays the same).

Engineer detail. The training pipeline is one TypeScript module that imports xgboost (the JS bindings), reads features from Postgres into a tabular structure, splits 80/20 train/test, fits the model, evaluates AUC on the held-out set, writes the artifact to a ModelArtifact row keyed on (shopId, modelType, featureSchemaVersion, trainedAt). The live scoring path reads the most-recent artifact row matching the current shop and feature schema; if no artifact matches (new merchant or post-schema-bump rebuild not yet run), it falls back to the global pooled model.

The cost story is genuinely small. A weekly retrain across 500 merchants at an average 4 minutes per merchant is 33 hours of compute per week, which on Cloud Run's per-second pricing comes to around $20-30/week at current rates. Adding a merchant adds roughly 4 minutes of weekly compute, so the marginal cost per new merchant is in the cents-per-week range. The budget for the entire training pipeline is well under the budget for the rest of the application's normal traffic.

The XGBoost artifact is small (under 100 KB for the typical merchant; up to 1 MB for the largest). Storage is in Postgres. We do not bother with object-storage tiering because the artifacts are small enough that Postgres is fine. The model-registry table is keyed for fast lookup at scoring time.

What this is not

This is not a fraud-prediction model. It is a refund-likelihood model. The risk engine consumes its output as one feature among many. A high refund-likelihood score does not on its own produce a HIGH-zone risk score; the engine's other signals have to corroborate. A low refund-likelihood score does not on its own produce a LOW-zone risk score; an active fraud-ring membership signal can override.

This is not a real-time-streaming model. The model trains weekly, predictions happen at scoring time (which is real-time), but the model itself does not update online during the week. Online learning produced more instability than lift on our scale.

This is not a model that replaces the calibrated risk engine. The engine remains the policy layer. The model contributes one feature, the engine integrates it.

Take-away

The "you don't have enough data for ML" objection collapses when you change what you mean by "label." Eight chargebacks a year is too few. Fifty refunds a month is plenty. Per-merchant XGBoost on weekly retrain runs on existing infrastructure at near-zero net new cost. The model contributes one strong feature; the calibrated risk engine does the policy work; the merchant gets an engine that learns their customer base while the cross-shop network signals (covered earlier in this series) keep them informed about what the rest of the network is seeing.

This is the capstone of the technical arc. The raw event store, the calibrated risk engine, the cross-shop network, the refund-driven label, the per-merchant model: each one is the prerequisite for the next. They compound.

RefundSentry is an intelligence layer for Shopify return fraud. See pricing for plans during the private beta.