What belongs in an ML integration test?

Most of an ML system isn't the model

A 2015 Google paper on hidden technical debt in ML systems estimated that a mature ML system might be at most 5% machine learning code. The rest is glue: data collection, feature pipelines, serving and monitoring. That ratio is why ML integration tests get confusing. People expect them to test the model, when most of what can break isn't the model.

So I split the checks by what each one can actually tell you. Unit tests cover code you wrote, like a tokenizer or a feature transform. Model evaluations tell you about quality on a dataset. Integration tests tell you the pieces agree with each other once they're connected. End-to-end checks tell you a realistic request comes out right for the user. An integration test that also tries to answer the quality question usually ends up doing both jobs badly.

Test the contracts

The integration layer is where the contracts live. Does the service accept the schema the app sends? Does preprocessing at serving time match what the model saw in training? What comes back when the input is malformed, empty or has a value the model has never seen? Does the app still work when the model version changes underneath it?

The preprocessing question deserves extra attention. When Google surveyed its own teams for The ML Test Score, integration testing had much lower adoption than most other practices, and the authors called checking that training and serving compute the same features "the most important and least implemented test." Martin Zinkevich's Rules of Machine Learning comes at it from the other side: test the infrastructure separately from the learning, and confirm a fixed model gives the same scores in training and in serving.

Use realistic bad inputs here, not only the clean examples from the docs. Missing fields, wrong types, very long text, an empty list. That's what production actually sends.

Exact where you can, tolerant where you have to be

The contract parts are deterministic, so assert them exactly: status codes, schema, field types and error responses. Don't loosen these just because there's a model nearby.

The model's output is different. Pinning an exact prediction in an integration test makes it fail on every retrain, for reasons that have nothing to do with integration. There are two better options. One is a tolerance with a reason behind it, like the ML Test Score's suggestion to compare a new model's quality against the previous one before it serves traffic. The other is behavioral checks like the ones in CheckList: a prediction shouldn't change when you swap one city name for another, and sentiment shouldn't get more positive when you add a complaint to the end. Those assert something you care about without depending on the exact output.

Keep the suite worth running

An integration test that calls a live model endpoint or a third-party API will fail for reasons unrelated to your change, and people learn to ignore it. The testing chapter of Software Engineering at Google says tests start losing value as flakiness approaches 1%, and argues for hermetic tests that don't depend on shared services or on the order they run in. For ML that usually means a small fixed model or recorded responses in the main suite.

The real dependencies still need checking, just on their own schedule. Martin Fowler's contract tests are a good pattern for this: a separate check, run regularly against the real service, that confirms your fakes still behave like it. When it fails you update the fakes, you don't block every build.

None of this replaces evaluating the model. It keeps the integration suite answering its own question, which is whether the parts still fit together.