Tips on Quality Assurance process in Machine Learning projects (Viktor Zhurbenko)

### Overview In this talk, Viktor Zhurbenko discusses practical Quality Assurance (QA) practices tailored specifically for machine learning (ML) projects. Rather than framing QA around traditional software testing mechanisms such as unit, integration, or system tests, he defines QA in ML as a continuous set of practices to assess and improve the state of the product throughout its entire lifecycle. In traditional software projects, data is used to test the software, whereas in machine learning, software tests are executed to evaluate and validate the data. Drawing on his experience building natural language processing (NLP) applications—specifically systems that extract structured reservation data from emails for calendar integration—the speaker outlines actionable techniques for data assessment, feature engineering, missing data handling, and production maintenance.

### Key Topics - **Role of the Expert vs. Automated Feature Assessment**: Evaluating whether data features are sufficient or overwhelming. While human domain experts are crucial in areas like medicine or industrial equipment monitoring, automated feature selection algorithms (e.g., using Scikit-learn to eliminate features with low variance) can serve as built-in experts to reduce parameter counts and accelerate training. - **Data Visualization in NLP**: The importance of plotting and color-coding tokens in text to discover and engineer compound features (e.g., spotting that a person and location token together represent an airport name). - **Handling Missing Samples**: Addressing missing data using imputation techniques, such as average imputation or multiple imputation (training small predictive models to infer missing feature values). - **Bias, Variance, and Transfer Learning**: Balancing bias and variance when training models on limited data, and leveraging transfer learning by chaining a generic model with a task-specific model. - **Production Degradation and Mutation Testing**: Managing the inevitable degradation of models in production when live data formats mutate. - **Pragmatic Prototyping**: Prioritizing fast experimentation, simple models, and tooling flexibility over rigid upfront software architecture.

### Technical Details - **Feature Reduction**: In a diagnostic scenario predicting flu based on symptoms (fever, ache, chills, and two blood pressure parameters), blood pressure features exhibited very low variance and had no predictive value. Using Scikit-learn's built-in feature selection to remove low-variance parameters reduced five features to three while maintaining the same hypothesis evaluation accuracy and speeding up training. - **Feature Discovery via Experts**: Speaking directly with domain experts helps uncover missing features without premature code changes (e.g., iteratively determining that identifying a crocodile requires adding a 'teeth' parameter alongside 'green', 'long', and 'in water'). - **Missing Data Imputation**: When samples have missing numerical values, average imputation calculates the mean to fill gaps. Alternatively, multiple imputation uses auxiliary machine learning models trained on available attributes to predict missing values. - **Transfer Learning Pipeline**: When samples are too scarce for complex models, transfer learning can be applied (noted by the speaker as applicable in roughly 5% of real-world cases). In sentiment analysis, an initial generic model learns sentence structure and identifies co-occurring words (e.g., coupling 'great' and 'loved'), while a second, simpler model classifies the sentiment polarity of those coupled words. - **Data Mutation in Production**: Live production inputs inevitably change (e.g., email providers changing structure or users sending images instead of plain text), breaking model assumptions. Teams must monitor production data, collect new features and samples, manually inspect them, and retrain models regularly.

### Takeaways - **Data-First QA**: ML testing is primarily about testing, assessing, and refining the underlying data and features rather than just testing code. - **Always Visualize**: Irrespective of data type—including text tokens—visualizing data reveals structural relationships and informs feature engineering. - **Fight Fire with Fire**: Use small, simple helper models to clean, impute, and assess data before committing to complex production architectures. - **Prototype Over Rigid Architecture**: Avoid spending excessive early effort on rigid frameworks or corporate architecture; pick familiar tools, prototype rapidly, and prove model feasibility first. - **Plan for Long-Term Maintenance**: Deploying an ML model is only the beginning; data mutates over time, requiring ongoing monitoring, manual evaluation, and model retraining.