Study guide
3.3 Machine Learning Development Frameworks
ISTQB Certified Tester AI Testing CT-AI
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts (CT-AI v2.0) 1.2 Scoring, K-levels & Chapter Weights
2 Introduction to AI: Systems and Capabilities
2.1 AI-Based vs Conventional Systems 2.2 Narrow AI, General AI and Super AI 2.3 Types of AI Technologies 2.4 Generative AI
3 AI Hardware, Hosting, Frameworks and Governance
3.1 Hardware for Machine Learning Systems 3.2 Development and Hosting of AI Models 3.3 Machine Learning Development Frameworks 3.4 Regulations and Standards for AI
4 Quality Characteristics and Acceptance Criteria
4.1 ISO/IEC 25059 AI Quality Characteristics 4.2 AI and Safety 4.3 Acceptance Criteria for AI-Based Systems
5 Machine Learning Forms, Workflow and Data Preparation
5.1 Forms of Machine Learning 5.2 The Machine Learning Workflow 5.3 Pretrained Models, Fine-Tuning and RAG 5.4 Data Preparation Activities
6 Datasets, Classification Metrics and Neural Networks
6.1 Training, Validation and Test Datasets 6.2 Calculating Classification Metrics 6.3 Structure of Deep Neural Networks 6.4 Neural Network Coverage Measures
7 Testing Challenges for AI-Based Systems
7.1 Locked vs Adaptive AI-Based Systems 7.2 Why Statistical Testing Is Needed 7.3 Test Oracles for AI-Based Systems
8 GenAI Testing, Test Levels and Risk-Based Testing
8.1 Testing Generative AI 8.2 Red Teaming for GenAI 8.3 Test Levels for Machine Learning Systems 8.4 Risk-Based Testing of ML Systems
9 Input Data Risks, Bias and Pipelines
9.1 Input Data Risks and Mitigations 9.2 Testing for Bias 9.3 Data Pipeline Testing
10 Representativeness, Constraints and Labels
10.1 Testing for Data Representativeness 10.2 Dataset Constraint Testing 10.3 Label Correctness Testing
11 Model Risks, Documentation and Performance Testing
11.1 ML Model Risks and Mitigations 11.2 ML Model Documentation and Review 11.3 Functional Performance Testing of Probabilistic MLS
12 Adversarial, Metamorphic and Drift Testing
12.1 Adversarial Testing 12.2 Metamorphic Testing 12.3 Drift Testing
13 Overfitting, A/B Testing and Back-to-Back Testing
13.1 Testing for Overfitting and Underfitting 13.2 A/B Testing for ML Systems 13.3 Back-to-Back Testing
14 ML Development and Deployment Testing
14.1 ML Development Risks and Mitigations 14.2 ML System Deployment Testing
3.3 Machine Learning Development Frameworks
Key Takeaways Explain This Topic AI
ML frameworks typically support data handling, model building, training and optimization, evaluation, and deployment to web, mobile, edge, or embedded targets.
Low-level APIs give more control and demand more skill; high-level APIs speed creation and reduce customization.
Frameworks may be general-purpose or specialized for domains such as image, speech, or translation.
Selection depends on application area, prototyping interface, configurability, user expertise, deployment constraints, community support, and ecosystem maturity.
Testers use framework evaluation tools but must remember those tools can hide defects if metric implementations are wrong.
Last updated: September 2026
What an ML development framework actually does
CT-AI v2.0 learning objective AI-1.1.7 (K2) asks you to summarize the functionality ML development frameworks provide. Think of a framework as a toolkit for building and training models, not as a guarantee of quality. Testers use the same toolkit the developers use. That is convenient and dangerous. Convenient, because evaluation helpers, data loaders, and export tools speed the work. Dangerous, because a green metric produced by the framework is still software, and software can implement a metric incorrectly.
Typical functionality falls into five groups. Learn the groups as a checklist you can apply to any toolkit, rather than as a brand list the exam will grade.
Data handling
Frameworks help load, preprocess, and manage the data used to train and test the model. That can include cleaning, formatting, and transforming examples into the tensors or tables the chosen model expects. From a testing view, every transform is a specification. If training images are resized, normalized, or augmented in the framework pipeline, inference must apply the compatible transform. A surprisingly large class of defects that look like model inaccuracy is actually a production path that skipped the training-time scaler.
Write those transforms down. If a loader drops corrupt rows in training but production accepts them, you have a silent distribution shift. If a tokenizer used in training is not the tokenizer bundled in the mobile build, you are not testing one system.
Model building
Frameworks offer libraries of ML algorithms and tools to design architecture: the type of model (neural network, decision tree, and so on), the number of layers and connections, and the mathematical operations inside the model. Testers should know which knobs were set, because those knobs are part of the configuration under test. Two runs that people describe as the same algorithm with different layer widths are different products. Architecture dumps, configuration files, and frozen graph exports belong in the evidence pack next to the metric.
Training and optimization
Frameworks provide algorithms that iteratively adjust internal parameters from training data toward a desired result, such as classification or ML regression. Some frameworks support distributed training across machines and can enhance or fine-tune pretrained models. Distributed training introduces extra failure modes: stale workers, mismatched seeds, and silent truncation of a shard. Fine-tuning a pretrained model introduces extra test obligations: you must know what the base model was, what data it must not memorize in your domain, and whether your fine-tune actually changed the behaviors you care about.
Optimization is not only faster loss. Early stopping, learning-rate schedules, and mixed precision are behaviors. If they are on in the trainer and off in a reproduction script, you will chase ghosts.
Evaluation
Frameworks offer tools to measure how the trained model performs on unseen data. Classification work commonly reports accuracy, precision, and recall. Regression work commonly reports error rates. These numbers are not self-proving. Testers should treat framework evaluation utilities as components under test. A precision function that averages the wrong way, ignores a class, or quietly drops rows with missing labels can hide defects. Independent checks (a confusion matrix from raw predictions, a second implementation, a hand-counted fixture of labeled cases) are how you keep the dashboard honest.
This is the point that separates a K2 summary from a slogan. Using the framework's evaluation tool is expected. Believing it without challenge is not. If the helper reports 97 percent accuracy while a minority class has zero recall, the defect may sit in the metric implementation rather than in some unexplained model mystery.
Deployment
Some frameworks can convert a trained model into a form that integrates with web applications, mobile devices, edge devices, or embedded systems. Export is another place defects hide. The notebook metric and the mobile runtime are different programs. If you only evaluate in the training framework, you have not evaluated the deployed artifact. Serialization flags, operator support on the target, and quantization at export time are all in scope for a tester who claims the shipped binary was checked.
- Function What the toolkit does Defect it can hide if you trust it blindly
- Data handling Load, clean, format, transform Train/serve transform mismatch
- Model building Algorithms and architecture wiring Unrecorded configuration drift
Training and optimization Parameter updates, distribution, fine-tuning Incomplete shards; unexpected fine-tune effects
- Evaluation Accuracy, precision, recall, or regression error Wrong metric math; dropped rows
- Deployment Export to web, mobile, edge, embedded Notebook score that the runtime cannot reproduce
- Abstraction level: low-level versus high-level APIs
The same family of tools can expose different abstraction levels. A lower-level API gives more control over model building and requires more coding expertise. A higher-level API simplifies creation and offers fewer customization options. Neither level is more correct. Low-level control can encode a subtle numeric bug that a high-level default would have avoided. High-level defaults can freeze a behavior you did not intend, such as a shuffle, a padding value, or an implicit preprocessing step.
Testers should ask which API layer produced the artifact under test and which defaults shipped with it. Rapid prototyping through a friendly high-level interface is a selection factor, not a substitute for evaluation on unseen data.
General-purpose versus domain-specific frameworks
Some frameworks are general purpose and support a wide range of application areas. Others specialize in image recognition, speech recognition, language translation, or similar domains. A domain-specific toolkit may get you to a working prototype faster because loaders, augmentations, and pretrained backbones already assume that domain. It may also smuggle in domain assumptions (color space, sample rate, tokenization) that are wrong for your product. General-purpose kits demand more wiring and give you fewer hidden assumptions. Domain kits hide wiring and hide assumptions. Testers should surface those assumptions as explicit checks.
How teams should choose, and how testers should read the choice
Selection can depend on:
- the application area
- the need for a user-friendly interface for rapid prototyping
- configurability for complex models
- the expertise of the users
- deployment considerations, including resource-constrained environments
- level of community support
- ecosystem maturity
Those factors are product-risk factors. A framework with a thin community may ship metric bugs that nobody has filed. A framework that is weak on edge export will tempt the team to test in a desktop language and ship through a converter, which is exactly the gap you should close with artifact-level tests.
Worked example: a team reports 97 percent accuracy from the framework's evaluation helper. A tester rebuilds a confusion matrix from saved predictions and finds that one minority class has zero recall. The helper was averaging in a way that hid the failure, or it had dropped the class's rows. The defect was in the evaluation implementation and in over-trusting the toolkit. Independent ExamFinch teaching treats that skepticism as core tester craft: use the framework, then verify the framework.