How to Deploy a Machine Learning Model on Kubernetes: Complete Step-by-Step Guide

Introduction: Why deploy an ML model on Kubernetes?

Deploying a machine learning model is more than training a model and saving a .pkl file. A useful model must be available through an API, respond reliably to users, restart when something fails, scale when traffic increases, and be updated without unnecessary downtime. Kubernetes provides the platform for managing those tasks in a repeatable way.

In this tutorial, we will build a small Python machine learning API with FastAPI, package it inside a Docker image, deploy it to Kubernetes, expose it through a Service, test predictions, scale the application, perform a rolling update, and review the important production practices. The example uses the Iris classification dataset so that the focus remains on deployment rather than on a complicated model.

Simple idea: the ML model makes the prediction, FastAPI provides the door to the model, Docker packages the application, and Kubernetes manages many running copies of that package.

What Kubernetes solves for machine learning applications

Imagine a restaurant. The trained model is the recipe, the FastAPI application is the chef who accepts an order, the Docker image is a sealed kitchen kit, and Kubernetes is the restaurant manager. The manager can start more kitchens during a rush, replace a kitchen that stops working, and gradually move customers from an old recipe to a new recipe. This is the practical value of Kubernetes for ML model serving.

• High availability: Kubernetes can run multiple replicas of the model API.

• Self-healing: failed containers are restarted automatically.

• Scaling: more replicas can be created when request traffic grows.

• Repeatable releases: the same Docker image can be deployed on a laptop, a server, or Azure Kubernetes Service.

• Controlled updates: rolling deployments reduce downtime when a new model version is released.

• Resource control: CPU and memory requests and limits protect the cluster from one application consuming everything.

Architecture used in this tutorial

The final request flow is straightforward. A client sends JSON features to the FastAPI endpoint. FastAPI validates the input and loads the trained model. Kubernetes keeps the required number of application Pods running. A Kubernetes Service gives the Pods a stable network name and distributes requests between them.

• Client or frontend: sends an HTTP POST request containing model features.

• Kubernetes Service: provides a stable address and load-balances traffic.

• Deployment: maintains the desired number of running Pods.

• FastAPI container: exposes /health and /predict endpoints.

• Model artifact: the trained model.joblib file loaded when the container starts.

Prerequisites

You can follow the first part on Windows, macOS, or Linux. For a beginner-friendly local setup, Docker Desktop with its Kubernetes cluster enabled is convenient. You can also use Minikube, kind, or an Azure Kubernetes Service cluster later.

• Python 3.10 or a newer supported Python version.

• Docker Desktop installed and running.

• kubectl installed and connected to a Kubernetes cluster.

• Basic knowledge of Python, REST APIs, and the command line.

• An Azure subscription only if you want to deploy the same image to AKS.

Check the tools before starting. If Docker Desktop Kubernetes is enabled, the kubectl context normally points to docker-desktop. You can confirm the active context with kubectl config current-context and verify the cluster with kubectl get nodes.

python --version
docker --version
kubectl version --client
kubectl config current-context
kubectl get nodes

Step 1: Create the machine learning project

Create a folder named ml-kubernetes-demo. The model should be trained separately from the prediction API. This is an important MLOps habit: training and serving are different responsibilities. The training script creates the model artifact once, while the API only loads that artifact and serves predictions.

ml-kubernetes-demo/
├── train_model.py
├── app.py
├── requirements.txt
├── model.joblib
├── Dockerfile
└── k8s/
    ├── deployment.yaml
    └── service.yaml

Create train_model.py

The following script trains a small Logistic Regression classifier with the built-in Iris dataset and saves it as model.joblib. In a real project, replace this dataset and algorithm with your own training pipeline.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import joblib

data = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    data.data,
    data.target,
    test_size=0.2,
    random_state=42,
    stratify=data.target,
)

model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)

predictions = model.predict(X_test)
accuracy = accuracy_score(y_test, predictions)
print(f"Validation accuracy: {accuracy:.2f}")

joblib.dump(model, "model.joblib")
print("Saved model.joblib")

Create requirements.txt

fastapi
uvicorn[standard]
scikit-learn
joblib
numpy

Train the model

Create a virtual environment if you prefer to keep the project dependencies isolated. Install the requirements and run the training script. The output file model.joblib must exist before you build the Docker image.

python -m venv .venv

# Windows PowerShell
.venv\Scripts\Activate.ps1

# macOS or Linux
source .venv/bin/activate

pip install -r requirements.txt
python train_model.py

Step 2: Build the FastAPI prediction service

Now create app.py. The API contains a health endpoint for Kubernetes probes and a prediction endpoint for clients. The Pydantic model validates the four Iris features before the model receives them. Returning a clear JSON response makes the service easy to connect to a website, mobile application, or another backend.

from fastapi import FastAPI
from pydantic import BaseModel
import joblib
import numpy as np

app = FastAPI(
    title="ML Prediction API",
    version="1.0.0",
    description="A machine learning model served with FastAPI",
)

model = joblib.load("model.joblib")


class IrisInput(BaseModel):
    sepal_length: float
    sepal_width: float
    petal_length: float
    petal_width: float


@app.get("/health")
def health():
    return {"status": "ok"}


@app.post("/predict")
def predict(payload: IrisInput):
    features = np.array([[
        payload.sepal_length,
        payload.sepal_width,
        payload.petal_length,
        payload.petal_width,
    ]])

    prediction = int(model.predict(features)[0])
    probability = float(np.max(model.predict_proba(features)))

    return {
        "prediction": prediction,
        "confidence": round(probability, 4),
    }

Test the API locally before Docker

Start FastAPI on your computer. Open http://127.0.0.1:8000/docs to see the interactive Swagger UI. First open the health endpoint. Then send sample Iris values to the prediction endpoint.

uvicorn app:app --reload --port 8000

# In another terminal
curl http://127.0.0.1:8000/health

curl -X POST http://127.0.0.1:8000/predict ^
  -H "Content-Type: application/json" ^
  -d "{\"sepal_length\":5.1,\"sepal_width\":3.5,\"petal_length\":1.4,\"petal_width\":0.2}"

On macOS or Linux, replace the caret line-continuation characters with backslashes. The expected health response is {"status":"ok"}. The prediction number identifies the Iris class learned by the model, and confidence is the model's estimated probability for its selected class.

Step 3: Dockerize the ML model API

Kubernetes runs containers, so the next step is to package the API and model artifact into a Docker image. Copying model.joblib into the image makes the container self-contained and easy to deploy. For large models, you may later use object storage, a model registry, or a dedicated model-serving platform instead of placing the artifact directly in the image.

Create the Dockerfile

FROM python:3.11-slim

WORKDIR /app

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .
COPY model.joblib .

EXPOSE 8000

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

The image uses a small Python base image, installs the dependencies, copies the application and model, and starts Uvicorn on port 8000. The host must be 0.0.0.0 inside a container; using 127.0.0.1 would make the service unreachable from outside the container.

Build and run the Docker image

docker build -t ml-api:v1 .

docker run --name ml-api-test -p 8000:8000 ml-api:v1

# Test in another terminal
curl http://127.0.0.1:8000/health

# Stop and remove the local test container
docker stop ml-api-test
docker rm ml-api-test

If the Docker container responds correctly, the same image is ready for Kubernetes. Before continuing, run docker images and confirm that ml-api:v1 exists.

Step 4: Prepare Kubernetes manifests

Kubernetes configuration is commonly stored as YAML. A Deployment describes how many copies of the application should exist and how each Pod should run. A Service provides a stable network endpoint. Keep these files in source control so every environment uses the same deployment definition.

Create a namespace

kubectl create namespace ml-demo

A namespace separates this tutorial application from other applications in the cluster. In a shared college, company, or cloud cluster, namespaces also make it easier to apply access control, quotas, and cleanup commands.

Create k8s/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-api
  namespace: ml-demo
  labels:
    app: ml-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ml-api
  template:
    metadata:
      labels:
        app: ml-api
    spec:
      containers:
        - name: ml-api
          image: ml-api:v1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8000
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 15
            periodSeconds: 20
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

The readiness probe tells Kubernetes whether a Pod is ready to receive traffic. The liveness probe tells Kubernetes whether the application is still healthy. Resource requests help the scheduler find a suitable node, while limits prevent one Pod from using unlimited resources. Two replicas are used so that one copy can continue serving while the other is restarted.

Create k8s/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: ml-api-service
  namespace: ml-demo
spec:
  type: NodePort
  selector:
    app: ml-api
  ports:
    - port: 80
      targetPort: 8000
      nodePort: 30080

The selector is the connection between the Service and the Deployment. It sends traffic to Pods carrying the label app: ml-api. The Service listens on port 80 and forwards requests to port 8000 inside the container. NodePort is convenient for local learning. For a cloud cluster, you can use type LoadBalancer or an Ingress controller with HTTPS.

Step 5: Deploy the ML model to Kubernetes

Docker Desktop Kubernetes can use locally built images. If you are using Minikube, build the image inside Minikube's Docker environment or load the image with minikube image load ml-api:v1. In a cloud cluster, push the image to a private registry such as Azure Container Registry before applying the Deployment.

kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

kubectl get deployments -n ml-demo
kubectl get pods -n ml-demo
kubectl get services -n ml-demo

Wait until both Pods show Running and READY 1/1. The rollout status command gives a clearer answer than looking at a single moment in the Pods table.

kubectl rollout status deployment/ml-api -n ml-demo
kubectl describe deployment ml-api -n ml-demo
kubectl describe pods -n ml-demo

Step 6: Access and test the Kubernetes Service

For a simple and reliable local test, use port-forwarding. It creates a temporary connection from your computer to the Kubernetes Service. This avoids guessing the Docker Desktop NodePort URL and is useful for debugging.

kubectl port-forward service/ml-api-service 8080:80 -n ml-demo

# In another terminal
curl http://127.0.0.1:8080/health

curl -X POST http://127.0.0.1:8080/predict ^
  -H "Content-Type: application/json" ^
  -d "{\"sepal_length\":5.1,\"sepal_width\":3.5,\"petal_length\":1.4,\"petal_width\":0.2}"

You can also open http://127.0.0.1:8080/docs in a browser. The request is now passing through Kubernetes rather than directly to the local Python process. To see which Pod handled traffic, add a hostname or Pod name to the API response in a later version.

Step 7: Scale the model API

Machine learning inference can be CPU-intensive, especially when several users upload images or send requests at the same time. Kubernetes lets you increase the number of replicas without changing the application code.

kubectl scale deployment ml-api --replicas=4 -n ml-demo
kubectl get pods -n ml-demo
kubectl get deployment ml-api -n ml-demo

The Service automatically distributes requests across the ready Pods. Scaling replicas does not make one prediction faster; it increases the number of requests the application can handle concurrently. For GPU models, the cluster must also contain suitable GPU nodes and the Deployment must request GPU resources.

Configure automatic scaling with HPA

A Horizontal Pod Autoscaler can change the replica count based on CPU or memory metrics. The cluster must have a working metrics server, and the Deployment must define resource requests. Start with conservative limits and observe real usage before choosing production thresholds.

kubectl autoscale deployment ml-api ^
  --cpu-percent=60 ^
  --min=2 ^
  --max=10 ^
  -n ml-demo

kubectl get hpa -n ml-demo
kubectl describe hpa ml-api -n ml-demo

Autoscaling is not a replacement for performance testing. A model may be limited by memory, GPU availability, database latency, or request queue length rather than CPU. Measure response time, throughput, error rate, and resource usage together.

Step 8: Release a new model version safely

A new trained model should receive its own image tag. Do not overwrite v1 and expect Kubernetes to detect the change. Immutable version tags make it possible to identify exactly which model is running and to roll back when the new version behaves unexpectedly.

# Build a new image after replacing model.joblib
docker build -t ml-api:v2 .

# Change image: ml-api:v1 to image: ml-api:v2
# Then apply the updated Deployment
kubectl apply -f k8s/deployment.yaml

kubectl rollout status deployment/ml-api -n ml-demo
kubectl rollout history deployment/ml-api -n ml-demo

Kubernetes normally performs a rolling update. It creates new Pods, waits for the readiness probe, and then removes old Pods. If the new release is not healthy, return to the previous revision.

kubectl rollout undo deployment/ml-api -n ml-demo
kubectl rollout status deployment/ml-api -n ml-demo

Step 9: Deploy the same model to Azure Kubernetes Service

Once the local deployment works, the cloud version uses the same container and Kubernetes YAML with a few changes. Azure Container Registry stores the image, and Azure Kubernetes Service runs the cluster. The key rule is that AKS nodes must be able to pull the image from the registry.

1. Create an Azure Container Registry and an AKS cluster in the Azure portal or with the Azure CLI.

2. Build and push a versioned image such as myregistry.azurecr.io/ml-api:v1.

3. Allow AKS to pull from the registry by attaching the registry or configuring a suitable pull identity.

4. Change the Deployment image field from ml-api:v1 to the full registry image name.

5. Change the Service type from NodePort to LoadBalancer for a basic public endpoint, or use an Ingress controller for a production HTTPS endpoint.

6. Apply the manifests and wait for the external IP address to appear.

docker tag ml-api:v1 myregistry.azurecr.io/ml-api:v1
docker push myregistry.azurecr.io/ml-api:v1

kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

kubectl get service ml-api-service -n ml-demo
kubectl get pods -n ml-demo

A public LoadBalancer is useful for learning, but do not expose an unprotected prediction API containing student, customer, or medical data. Use HTTPS, authentication, rate limiting, network controls, and a proper domain before real users access it.

Production improvements for an ML Kubernetes deployment

1. Keep model versions traceable

Store the model version, training date, dataset version, feature order, and evaluation metrics with every release. Include the model version in the API response or an /info endpoint. This makes it possible to explain which model produced a prediction.

2. Do not store important files inside a temporary container

Container filesystems are disposable. Store uploaded images, reports, logs, and user data in a managed database or object storage. Use persistent volumes only when the application genuinely needs filesystem access, and define backups and restore tests.

3. Protect secrets

Never place passwords, API keys, cloud credentials, or database connection strings in a Dockerfile or Git repository. Use Kubernetes Secrets, an external secret manager, or Azure Key Vault. Apply least privilege so the Pod receives only the permissions it needs.

4. Add security to the container

Run the application as a non-root user, use a small and patched base image, scan images for vulnerabilities, pin dependency versions, and rebuild regularly. A production Dockerfile can create an unprivileged user and switch to it with USER appuser.

5. Monitor more than Pod health

A Pod can be Running while the model is producing poor predictions. Monitor request count, latency, HTTP errors, CPU, memory, queue time, model confidence, data drift, and prediction distribution. Alert separately on infrastructure failures and model-quality failures.

6. Use a proper inference pattern for large models

For large language models, computer vision models, or GPU workloads, consider specialized serving systems such as KServe, NVIDIA Triton Inference Server, TorchServe, or a managed cloud endpoint. FastAPI is excellent for learning and for many lightweight models, but every workload should be tested against its actual latency and memory requirements.

Common errors and how to fix them

• ImagePullBackOff: Kubernetes cannot find or download the image. Check the image name, registry permissions, pull secret, and whether the image tag really exists.

• CrashLoopBackOff: the container starts and exits repeatedly. Run kubectl logs deployment/ml-api -n ml-demo and check missing files, dependency errors, or an incorrect Uvicorn command.

• Pods remain Pending: the cluster does not have enough CPU or memory, or a node selector or GPU requirement cannot be satisfied. Run kubectl describe pod and read the Events section.

• Service returns connection refused: confirm that the container listens on 0.0.0.0:8000 and that targetPort 8000 matches the container port.

• Readiness probe fails: open the /health endpoint inside the container and verify the model loads successfully before the probe starts.

• Predictions are wrong: check feature order, preprocessing, model version, input units, and whether the serving code uses exactly the same transformations as training.

Useful Kubernetes commands for daily work

kubectl get all -n ml-demo
kubectl get pods -o wide -n ml-demo
kubectl logs deployment/ml-api -n ml-demo
kubectl logs -f deployment/ml-api -n ml-demo
kubectl describe pod <pod-name> -n ml-demo
kubectl exec -it <pod-name> -n ml-demo -- sh
kubectl get events --sort-by=.lastTimestamp -n ml-demo
kubectl delete namespace ml-demo

Use kubectl delete namespace ml-demo only when you intentionally want to remove the complete tutorial application. In a production environment, confirm the namespace and backup requirements before deleting resources.

Deployment checklist

• The model is trained and model.joblib is created.

• The FastAPI /health endpoint responds successfully.

• The Docker image builds and runs locally.

• Kubernetes Deployment and Service YAML files are stored in source control.

• Pods are Ready and readiness and liveness probes pass.

• The prediction endpoint works through the Kubernetes Service.

• Resource requests and limits are defined.

• The image uses a unique version tag for every model release.

• Secrets are outside the Docker image and Git repository.

• Logs, backups, monitoring, and rollback procedures are documented.

• The public API uses HTTPS, authentication, and rate limiting before real data is used.

Conclusion

The complete path is: train the model, save the model artifact, expose it through FastAPI, build a Docker image, create a Kubernetes Deployment, expose it with a Service, test it, scale it, and release new versions with rolling updates. Kubernetes does not improve a model's accuracy by itself; it makes the serving system more reliable, repeatable, and easier to operate.

If you are learning DevOps, cloud computing, or MLOps, start with this small Iris example and then replace it with a model from your own project. The same deployment ideas apply to student projects, business APIs, computer vision services, and many real-world machine learning systems. For official reference material, use the Kubernetes documentation at https://kubernetes.io/docs/, FastAPI documentation at https://fastapi.tiangolo.com/, and Azure Kubernetes Service documentation at https://learn.microsoft.com/azure/aks/.

This tutorial is published by NeoAI Tech for developers, students, educators, and technology teams who want a practical path from a trained machine learning model to a manageable Kubernetes service.