Thursday, September 17, 2026

ML Platform — Architecture Field Notes
Engineering Field Notes · No. 01
Machine Learning Infrastructure
Architecture Analysis / Repository Guide

ML Platform

From “it works on my machine” to “the cluster has opinions.”

A practical analysis of evolving a Machine Learning platform from a lightweight Proof-of-Concept into a resilient, automated and production-oriented architecture.

System ML Platform
Pattern MLOps / GitOps
Deployment Kubernetes
01

The Problem

Machine Learning systems rarely fail because somebody forgot how to train a model.

They usually become difficult when the model leaves the notebook and enters the real world: multiple services, multiple versions, multiple environments, automated deployments, security checks, monitoring, scaling and—eventually—someone asking, “Who deployed this?”

The purpose of this architecture is therefore not simply to deploy an ML model. It is to establish a repeatable path from data ingestion and training through validation, packaging, deployment and inference.

POC Architecture

Optimized for experimentation, fast iteration and lightweight validation. A single-node Kubernetes environment is sufficient for proving that the system works.

Production Architecture

Optimized for availability, automated delivery, security validation, horizontal scaling and observable model-serving workloads.

02

POC vs Production

The architectural difference is less about replacing one technology with another and more about introducing operational guarantees.

Dimension
POC
Production
Infrastructure
Single-node Kubernetes / AKS
Multi-node AKS with dedicated worker capacity
Storage
Local or single-instance MinIO
Enterprise data lake using Parquet / Delta Lake
CI/CD
Build, test, train and registry push
Jenkins with security, dependency, performance and model-quality gates
Deployment
Manual or scripted kubectl
Declarative GitOps through Argo CD
Serving
Static model-serving pod
Horizontal Pod Autoscaler with multiple replicas
Observability
MLflow logs and metrics
MLflow + Prometheus + Grafana
03

POC Architecture

The Proof-of-Concept architecture focuses on one important objective: prove the complete ML lifecycle without building an entire cloud empire first.

Figure 01 · POC Machine Learning Platform
graph TD subgraph DP["DATA / ML PIPELINE"] A["Source Systems"] B["Kafka"] C["Staging DB / Raw Zone"] D["Airflow"] E["Data Processing / Cleaning"] F["Data Validation"] G[("S3 / MinIO")] A -->|Ingestion| B B --> C C --> D D --> E E --> F F --> G H["Model Training"] I["MLflow Tracking"] J["Model Evaluation"] K["Model Registry"] G --> H H --> I I --> J J --> K end subgraph CICD["CI / CD"] L["Developer"] M["Git"] N["Jenkins"] O["Tests / Security Scan"] P["Build Container"] Q["GHCR"] L -->|Commit| M M --> N N --> O O --> P P --> Q end subgraph SERVING["MODEL SERVING"] R["Kubernetes / K3s"] S["FastAPI ML Serving"] T["Model Inference"] Q -->|Deploy Image| R R --> S S --> T end K -->|Promoted Model| N
The POC keeps the moving parts deliberately small: enough infrastructure to validate the lifecycle, but not enough to require a dedicated operations department.
04

Production Direction

Once the POC proves the workflow, production architecture introduces controls around reliability, security and repeatability.

Engineering Principle

The production system should make the correct deployment path the easiest deployment path.

That means automated tests, security scans, artifact promotion, declarative infrastructure and Git-based deployment state.

Instead of manually applying Kubernetes manifests, the desired state is stored in Git and reconciled by Argo CD. Instead of assuming a model is healthy because its container is running, the system introduces application and model-level observability.

05

Repository Setup

The following workflow runs the ML serving project locally using Kubernetes or K3s.

01
Clone
Obtain the repository and enter the project directory.
02
Services
Start K3s/Kubernetes, MLflow, MinIO where required, Jenkins and Git/GHCR access.
03
Namespace
Create the dedicated ml-serving namespace.
04
Registry
Configure Kubernetes authentication for GitHub Container Registry.
05
Deploy
Validate the Kubernetes manifest and deploy the serving workload.
06
Observe
Check deployment status, pods and application logs.
07
Verify
Port-forward the service and test both health and prediction APIs.
06

The Terminal

A production architecture is impressive. A working one is better. The following is the deliberately unglamorous part: commands.

# Clone the repository

git clone https://github.com/Mainak23/project1o1.git
cd main


# Check Kubernetes

sudo systemctl status k3s

kubectl get nodes


# Create namespace

kubectl create namespace ml-serving


# Configure GHCR

kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=<GITHUB_USERNAME> \
  --docker-password='<GITHUB_TOKEN>' \
  -n ml-serving


# Validate deployment

kubectl apply \
  --dry-run=client \
  -f model-deployment.yaml


# Deploy

kubectl apply \
  -f model-deployment.yaml


# Inspect workload

kubectl get deployment -n ml-serving

kubectl get pods -n ml-serving


# Check rollout

kubectl rollout status \
  deployment/ml-serving2 \
  -n ml-serving
Field Note

Replace the container image version in model-deployment.yaml with the intended GHCR image version before deployment.

07

Inspect Before You Panic

Kubernetes gives us several useful places to look before declaring the entire platform deceased.

# Deployment

kubectl get deployment \
  -n ml-serving


# Pods

kubectl get pods \
  -n ml-serving


# Logs

kubectl logs deployment/ml-serving2 \
  -n ml-serving \
  --tail=100


# Follow logs

kubectl logs deployment/ml-serving2 \
  -n ml-serving \
  -f

The expected pod state is straightforward:

1/1 Running

This is one of the few places in engineering where “Running” is a genuinely pleasant thing to see.

08

Test the API

Forward the Kubernetes service to the local machine.

kubectl port-forward \
  svc/ml-serving2 \
  8081:8080 \
  -n ml-serving

Health check:

curl http://127.0.0.1:8081/health
{"status":"healthy"}

Once the service responds, the prediction endpoint can be tested using the expected 30-feature input vector.

curl -X POST http://127.0.0.1:8081/predict \
-H "Content-Type: application/json" \
-d '{
  "data": [[
    17.99,10.38,122.8,1001.0,0.1184,
    0.2776,0.3001,0.1471,0.2419,0.07871,
    1.095,0.9053,8.589,153.4,0.006399,
    0.04904,0.05373,0.01587,0.03003,0.006193,
    25.38,17.33,184.6,2019.0,0.1622,
    0.6656,0.7119,0.2654,0.4601,0.1189
  ]]
}'

Expected response:

{"prediction":[0]}
09

Production Quality Gates

The next architectural step is to move from “Can we deploy this?” toward “Should this be allowed to deploy?”

Trivy
Container and dependency vulnerability scanning.
SonarQube
Static code analysis and maintainability checks.
Dependency Checks
Detect vulnerable or outdated software dependencies.
Performance Tests
Verify inference latency and throughput characteristics.
Model Validation
Confirm that a newly trained model meets defined quality thresholds.
Model Drift
Detect degradation between training data assumptions and production behavior.
10

Verification Checklist

[✓] Repository cloned
[✓] K3s / Kubernetes running
[✓] Kubernetes node is Ready
[✓] MLflow accessible
[✓] Container image built
[✓] Image pushed to GHCR
[✓] GHCR secret created
[✓] Kubernetes YAML validated
[✓] Deployment created
[✓] Pod is 1/1 Running
[✓] Application logs inspected
[✓] Port forwarding enabled
[✓] /health returns healthy
[✓] /predict returns a prediction
“The goal of production MLOps is not to make the architecture complicated. It is to make the complicated parts predictable.” Engineering Note · ML Platform
Repository
github.com/Mainak23/project1o1
ML Platform Field Notes Built with Kubernetes, coffee & questionable optimism.

No comments:

Post a Comment