MLFlow

MLflow is a platform to streamline machine learning development, including tracking experiments, packaging code into reproducible runs, and sharing and deploying models.

First Start

InitContainer is not needed on first start; it will cause the deployment to fail because it cannot find tables to update!

Tracking Server

MLflow provides for diverse tracking server configurations; among them are:

• MLflow as remote Tracking Server, providing tracking backend and proxied access to artifact stores
• MLflow as Artifact Server only, providing proxied access to artifacts but no tracking
• MLflow Tracking Server only, and requiring direct access to the artifact store. In this configuration, the user must manage their direct connection to the artifact store

MLflow uses two components for storage: backend store and artifact store. The backend store persists MLflow entities (runs, parameters, metrics, tags, notes, metadata, etc), and these data can be recorded to local files, to a SQLAlchemy compatible database, or remotely to a tracking server. The artifact store persists artifacts (files, models, images, in-memory objects, or model summary, etc) to local files or a variety of remote file storage solutions.

IMPORTANT See notes in MLFlow docs regarding the differences between options 4 and 5 with respect to user authorization / access permissions

Cleaning up deleted experiments

• Run mlflow gc in the mlflow container to clean up deleted runs and artifacts (this retains the experiment). The python script cleanup-runs.py may also be used to clean up runs from the database (this may orphan the artifacts). See more: [BUG] gc fails with postgres backend, violates foreign key constraint · Issue #13254 · mlflow/mlflow
• Run the python script cleanup-experiments.py to fully delete experiments from the database

Deleting traces

The repo includes a just module for MLflow trace deletion using MlflowClient.delete_traces():

# delete traces older than 7 days
just mlflow delete-traces-older 1 7

# delete explicit trace IDs (comma-separated)
just mlflow delete-traces-ids 1 trace_id_1,trace_id_2

# delete trace IDs from a file (one ID per line)
just mlflow delete-traces-file 1 ./trace_ids.txt

Use tracking_uri when deleting against a non-default endpoint:

just mlflow delete-traces-older 1 7 https://mlflow.example.com

AI Gateway

As of MLflow v3.9, the MLflow AI Gateway is now a part of the tracking server and does not require independent deployment.

The standalone Gateway Server mode (mlflow gateway start --config-path ...) is still available as a legacy path, but is not used in this deployment.

Source: kubernetes/apps/datasci/mlflow/README.md