Documentation Index Fetch the complete documentation index at: https://mintlify.com/zenml-io/zenml/llms.txt
Use this file to discover all available pages before exploring further.
A ZenML stack is a collection of stack components that define the infrastructure for your ML pipelines. Understanding how to configure stacks is essential for running pipelines in different environments, from local development to production.
Understanding Stacks A stack consists of:
Orchestrator (required): Manages pipeline executionArtifact Store (required): Stores pipeline artifactsContainer Registry (optional): Stores Docker imagesStep Operator (optional): Runs specific steps on different infrastructureExperiment Tracker (optional): Tracks experiments and metricsModel Deployer (optional): Deploys trained modelsAnd more : Alerters, annotators, data validators, etc.
Viewing Your Current Stack
Step 1: Check Active Stack
See which stack is currently active:
Stack name
All components and their configuration
Stack ID
View all available stacks:
Step 3: Get Stack Details
View details of a specific stack:
zenml stack describe my_stack Creating Your First Custom Stack
Step 1: Register an Artifact Store
Start by registering an artifact store:
# Local artifact store zenml artifact-store register local_store \ --flavor=local \ --path=/path/to/artifacts # S3 artifact store zenml artifact-store register s3_store \ --flavor=s3 \ --path=s3://my-bucket/zenml-artifacts # GCS artifact store zenml artifact-store register gcs_store \ --flavor=gcp \ --path=gs://my-bucket/zenml-artifacts # Azure Blob Storage zenml artifact-store register azure_store \ --flavor=azure \ --path=az://my-container/zenml-artifacts
Step 2: Register an Orchestrator
Add an orchestrator to manage pipeline execution:
# Local orchestrator (default) zenml orchestrator register local_orchestrator \ --flavor=local # Kubernetes orchestrator zenml orchestrator register k8s_orchestrator \ --flavor=kubernetes \ --kubernetes_context=my-k8s-cluster # Airflow orchestrator zenml orchestrator register airflow_orchestrator \ --flavor=airflow \ --airflow_home=/path/to/airflow # Kubeflow orchestrator zenml orchestrator register kubeflow_orchestrator \ --flavor=kubeflow \ --kubeflow_hostname=https://kubeflow.example.com
Combine components into a stack:
zenml stack register my_custom_stack \ --orchestrator=local_orchestrator \ --artifact-store=s3_store
Step 4: Activate the Stack
Switch to your new stack:
zenml stack set my_custom_stack
Confirm the stack is active and properly configured:
Common Stack Configurations Local Development Stack Perfect for local development and testing:
# Register components zenml artifact-store register local_artifact_store \ --flavor=local \ --path=./artifacts zenml orchestrator register local_orchestrator \ --flavor=local # Create stack zenml stack register local_dev_stack \ --orchestrator=local_orchestrator \ --artifact-store=local_artifact_store # Activate zenml stack set local_dev_stack
Cloud Production Stack Production stack with cloud infrastructure:
# Register S3 artifact store zenml artifact-store register production_artifacts \ --flavor=s3 \ --path=s3://my-prod-bucket/artifacts # Register Kubernetes orchestrator zenml orchestrator register k8s_prod_orchestrator \ --flavor=kubernetes \ --kubernetes_context=production-cluster \ --kubernetes_namespace=zenml # Register container registry zenml container-registry register ecr_registry \ --flavor=aws \ --uri=123456789.dkr.ecr.us-east-1.amazonaws.com # Create production stack zenml stack register production_stack \ --orchestrator=k8s_prod_orchestrator \ --artifact-store=production_artifacts \ --container-registry=ecr_registry
Experiment Tracking Stack Stack with MLflow for experiment tracking:
# Register MLflow experiment tracker zenml experiment-tracker register mlflow_tracker \ --flavor=mlflow \ --tracking_uri=http://localhost:5000 # Create or update stack with experiment tracker zenml stack register experiment_stack \ --orchestrator=local_orchestrator \ --artifact-store=local_artifact_store \ --experiment-tracker=mlflow_tracker
Complete ML Stack Full-featured stack for end-to-end ML:
# Artifact Store zenml artifact-store register s3_artifacts \ --flavor=s3 \ --path=s3://mlops-bucket/artifacts # Orchestrator zenml orchestrator register k8s_orch \ --flavor=kubernetes \ --kubernetes_context=ml-cluster # Container Registry zenml container-registry register dockerhub_registry \ --flavor=dockerhub \ --uri=docker.io/mycompany # Experiment Tracker zenml experiment-tracker register mlflow \ --flavor=mlflow \ --tracking_uri=http://mlflow.example.com # Model Registry zenml model-registry register mlflow_registry \ --flavor=mlflow \ --registry_uri=http://mlflow.example.com # Model Deployer zenml model-deployer register seldon \ --flavor=seldon \ --kubernetes_context=ml-cluster # Alerter zenml alerter register slack_alerts \ --flavor=slack \ --slack_token= $SLACK_TOKEN \ --default_slack_channel_id=C1234567890 # Create comprehensive stack zenml stack register ml_production_stack \ --orchestrator=k8s_orch \ --artifact-store=s3_artifacts \ --container-registry=dockerhub_registry \ --experiment-tracker=mlflow \ --model-registry=mlflow_registry \ --model-deployer=seldon \ --alerter=slack_alerts
Managing Stack Components List Components View all registered components of a type:
# List all artifact stores zenml artifact-store list # List all orchestrators zenml orchestrator list # List all experiment trackers zenml experiment-tracker list
Describe Components Get details about a specific component:
zenml artifact-store describe s3_store
Update Components Modify component configuration:
zenml artifact-store update s3_store \ --path=s3://new-bucket/artifacts
Delete Components Remove a component (must not be in use):
zenml artifact-store delete old_store
Stack Validation ZenML validates stacks before use:
# Validate current stack zenml stack describe # The output shows any validation errors
Common validation issues:
Missing required components
Incompatible component combinations
Unreachable infrastructure
Missing credentials
Using Stacks in Code Access Current Stack Access stack components in your steps:
from zenml import step, get_step_context @step def stack_aware_step () -> dict : """Access stack components at runtime.""" context = get_step_context() stack = context.stack print ( f "Stack name: { stack.name } " ) print ( f "Orchestrator: { stack.orchestrator.name } " ) print ( f "Artifact store: { stack.artifact_store.name } " ) # Check for optional components if stack.experiment_tracker: print ( f "Experiment tracker: { stack.experiment_tracker.name } " ) return { "stack" : stack.name}
Programmatic Stack Access Use the Client to work with stacks:
from zenml.client import Client client = Client() # Get active stack active_stack = client.active_stack print ( f "Active stack: { active_stack.name } " ) # Get specific stack stack = client.get_stack( "production_stack" ) # List all stacks all_stacks = client.list_stacks() for stack in all_stacks: print ( f "Stack: { stack.name } " ) # Get stack components artifact_store = active_stack.artifact_store orchestrator = active_stack.orchestrator
Stack Recipes ZenML provides stack recipes for quick setup:
# Install mlstacks (if not already installed) pip install mlstacks # List available recipes zenml stack recipe list # Deploy a stack recipe zenml stack recipe deploy < recipe_nam e > # For example, deploy a GCP stack zenml stack recipe deploy gcp-minimal
Environment-Specific Stacks Manage stacks for different environments:
# Development stack zenml stack register dev_stack \ --orchestrator=local_orchestrator \ --artifact-store=local_artifacts # Staging stack zenml stack register staging_stack \ --orchestrator=k8s_staging \ --artifact-store=s3_staging # Production stack zenml stack register prod_stack \ --orchestrator=k8s_prod \ --artifact-store=s3_prod # Switch between environments zenml stack set dev_stack # For development zenml stack set staging_stack # For testing zenml stack set prod_stack # For production
Stack Sharing Share stacks across team members:
# Share a stack (requires ZenML server) zenml stack share production_stack # List shared stacks zenml stack list # Copy a shared stack zenml stack copy production_stack --name my_prod_stack
Best Practices
Use Descriptive Names Name stacks and components clearly (e.g., prod_stack, dev_stack)
Separate Environments Maintain separate stacks for dev, staging, and production
Version Infrastructure Track stack configurations in version control
Validate Before Use Always validate stacks after configuration changes
Use Stack Recipes Leverage pre-built recipes for common configurations
Document Dependencies Document any special setup or credentials needed for stacks
Stack Configuration Files Export and import stack configurations:
# Export stack configuration zenml stack export production_stack --filename prod_stack.yaml # Import stack configuration zenml stack import prod_stack.yaml
Example stack configuration file:
stack_name : production_stack components : orchestrator : name : k8s_orchestrator flavor : kubernetes configuration : kubernetes_context : production-cluster kubernetes_namespace : zenml artifact_store : name : s3_artifacts flavor : s3 configuration : path : s3://prod-bucket/artifacts experiment_tracker : name : mlflow_tracker flavor : mlflow configuration : tracking_uri : http://mlflow.prod.example.com
Troubleshooting Stack Component Not Found # List all components to find the right name zenml artifact-store list zenml orchestrator list
Connection Issues # Test artifact store connectivity zenml artifact-store describe my_store # Check credentials zenml stack describe
Stack Validation Fails # Get detailed error message zenml stack describe # Verify each component individually zenml artifact-store describe my_artifact_store zenml orchestrator describe my_orchestrator
Next Steps
Stack Switching Learn how to switch between stacks efficiently
Creating Pipelines Build pipelines that run on your configured stacks
Deploying Pipelines Deploy pipelines to production stacks
