Alauda AI
Deploy
Inference Service

Architecture, Version and Components

Validated Models

Qwen3.6-27B (W8A8)

DeepSeek-V4-Flash (W4A8)

DeepSeek-V4-Flash (W8A8)

MiniMax-M2.5 (W8A8)

About Alauda Build of Hami

About Alauda Build of NVIDIA GPU Device Plugin

Pre-installation Configuration

Install Workbench

Configure Alauda AI Entry and Branding

Install Alauda AI

Tools Menu Configuration

Upgrade Alauda AI

Migrating to Knative Operator

Upgrade Workbench

Hardware Profile

Hardware Profile Management

Create Hardware Profile using CLI

Schedule Workloads to Specific GPU Nodes

Creating CPU-Only and GPU-Accelerated Profiles

Namespace Management

MLflow Workspaces and Access Control

Install the Secure-Profile Dependencies

Enable the Secure Profile

Create Workbench

Kubeflow Notebooks

Use Kubeflow Notebooks

Use Kubeflow Volumes

Use Kubeflow Tensorboards

Using Connections

Create a Data Science Pipelines Application

Use Kubeflow Pipelines

Run Kubeflow Pipelines from JupyterLab with Elyra

Use Reusable Kubeflow Pipeline Components

Kubeflow Pipelines Execution and Storage Behavior

Schedule Alauda DevOps Pipelines with Kueue

Distributed Workloads

CodeFlare SDK Tutorial

Run a Spark Application

Use Kubeflow Model Registry

Experiment Tracking

Using the MLflow Python SDK with Authentication and RBAC

Kubeflow Pipeline + MLflow Integration

Trace AI agents with MLflow

Use Coding Agents with On-Premise Inference Services

Run MLOps with Coding Agents and On-Premise LLMs

Alauda support for Kubeflow

Install Kubeflow Operators

Upgrade Kubeflow Operators

Data Science Pipelines

Data Science Pipelines Operator

Alauda Build of KubeRay Operator

Alauda Build of Spark Operator

Install Label Studio

Alauda Build of Feast

Training Guides

Kubeflow Trainer Quick Start

Checkpointing and Resuming TrainJobs

Preemptible TrainJobs with Kueue, Checkpointing, and Inference Coexistence

GPU Slicing with Dynamic Resource Allocation (DRA)

Training Runtime Images

Fine-tuning LLMs with Training Hub

Daily Fine-Tuning Pipeline with MLflow Tracking and TrustyAI Evaluation

Fine-tuning LLMs using Workbench

Fine-tune and Pretrain LLMs on Ascend NPU

Quota & Scheduling

Configuring quotas

Configuring fair sharing

Gang scheduling

Manage Ascend NPU quota with Kueue

Alauda Build of Kueue

Alauda support for Volcano

Install Volcano

Alauda Build of JobSet

Model Management

Model Repository

Upload Models Using Notebook

Inference Service

Managing Inference Services

Create Inference Service using CLI

Deploy Inference Services from the Kubeflow Dashboard

Extend Inference Runtimes

Using KServe Modelcar for Model Storage

Configure External Access for Inference Services

Configure Scaling for Inference Services

Set Up Autoscaling for Inference Services with KEDA

Scheduling Inference Services based on the CUDA version

Schedule Inference Services with Kueue

Enable Expert Parallel for vLLM Inference Services

Speculative Decoding for vLLM Inference Services

Troubleshooting

Experiencing Inference Service Timeouts with MLServer Runtime

Inference Service Fails to Enter Running State

Model as a Service (MaaS)

Inference Gateway

Authenticating Consumers

Configuring Token Quotas

Metering Token Usage

Routing to LLM Providers

Charging Back Token Usage

LLM Compressor with Alauda AI

Alauda Build of KServe

Envoy AI Gateway

Alauda Build of Envoy AI Gateway

Install Envoy AI Gateway

LeaderWorkerSet

Alauda Build of LeaderWorkerSet

Install LeaderWorkerSet

InferNex Bridge

Alauda Build of InferNex Bridge

Install InferNex Bridge

Build AI Applications

Quick Start with Core Profile

Demo with Secure Profile

Demo Advanced: AuthBridge on the Tool with Token Exchange

Kagenti with two MCP servers behind Envoy AI Gateway

Alauda Build of Llama Stack

Install Llama Stack

Kagenti Operator

Security Architecture

MCP Lifecycle Operator

Alauda Build of MCP Lifecycle Operator

Evaluate & Safety

Evaluating RAG with Ragas

AI Guardrails for LLM safety

NeMo Guardrails

Alauda Build of TrustyAI

Install TrustyAI

Deploy TrustyAI Service

Logging & Tracing

Resource Monitoring

Monitoring Metrics and Views

Add a Monitoring Dashboard

Monitoring pending workloads

Monitor Dashboard Stuck at Loading

Bias and Drift Monitoring

Kubernetes APIs

Inference Service APIs

ClusterServingRuntime [serving.kserve.io/v1alpha1]

InferenceService [serving.kserve.io/v1beta1]

Workspace Kind [kubeflow.org/v1beta1]

Workspace [kubeflow.org/v1beta1]

AmlNamespace [manage.aml.dev/v1alpha1]

AmlCluster [amlclusters.aml.dev/v1alpha1]

Inference Service

Introduction

Managing Inference Services

Guides

Troubleshooting

Previous pageShare Models Next pageIntroduction