If you’re interested in running local/private LLMs (leveraging Ollama and similar solutions) on Kubernetes, take a look at this new project.

KubeAI serves an OpenAI-compatible HTTP API in Kubernetes, providing you with a drop-in OpenAI replacement and simplifying the needed operations. The project’s authors call it “a Model Operator that manages vLLM and Ollama servers [inside Kubernetes].” Most noticeable KubeAI features include:

- Support for various Open Source model servers, including vLLM, Ollama, FasterWhisper, and Infinity. (Speech-to-Text and Text-Embedding are supported.)
- An option to preload LLMs in custom container images.
- Autoscaling based on load.
- A Chat UI based on OpenWebUI.
- An ability to work in the CPU-only mode and with GPUs. TPU support is planned.
- No dependencies (such as service meshes) and installable in regular K8s clusters, OpenShift, and managed K8s solutions (currently, there’s an instruction for GKE with Autopilot).

▶️ GitHub repo

#tools #aiml

We’ve been witnessing the emergence of more AI assistants lately, including those related to DevOps. Here’s a new tool focusing on troubleshooting Kubernetes.

Klama was created to help troubleshoot DevOps-related issues by interpreting user queries and suggesting and executing relevant commands. Currently, it comes with very basic features:

- Interpreting queries, suggesting and executing relevant commands for better investigation.
- Compatibility with any server implementing OpenAI API. It was tested with OpenAI models, self-hosted models using vLLM, Amazon Bedrock models, and Azure AI.
- Session price calculation based on specified prices per 1K input/output tokens.

▶️ GitHub repo
📢 Reddit announcement

#tools #aiml

Managing GenAI workloads on Kubernetes is surely gaining momentum. If it piques your interest as well, consider this new Open Source project.

LLMariner is an extensible platform for hosting and managing LLMs on K8s. It consists of a control plane and worker planes, which can be run in a single or across multiple Kubernetes clusters. Some of the project’s highlights are:

- Support for various inference runtimes, including vLLM, Ollama, and Triton.
- Support for numerous models (Llama 3.1, Gemma, TinyLlama, DeepSeek Coder, Mistral, and more), as well as other models via HuggingFace.
- Works with Retrieval-Augmented Generation (RAG).
- Allows to fine-tune models, run general-purpose training jobs, and run Jupyter Notebooks.
- User management via Dex and access control via organizations and projects.
- Integrates with Open WebUI and other tooling via OpenAI-compatible APIs.

Language: Go | License: Apache 2.0 | 53 ⭐️

▶️ GitHub repo
📢 Reddit announcement

#tools #aiml

Yesterday, Solo.io announced its new Open Source project called kagent. It is a Kubernetes-native framework for DevOps and platform engineers to build and run AI agents that will automate configuration, troubleshooting, and other similar tasks.

kagent uses custom resources to define the AI agents and tools used by those agents, and offers CLI and web UI to manage them. The project also boasts the tools registry, where you can find predefined functions for interacting with Kubernetes, Prometheus, Istio, Argo, Helm, and other projects.

The tool is written in Python, based on the AutoGen framework, and licensed under the Apache 2.0 license. The authors plan to donate the project to the CNCF. Learn more about kagent on its website, GitHub, and in this announcement.

#news #tools #aiml

Perhaps most engineers still prefer to type full, even if quite long, CLI commands in their terminals. However, with all the GenAI hype we have now, automated assistance comes everywhere, and good old kubectl is not an exception.

kubectl-ai was created in the Google GKE team, enabling a chatbot-style experience for your kubectl commands. It runs in interactive mode or as a kubectl plugin and allows you to:
- use simple English to construct and execute the kubectl commands, including the required sequences of such;
- benefit from the UNIX shell integration by leveraging pipelines and basic CLI tools (echo, cat, etc.);
- run commands only after your explicit approval;
- use OpenAI, Gemini, Azure OpenAI, Grok, and local AI models (via Ollama or llama.cpp);
- evaluate the performance of different LLM models [for Kubernetes-related tasks] via its own benchmark called k8s-bench.

Language: Go | License: Apache 2.0 | 2529 ⭐️

▶️ GitHub repo
💬 Reddit discussion

#tools #CLI #aiml

llm-d is a new Open Source project and community for scalable GenAI deployments in Kubernetes.

Described as “a Kubernetes-native high-performance distributed LLM inference framework,” llm-d leverages existing technologies, such as vLLM, Kubernetes, and Inference Gateway, to provide a vLLM-optimised inference scheduler, disaggregated serving, and disaggregated prefix caching. The authors also plan to implement a traffic- and hardware-aware autoscaler.

You can find more details on llm-d in:
- yesterday’s announcement by CoreWeave, Google, IBM Research, NVIDIA, and Red Hat;
- the project’s GitHub repo.

#news #aiml #tools

Kubernetes Gateway API Inference Extension is a new project focused on solving traffic-routing challenges for GenAI and LLM services running on Kubernetes. Built on top of the Gateway API, it adds inference-specific routing capabilities, allowing you to transform existing gateways into a specialised (i.e. inference) one for self-hosted GenAI/LLMs. The project was launched as part of Kubernetes SIGs.

Learn more about Gateway API Inference Extension from this announcement in the Kubernetes blog and the project's GitHub repo.

#news #aiml

AI chatbot assistant right in the Argo CD UI? Here we come…

A few days ago, the assistant-for-argocd project was announced. It’s an Argo CD extension that adds a chatbot to the Argo CD UI. It relies on llama-stack as a backend and adds the Assistant tab to the resources view, where you can ask your LLM of choice about this resource.

Language: TypeScript | License: Apache 2.0 | 35 ⭐️

▶️ GitHub repo
💬 LinkedIn announcement

#tools #gui #gitops #aiml

KServe joins CNCF as an Incubating project

KServe is a standardised distributed generative and predictive AI inference platform for scalable, multi-framework deployment on Kubernetes. Technically, it provides CRDs for serving predictive and generative ML models and offers various features for that, such as intelligent routing, advanced deployments, model caching, autoscaling, and many more.

Today, KServe is adopted by numerous well-known organisations, including AMD, Bloomberg, Canonical, Cisco, IBM, NVIDIA, Red Hat, and the Wikimedia Foundation. Partly thanks to that, when a relevant CNCF TOC vote passed, the project was able to join the CNCF at the Incubator level.

#news #cncfprojects #aiml

Certified Kubernetes AI Conformance Program 1.0

KubeCon NA also marked the official launch of the Certified Kubernetes AI Platform Conformance Program v1.0 from CNCF, which defines capabilities and configurations for running AI and ML frameworks on Kubernetes and other CNCF projects. It was a community-driven initiative supported by various companies, including Google Cloud, Kubermatic, Microsoft, and Red Hat.

Currently, the following K8s distributions are certified with this program:
- Alibaba Cloud Container Service for Kubernetes
- Amazon Elastic Kubernetes Service (EKS)
- Azure Kubernetes Service (AKS)
- CCE (Cloud Container Engine) by Baidu Cloud
- CoreWeave Kubernetes Service
- DaoCloud Enterprise
- Gardener by NeoNephos Foundation
- Giant Swarm Platform
- Google Kubernetes Engine
- Kubermatic Kubernetes Platform
- Linode Kubernetes Engine (LKE) by Akamai
- OCI Kubernetes Engine (OKE) by Oracle
- OpenShift Container Platform and Red Hat OpenShift Service on AWS
- RKE2 by SUSE
- Talos Linux
- vSphere Kubernetes Service

#news #aiml

Looking for a way to simplify deploying LLMs on Kubernetes? This project provides everything you might need.

llmaz is an inference platform that integrates various Open Source projects for running LLMs. It supports:

- Different inference backends: vLLM, llama.cpp, Ollama, Text-Generation-Inference, SGLang, and TensorRT-LLM.
- Different model providers: HuggingFace, ModelScope, and ObjectStores.
- Chatbot interface based on Open WebUI.
- Heterogeneous devices.
- Distributed inference via multi-host and homogeneous xPyD support with LeaderWorkerSet.
- Envoy AI Gateway for token-based rate limiting, model routing, and more.
- Horizontal Pod scaling (HPA) and node autoscaling (Karpenter).

▶️ GitHub repo

Language: Go | License: Apache 2.0 | 278 ⭐️

#tools #aiml

Kubernetes WG Serving is disbanded

Yuan Tang, on behalf of the Serving working group co-chairs, announced that the WG Serving’s goal had been accomplished and that the group is disbanded.

WG Serving was created to support the development of the AI inference stack on Kubernetes, making it "an orchestration platform of choice for inference workloads". In particular, it contributed to the design of AIBrix (a part of vLLM), while other unresolved problems were implemented by llm-d. The working group also helped with Kubernetes AI Conformance requirements.

All existing related efforts are now covered by other SIGs and working groups (including SIG Node, SIG Scheduling, and WG Device Management) or specific projects (such as Gateway API Inference Extension and Inference Perf).

#news #aiml

New Kubernetes working group: AI Gateway

The “AI Gateway” term refers to network gateway infrastructure, such as proxy servers and load balancers, that implements the Gateway API specification with enhanced capabilities for AI workloads. The newly announced AI Gateway WG will create declarative APIs, standards, and guidance for AI workload networking in Kubernetes.

P.S. This announcement came shortly after disbanding the Kubernetes WG Serving.

#news #networking #aiml

NVIDIA introduced AI Cluster Runtime (AICR)

Yesterday, the company released its recipes for GPU-accelerated Kubernetes clusters across cloud and on-premises AI factories. These recipes are “version-locked configurations for specific environments” — the combinations of drivers, runtimes, operators, kernel modules, and system settings for AI workloads that have been validated by NVIDIA. They include specific components, their versions, constraints, and the configuration values for each environment.

You can find more details in this blog post and the AICR repo on GitHub.

#news #aiml

Kubernetes updates its AI usage policy

Earlier this month, the Kubernetes Steering Committee announced an update to the project’s policy on the use of AI tools for contributors. While the policy allows AI-assisted contributions, it’s essential to follow a few rules when creating such PRs:

- There should always be a human who is responsible for understanding all the suggested changes. This author should reply to the reviewers’ comments personally, without engaging AI tools.
- AI tools shouldn’t be used for so-signing commits, specified as co-authors or within assisted-by/co-developed commit trailers. (Using this metadata leads to unwanted marketing behaviour.)

You can find the latest version of the updated Kubernetes AI policy at kubernetes.dev.

#news #aiml

Upbound introduced Modelplane for AI inference

The company behind Crossplane, a CNCF project that graduated at the end of 2025, announced its new Open Source project called Modelplane. The CNCF's CTO already mentioned on LinkedIn that this project would be a great fit for CNCF.

Basically, Modelplane is a control plane for AI inference built on Crossplane, which allows you to run any model on any serving engine on any infrastructure. Here's its brief description from the official announcement:

Modelplane sits above your inference clusters and operates them as one inference platform, reconciling the whole fleet toward the state you declare. It does for the fleet what Kubernetes does for the cluster: platform teams and developers describe what they want as Kubernetes resources, and Modelplane composes the clusters, places the models, and exposes the endpoints to match.

Additional resources include the getting started documentation and the GitHub repo.

#news #tools #aiml

Kubeflow became a CNCF Graduated project

The summer series of CNCF level-ups keeps going, and Kubeflow has just joined the list of the organisation’s most mature (i.e. Graduated) projects. The CNCF Technical Oversight Committee (TOC) voted for this graduation last Friday.

Kubeflow is a machine learning toolkit for Kubernetes that serves as a foundation for K8s-based AI platforms. Its subprojects include Pipelines for end-to-end ML workflows, Trainer for distributed AI model training and LLM fine-tuning, Katib for AutoML, and a Kubernetes operator for Apache Spark.

Kubeflow originated in Google, with its first public announcement in 2017. It was accepted into the CNCF as an Incubating project in July 2023. Today, its adopters include Alibaba Cloud, AWS, Cisco, Huawei, IBM, NVIDIA, Red Hat, and Tencent.

#news #cncfprojects #aiml

Agent Orca, a Kubernetes-native platform for managing AI agents

A few days ago, Matthew Fisher, solutions architect at NVIDIA, announced an Open Source project called Agent Orca. It's a platform for deploying, managing, and running AI agents in Kubernetes at scale. It allows you to use K8s resources to "declaratively define agents, route them to the correct LLM, equip them with tools, and execute them as one-off jobs or long-running services with checkpoints, cost tracking, guardrails, and RAG built in."

There's a quick-start manual for giving Agent Orca a try, seven demos, and quite extensive documentation for both developers and operators.

▶️ GitHub repo
 
Language: Go | License: Apache 2.0 | 61 ⭐️

#news #tools #aiml