# Médéric Hurier — Fmind > AI Architect (PhD) • VC Expert Advisor • AAIF Ambassador • GCP Certified Cloud Architect • AI Agents, MLOps & Security ## Machine-readable portfolio - [MCP server](https://www.fmind.dev/mcp): Read-only portfolio tools, resources, and prompts. - [JSON profile](https://www.fmind.dev/api/profile): Canonical portfolio and article index. - [Full LLM context](https://www.fmind.dev/llms-full.txt): This index plus every public article in Markdown. - Article source: append `.md` to any article slug (https://www.fmind.dev/articles/.md) for its raw Markdown. - [Atom feed](https://www.fmind.dev/articles/feed.xml): Reverse-chronological publication feed. - [Sitemap](https://www.fmind.dev/sitemap.xml): Canonical hosted pages. ## Articles - [MCP 2026–07–28: Stateless core, enterprise authorization, and SDK betas](https://www.fmind.dev/articles/mcp-2026-07-28-stateless-core-enterprise-authorization-and-sdk-betas/) — Running a remote Model Context Protocol (MCP) server behind multiple instances presents immediate infrastructure challenges. Because the… ([Markdown](https://www.fmind.dev/articles/mcp-2026-07-28-stateless-core-enterprise-authorization-and-sdk-betas.md)) - [The Affordable AI Agents](https://www.fmind.dev/articles/the-affordable-ai-agents/) — As AI agents transition from isolated developer experiments into production-scale workflows, engineering organizations are waking up to a… ([Markdown](https://www.fmind.dev/articles/the-affordable-ai-agents.md)) - [Agent Evolutions: Stop Guessing the Design — Evolve It](https://www.fmind.dev/articles/agent-evolutions-stop-guessing-the-design-evolve-it/) — Most coding-agent tasks have one right answer hiding behind a workflow. “Add password sign-in.” “Fix the off-by-one in the pagination… ([Markdown](https://www.fmind.dev/articles/agent-evolutions-stop-guessing-the-design-evolve-it.md)) - [Agent Levers: A Plan-Do-Check-Act Loop That Makes Coding Agents Finish What They Start](https://www.fmind.dev/articles/agent-levers-a-plan-do-check-act-loop-that-makes-coding-agents-finish-what-they-start/) — A coding agent is only as useful as the spec you give it. That sentence sounds like a truism until you watch it play out for an afternoon. ([Markdown](https://www.fmind.dev/articles/agent-levers-a-plan-do-check-act-loop-that-makes-coding-agents-finish-what-they-start.md)) - [Agent Docs: Answer Locally Before the Web — A Shared Reference for Every Coding Agent](https://www.fmind.dev/articles/agent-docs-answer-locally-before-the-web-a-shared-reference-for-every-coding-agent/) — Pair-program with a coding agent for an afternoon and you start noticing a quiet, recurring tax. Every time the agent reaches for a flag, a… ([Markdown](https://www.fmind.dev/articles/agent-docs-answer-locally-before-the-web-a-shared-reference-for-every-coding-agent.md)) - [Supagents: One Source to Rule Your Coding Subagents](https://www.fmind.dev/articles/supagents-one-source-to-rule-your-coding-subagents/) — The AI providers are in a horse race. Every quarter, a new model lands, a new harness ships, and the gap between “the best tool for this… ([Markdown](https://www.fmind.dev/articles/supagents-one-source-to-rule-your-coding-subagents.md)) - [AI Agents as an Operating System: Rediscovering the Linux Philosophy](https://www.fmind.dev/articles/ai-agents-as-an-operating-system-rediscovering-the-linux-philosophy/) — We are building the most advanced AI systems in history, yet the best way to control them relies on paradigms from the 1970s. I see… ([Markdown](https://www.fmind.dev/articles/ai-agents-as-an-operating-system-rediscovering-the-linux-philosophy.md)) - [The 5xP Framework: Steering AI Coding Agents from Chaos to Success](https://www.fmind.dev/articles/the-5xp-framework-steering-ai-coding-agents-from-chaos-to-success/) — AI Coding — or Spec-Driven Development — is a recent trend that amazes as much as it scares IT engineers. We now have access to powerful… ([Markdown](https://www.fmind.dev/articles/the-5xp-framework-steering-ai-coding-agents-from-chaos-to-success.md)) - [How I Revamped My Portfolio Website in 5 Nights Using AI Agents](https://www.fmind.dev/articles/how-i-revamped-my-portfolio-website-in-5-nights-using-ai-agents/) — I see many friends and acquaintances generating amazing applications in mere weeks. We are in the midst of a craze of innovation, an era… ([Markdown](https://www.fmind.dev/articles/how-i-revamped-my-portfolio-website-in-5-nights-using-ai-agents.md)) - [Chaigent: An affordable alternative to Gemini Enterprise on Google Cloud](https://www.fmind.dev/articles/chaigent-an-affordable-alternative-to-gemini-enterprise-on-google-cloud/) — Build a cost-effective, private AI Agent solution on Google Cloud using Chainlit and Vertex AI. Detailed architecture and code guide. ([Markdown](https://www.fmind.dev/articles/chaigent-an-affordable-alternative-to-gemini-enterprise-on-google-cloud.md)) - [mAIdAI: Building a Personal Assistant with Google Cloud and Vertex AI](https://www.fmind.dev/articles/maidai-building-a-personal-assistant-with-google-cloud-and-vertex-ai/) — Build a personal AI assistant using Google Cloud Run, Vertex AI, and Google Chat. A guide to architecting context-aware agents. ([Markdown](https://www.fmind.dev/articles/maidai-building-a-personal-assistant-with-google-cloud-and-vertex-ai.md)) - [Building with A2UI: Extending the Expressiveness of AI Agent Interfaces](https://www.fmind.dev/articles/building-with-a2ui-extending-the-expressiveness-of-ai-agent-interfaces/) — Building AI-First apps with A2UI: from the Agent-View-Controller (AVC) pattern to managing latency. A practical case study with Featest. ([Markdown](https://www.fmind.dev/articles/building-with-a2ui-extending-the-expressiveness-of-ai-agent-interfaces.md)) - [MLOps Coding Skills: Bridging the Gap Between Specs and Agents](https://www.fmind.dev/articles/mlops-coding-skills-bridging-the-gap-between-specs-and-agents/) — Bridge the gap between engineering specs and AI agents using Agent Skills. Learn to inject ‘Senior Engineer’ context for production MLOps. ([Markdown](https://www.fmind.dev/articles/mlops-coding-skills-bridging-the-gap-between-specs-and-agents.md)) - [Finding the Holy Grail of AI Agent UIs: From AI-Orchestrated Development to A2UI](https://www.fmind.dev/articles/finding-the-holy-grail-of-ai-agent-uis-from-ai-orchestrated-development-to-a2ui/) — Discover how A2UI solves the AI Agent UI bottleneck. A deep dive into replacing chatbots with agents that project their own native UIs. ([Markdown](https://www.fmind.dev/articles/finding-the-holy-grail-of-ai-agent-uis-from-ai-orchestrated-development-to-a2ui.md)) - [Architecting the AI Agent Platform: A Definitive Guide](https://www.fmind.dev/articles/architecting-the-ai-agent-platform-a-definitive-guide/) — Build a production-grade AI Agent Platform. A guide to the 7-layer architecture for scaling secure, autonomous enterprise agents. ([Markdown](https://www.fmind.dev/articles/architecting-the-ai-agent-platform-a-definitive-guide.md)) - [Powering Up your Agent in Production with ADK, OAuth and Gemini Enterprise](https://www.fmind.dev/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise/) — Power up your AI agent in production. Learn to deploy a secure ‘Slides Translator’ using ADK, OAuth, and Gemini Enterprise. ([Markdown](https://www.fmind.dev/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise.md)) - [The Real AI Agent Bottleneck is the Damn UI](https://www.fmind.dev/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/) — For AI agents, the biggest challenge isn’t the model, it’s the UI. A deep dive into the user interface bottleneck for enterprise AI. ([Markdown](https://www.fmind.dev/articles/the-real-ai-agent-bottleneck-is-the-damn-ui.md)) - [Da2a: The Future of Data Platforms is Agentic, Distributed, and Collaborative](https://www.fmind.dev/articles/da2a-the-future-of-data-platforms-is-agentic-distributed-and-collaborative/) — For decades, the story of data platforms has been one of centralization and heavy engineering. We built massive data warehouses and data… ([Markdown](https://www.fmind.dev/articles/da2a-the-future-of-data-platforms-is-agentic-distributed-and-collaborative.md)) - [Ackgent: Rapid Agent Development on GCP with ADK and Agent Config](https://www.fmind.dev/articles/ackgent-rapid-agent-development-on-gcp-with-adk-and-agent-config/) — Speed up AI agent development on GCP with Ackgent. Build, deploy, and iterate faster using Google ADK and Agent Config's declarative approach. ([Markdown](https://www.fmind.dev/articles/ackgent-rapid-agent-development-on-gcp-with-adk-and-agent-config.md)) - [Combo-Banana: Building Custom Image Workflows in Record Time](https://www.fmind.dev/articles/combo-banana-building-custom-image-workflows-in-record-time/) — Build custom AI image workflows fast with Combo-Banana. Empower designers to automate tedious editing tasks with Google Nano-Banana. ([Markdown](https://www.fmind.dev/articles/combo-banana-building-custom-image-workflows-in-record-time.md)) - [Vibe Youtubing with NotebookLM: The MLOps Coding Course Gets a Video Upgrade in Under 48 Hours](https://www.fmind.dev/articles/vibe-youtubing-with-notebooklm-the-mlops-coding-course-gets-a-video-upgrade-in-under-48-hours/) — I’m incredibly excited to announce the launch of the first video series for my MLOps Coding Course, covering the introduction and the… ([Markdown](https://www.fmind.dev/articles/vibe-youtubing-with-notebooklm-the-mlops-coding-course-gets-a-video-upgrade-in-under-48-hours.md)) - [Deploying AI Agents in the Enterprise using ADK and Google Cloud](https://www.fmind.dev/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/) — Learn to deploy AI agents in the enterprise using ADK & Google Cloud. We compare Cloud Run & Vertex AI for secure, scalable AgentOps. ([Markdown](https://www.fmind.dev/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud.md)) - [Happy Birthday XP: Celebrating Gemini Deep Think (and My Daughter’s 6th Birthday)](https://www.fmind.dev/articles/happy-birthday-xp-celebrating-gemini-deep-think-and-my-daughters-6th-birthday/) — A father tests Gemini AI’s coding skills to create a magical birthday for his 6-year-old. See the staggering results from Deep Think! ([Markdown](https://www.fmind.dev/articles/happy-birthday-xp-celebrating-gemini-deep-think-and-my-daughters-6th-birthday.md)) - [Slides-To-Translate: When IT Says No, Build a $0.04 Solution on Your Lunch Break](https://www.fmind.dev/articles/slides-to-translate-when-it-says-no-build-a-0-04-solution-on-your-lunch-break/) — It is a classic tale of corporate life. A colleague of mine at Decathlon had a mammoth task: translate a 155-slide presentation into… ([Markdown](https://www.fmind.dev/articles/slides-to-translate-when-it-says-no-build-a-0-04-solution-on-your-lunch-break.md)) - [It’s Not Artificial: Recreating a Conversational Format with Gemini’s Multi-Speaker Text to Speech](https://www.fmind.dev/articles/its-not-artificial-recreating-a-conversational-format-with-gemini-s-multi-speaker-text-to-speech/) — A guide to creating conversational AI with Gemini. Generate custom educational audio with multiple voices using a simple Python script. ([Markdown](https://www.fmind.dev/articles/its-not-artificial-recreating-a-conversational-format-with-gemini-s-multi-speaker-text-to-speech.md)) - [The Agentic Cloud: Forging the Next Era of Infrastructure](https://www.fmind.dev/articles/the-agentic-cloud-forging-the-next-era-of-infrastructure/) — Agentic Cloud: Disrupting cloud giants with AI. Learn how intelligent agents will commoditize hyperscalers & reshape infrastructure. ([Markdown](https://www.fmind.dev/articles/the-agentic-cloud-forging-the-next-era-of-infrastructure.md)) - [Scaling the Summit: Challenges for Serving LLMs at Scale on GCP](https://www.fmind.dev/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/) — Navigate the challenges of scaling LLMs on GCP and get benchmark insights for cost-effective, reliable, and quick LLM serving. ([Markdown](https://www.fmind.dev/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp.md)) - [The Great Data Divergence: Why Generative AI Demands a New Approach Beyond the Data Lake](https://www.fmind.dev/articles/the-great-data-divergence-why-generative-ai-demands-a-new-approach-beyond-the-data-lake/) — Explore the shift from slow data pipelines to real-time API management for building Gen AI systems and modern enterprise data architecture. ([Markdown](https://www.fmind.dev/articles/the-great-data-divergence-why-generative-ai-demands-a-new-approach-beyond-the-data-lake.md)) - [Hackathon Speedrun: Build & Deploy a RAG App in Minutes with Vertex AI Studio & Vertex AI Search!](https://www.fmind.dev/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/) — Build & deploy a RAG app in minutes! Use Vertex AI Studio & Vertex AI Search for a rapid hackathon win. Ground LLMs with your data. ([Markdown](https://www.fmind.dev/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search.md)) - [Introducing GentWriter: Building a Multi-Agent Content Generator with Google’s ADK 👾](https://www.fmind.dev/articles/introducing-gentwriter-building-a-multi-agent-content-generator-with-googles-adk/) — In the fast-paced world of content creation, efficiency is key. Manually crafting social media posts for different platforms based on a… ([Markdown](https://www.fmind.dev/articles/introducing-gentwriter-building-a-multi-agent-content-generator-with-googles-adk.md)) - [CAG vs. RAG: Choosing the Right Strategy for Your AI Application](https://www.fmind.dev/articles/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application/) — CAG vs RAG for Generative AI: Compare latency, cost & complexity. Choose the best LLM context strategy for your app. ([Markdown](https://www.fmind.dev/articles/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application.md)) - [Structured LLM Outputs Made Easy: Building a Slack Digest Agent with Pydantic AI](https://www.fmind.dev/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/) — As developers and AI practitioners, many of us belong to vibrant online communities, like the MLOps Community on Slack. These platforms are… ([Markdown](https://www.fmind.dev/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai.md)) - [GitWorks: an AI Agent for Enforcing GitHub Repository Standards](https://www.fmind.dev/articles/gitworks-an-ai-agent-for-enforcing-github-repository-standards/) — A Python notebook agent that reviews GitHub repositories against your own guidelines automatically, using Google Colab, the GitHub API, and Gemini models. ([Markdown](https://www.fmind.dev/articles/gitworks-an-ai-agent-for-enforcing-github-repository-standards.md)) - [GenV: An Agentic Workflow for Actionable Insights from Google Meet Recordings](https://www.fmind.dev/articles/genv-an-agentic-workflow-for-actionable-insights-from-google-meet-recordings/) — Video meetings on platforms like Google Meet are essential for collaboration, but how often do crucial details get lost moments after the… ([Markdown](https://www.fmind.dev/articles/genv-an-agentic-workflow-for-actionable-insights-from-google-meet-recordings.md)) - [BKFC: An Agentic Workflow for Gathering Knowledge from Google Chat](https://www.fmind.dev/articles/bkfc-an-agentic-workflow-for-gathering-knowledge-from-google-chat/) — Team collaboration often lives and breathes within chat applications like Google Chat or Slack. It’s where questions are asked, decisions… ([Markdown](https://www.fmind.dev/articles/bkfc-an-agentic-workflow-for-gathering-knowledge-from-google-chat.md)) - [Framework, Template, or Example? 🤔 Choosing the Right AI Starter Kit for Your Team ✨](https://www.fmind.dev/articles/framework-template-or-example-choosing-the-right-ai-starter-kit-for-your-team/) — Building AI applications is incredibly exciting. But let’s be honest: getting started can be challenging, especially in large… ([Markdown](https://www.fmind.dev/articles/framework-template-or-example-choosing-the-right-ai-starter-kit-for-your-team.md)) - [More Automation + More Reproducibility = MLOps Python Package v4.1.0](https://www.fmind.dev/articles/more-automation-more-reproducibility-mlops-python-package-v4-1-0/) — The MLOps Python Package is your go-to solution for building robust and reproducible AI/ML workflows. Check out the latest v4.1.0 release! ([Markdown](https://www.fmind.dev/articles/more-automation-more-reproducibility-mlops-python-package-v4-1-0.md)) - [Meet Kate: Your AI-Powered, Live Multimodal Website Assistant 🤖](https://www.fmind.dev/articles/meet-kate-your-ai-powered-live-multimodal-website-assistant/) — Meet Kate: an AI-powered, live, multimodal website assistant. She uses voice, understands your needs, and retrieves info from a website. ([Markdown](https://www.fmind.dev/articles/meet-kate-your-ai-powered-live-multimodal-website-assistant.md)) - [Lessons Learned from the Gemini Long Context Kaggle Competition 🧠](https://www.fmind.dev/articles/lessons-learned-from-the-gemini-long-context-kaggle-competition/) — Explored Gemini’s long context in a Kaggle competition. Key lessons learned about its potential and limitations for real-world… ([Markdown](https://www.fmind.dev/articles/lessons-learned-from-the-gemini-long-context-kaggle-competition.md)) - [Stop Building Rigid AI/ML Pipelines: Embrace Reusable Components for Flexible MLOps](https://www.fmind.dev/articles/stop-building-rigid-ai-ml-pipelines-embrace-reusable-components-for-flexible-mlops/) — MLOps pipelines are often too rigid. We propose using reusable artifacts orchestrated by DAGs. Think functional programming for ML! ([Markdown](https://www.fmind.dev/articles/stop-building-rigid-ai-ml-pipelines-embrace-reusable-components-for-flexible-mlops.md)) - [Poetry Was Good, Uv Is Better: An MLOps Migration Story](https://www.fmind.dev/articles/poetry-was-good-uv-is-better-an-mlops-migration-story/) — Poetry to Uv: A faster, simpler way to manage dependencies for MLOps projects. ([Markdown](https://www.fmind.dev/articles/poetry-was-good-uv-is-better-an-mlops-migration-story.md)) - [Taming Python Installation with Pyenv for AI/ML and MLOps Projects](https://www.fmind.dev/articles/taming-python-installation-with-pyenv-for-ai-ml-and-mlops-projects/) — Tame your Python chaos! Learn how pyenv simplifies managing Python versions for AI/ML & MLOps, ensuring smooth, reproducible development. ([Markdown](https://www.fmind.dev/articles/taming-python-installation-with-pyenv-for-ai-ml-and-mlops-projects.md)) - [Bromate: Automate Your Browser with Agentic Workflows 🧭](https://www.fmind.dev/articles/bromate-automate-your-browser-with-agentic-workflows/) — The AI landscape is constantly evolving, pushing the boundaries of what’s possible with technology. One of the most exciting frontiers is… ([Markdown](https://www.fmind.dev/articles/bromate-automate-your-browser-with-agentic-workflows.md)) - [MLOps Package Template: Turbocharge the Creation of AI/ML Projects ⚡](https://www.fmind.dev/articles/mlops-package-template-turbocharge-the-creation-of-ai-ml-projects/) — The Cookiecutter MLOps Package offers a powerful code template to jumpstart your MLOps journey and accelerate your AI/ML development… ([Markdown](https://www.fmind.dev/articles/mlops-package-template-turbocharge-the-creation-of-ai-ml-projects.md)) - [MLOps Coding Course: Mastering Observability for Reliable ML 💡](https://www.fmind.dev/articles/mlops-coding-course-mastering-observability-for-reliable-ml/) — This article dives deep into the essential tools and practices for achieving comprehensive observability in your AI/ML projects. ([Markdown](https://www.fmind.dev/articles/mlops-coding-course-mastering-observability-for-reliable-ml.md)) - [MLOps Coding Course: Bridging the gap between Data Scientists and Machine Learning Engineers](https://www.fmind.dev/articles/mlops-coding-course-bridging-the-gap-between-data-scientists-and-machine-learning-engineers/) — The MLOps Coding Course is an open-source resource specifically designed to bridge the gap between data science and software engineering. ([Markdown](https://www.fmind.dev/articles/mlops-coding-course-bridging-the-gap-between-data-scientists-and-machine-learning-engineers.md)) - [Make your MLOps code base SOLID with Pydantic and Python’s ABC](https://www.fmind.dev/articles/make-your-mlops-code-base-solid-with-pydantic-and-pythons-abc/) — MLOps projects are straightforward to initiate, but challenging to perfect. While AI/ML projects often start with a notebook for… ([Markdown](https://www.fmind.dev/articles/make-your-mlops-code-base-solid-with-pydantic-and-pythons-abc.md)) - [Become the maestro of your MLOps abstractions](https://www.fmind.dev/articles/become-the-maestro-of-your-mlops-abstractions/) — The MLOps ecosystem is evolving into a sophisticated symphony, composed of diverse tools, methodologies, and cultures. This diversity… ([Markdown](https://www.fmind.dev/articles/become-the-maestro-of-your-mlops-abstractions.md)) - [How to configure VS Code for AI, ML and MLOps development in Python 🛠️️](https://www.fmind.dev/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/) — Visual Studio Code is a remarkable application. In the past, developers faced a choice between simple, lightweight text editors like Vim… ([Markdown](https://www.fmind.dev/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python.md)) - [Forging a Personal Chatbot with OpenAI API, Chroma DB, HuggingFace Spaces, and Gradio 🔥](https://www.fmind.dev/articles/forging-a-personal-chatbot-with-openai-api-chroma-db-huggingface-spaces-and-gradio/) — If you have checked the Internet in 2023, you’re likely familiar with Generative AI. The launch of ChatGPT has sparked a surge in interest… ([Markdown](https://www.fmind.dev/articles/forging-a-personal-chatbot-with-openai-api-chroma-db-huggingface-spaces-and-gradio.md)) - [Finding Harmony in MLOps: Balancing Functional and Object-Oriented Approaches ☯](https://www.fmind.dev/articles/finding-harmony-in-mlops-balancing-functional-and-object-oriented-approaches/) — Programmers have always been passionate about their preferences, whether they discuss spaces vs. tabs, Vim vs. Emacs, or light mode vs… ([Markdown](https://www.fmind.dev/articles/finding-harmony-in-mlops-balancing-functional-and-object-oriented-approaches.md)) - [Is AI/ML Monitoring just Data Engineering? 🤔](https://www.fmind.dev/articles/is-ai-ml-monitoring-just-data-engineering/) — While the future of machine learning and MLOps is being debated, practitioners still need to attend to their machine learning models in… ([Markdown](https://www.fmind.dev/articles/is-ai-ml-monitoring-just-data-engineering.md)) - [A great MLOps project should start with a good Python Package 🐍](https://www.fmind.dev/articles/a-great-mlops-project-should-start-with-a-good-python-package/) — MLOps practitioners (rightfully) point out that running notebooks in production is a bad software practice, but what are the alternatives… ([Markdown](https://www.fmind.dev/articles/a-great-mlops-project-should-start-with-a-good-python-package.md)) - [Fixing the MLOps Survey on LLMs with ChatGPT API: Lessons Learned](https://www.fmind.dev/articles/fixing-the-mlops-survey-on-llms-with-chatgpt-api-lessons-learned/) — Large Language Model (LLM) is such an exciting topic. Since the release of ChatGPT, we saw a surge of innovation ranging from education… ([Markdown](https://www.fmind.dev/articles/fixing-the-mlops-survey-on-llms-with-chatgpt-api-lessons-learned.md)) - [We need POSIX for MLOps](https://www.fmind.dev/articles/we-need-posix-for-mlops/) — There is an ever-growing landscape of tools and solutions for MLOps. In this article, I propose a solution to address this challenge. ([Markdown](https://www.fmind.dev/articles/we-need-posix-for-mlops.md)) - [How to install Kubeflow Pipelines v2 on Apple Silicon](https://www.fmind.dev/articles/how-to-install-kubeflow-on-apple-silicon/) — Kubeflow Pipelines (KFP) is a powerful platform for building machine learning pipelines at scale with Kubernetes. ([Markdown](https://www.fmind.dev/articles/how-to-install-kubeflow-on-apple-silicon.md)) - [3P Principle: Purpose, Productivity, Performance](https://www.fmind.dev/articles/3p-principle-purpose-productivity-performance/) — A fundamental trade-off that affects the design of programming languages and the success of software projects. ([Markdown](https://www.fmind.dev/articles/3p-principle-purpose-productivity-performance.md)) ## Optional - [PhD thesis](https://orbilu.uni.lu/handle/10993/39903): Creating better ground truth to further understand Android malware - [Euphony: Harmonious Unification of Cacophonous Anti-Virus Vendor Labels](https://orbilu.uni.lu/handle/10993/31441) — MSR 2017 • Mining Software Repositories - [On the Lack of Consensus in Anti-Virus Decisions](https://orbilu.uni.lu/handle/10993/27845) — DIMVA 2016 • Detection of Intrusions and Malware ## Full articles ### [MCP 2026–07–28: Stateless core, enterprise authorization, and SDK betas](https://www.fmind.dev/articles/mcp-2026-07-28-stateless-core-enterprise-authorization-and-sdk-betas/) Running a remote [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server behind multiple instances presents immediate infrastructure challenges. Because the original spec requires an `initialize` handshake and pins sessions to a specific instance via the `Mcp-Session-Id` header, you are forced to configure sticky routing or a shared session store at the load balancer level. This makes scaling beyond a single process unnecessarily complex. To address these limitations, the MCP project released three updates in mid-2026: a [release candidate for the 2026–07–28 specification](https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/), a stable [enterprise-managed authorization extension](https://blog.modelcontextprotocol.io/posts/enterprise-managed-auth/), and [beta SDKs](https://blog.modelcontextprotocol.io/posts/sdk-betas-2026-07-28/) across Python, TypeScript, Go, and C#. Together, these changes address three core production challenges: stateless scaling, centralized identity management, and protocol version stability. ![Photo by Robynne O on Unsplash](/static/img/articles/mcp-2026-07-28-stateless-core-enterprise-authorization-and-sdk-betas/cover.webp) Photo by [Robynne O](https://unsplash.com/@roborobs?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### Stateless core: scaling without handshakes The 2026–07–28 specification removes the stateful handshake and session management. The `initialize`/`initialized` handshake is gone, and so is the protocol-level session. Client information, protocol version, and capabilities now travel in a request’s `_meta` field on every call, making each request self-describing. This allows remote servers to run behind standard round-robin load balancers without session affinity or synchronization. Any instance can answer any request without first replaying a handshake or looking up shared session state. A standard tool call now requires no prior initialization and carries all context self-contained: ```http POST /mcp HTTP/1.1 Mcp-Protocol-Version: 2026-07-28 Mcp-Method: tools/call Mcp-Name: search {"jsonrpc":"2.0","id":1,"method":"tools/call", "params":{"name":"search","arguments":{"q":"otters"}, "_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28", "io.modelcontextprotocol/clientInfo":{"name":"my-app","version":"1.0"}}}} ``` The `_meta` block carries the client identity and protocol version. The `Mcp-Method` and `Mcp-Name` HTTP headers allow load balancers or gateways to route the call without parsing the JSON payload body. Because the protocol is now stateless, servers can no longer interrupt an active connection to request input. Instead, the spec introduces Multi Round-Trip Requests. When a tool requires input, the server returns an `InputRequiredResult` containing an opaque `requestState`. The client must then re-invoke the tool call, passing the user’s input alongside the `requestState` token. This design allows any available instance to process the response without holding a connection open. This stateless model increases request payload sizes because client capabilities and session parameters must be transmitted with each call. However, it enables standard production features: full JSON Schema 2020–12 validation, cache-control hints (`ttlMs` and `cacheScope`), W3C Trace Context propagation for OpenTelemetry, and a modular extensions framework. The first two official extensions are MCP Apps (sandboxed HTML-based user interfaces) and a stateless version of Tasks. ### Centralized identity: enterprise-managed authorization Beyond horizontal scaling, managing MCP server access across a team requires central authorization. The standard per-user OAuth flow forces every employee to authorize each MCP server individually, which complicates onboarding, lacks centralized audit logs, and mixes personal and corporate access. To address this, the enterprise-managed authorization extension (stabilized in June 2026) implements the [Identity Assertion JWT Authorization Grant](https://datatracker.ietf.org/doc/draft-ietf-oauth-identity-assertion-authz-grant/) (ID-JAG). During single sign-on (SSO), the client trades the user’s identity token for an ID-JAG scoped to a specific target server via [RFC 8693](https://www.rfc-editor.org/rfc/rfc8693) token exchange. The client then presents this grant to the MCP server’s authorization server to obtain an access token. This token exchange runs entirely in the background, removing the need for individual interactive consent screens: ```text # 1. IdP: swap the SSO identity for a grant good only for one MCP server grant_type=urn:ietf:params:oauth:grant-type:token-exchange requested_token_type=urn:ietf:params:oauth:token-type:id-jag resource=https://mcp.example/ # the specific server this grant targets subject_token= # 2. MCP authorization server: swap that grant for the server's access token grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer assertion= ``` Access control is governed by the organization’s existing identity provider (IdP) groups and roles. Administrators can configure permissions from a single control plane, and all access generates centralized audit trails. This extension is additive and opt-in. The default remains per-user OAuth for consumer configurations. MCP servers advertise support for enterprise-managed authorization as a capability, and clients negotiate it during connection. The IdP only validates policy and mints the grant, meaning it never intercepts raw MCP traffic, and each access token is strictly audience-restricted to its target server. ### Version stability: release deprecation windows and SDK betas To make the protocol viable for long-term roadmaps, MCP has introduced a formal specification lifecycle. Features now transit through `Active → Deprecated → Removed` states, with a minimum deprecation window of twelve months. Furthermore, proposed changes to the Standards Track must be validated against a formal conformance suite, and specifications are versioned by calendar date. In the 2026–07–28 revision, several legacy features (Roots, Sampling, and Logging) are deprecated. They will be replaced by tool parameters, resource URIs, direct provider APIs, and standard error output or OpenTelemetry streams. To allow teams to validate these changes ahead of the final specification release on July 28, the project has published beta SDKs across four languages: ```bash # Python — opt-in pre-release pip install "mcp[cli]==2.0.0b1" # TypeScript v2 — new split server/client packages npm install @modelcontextprotocol/server@beta @modelcontextprotocol/client@beta # Go — same module path, pre-release tag go get github.com/modelcontextprotocol/go-sdk@v1.7.0-pre.1 # C# dotnet add package ModelContextProtocol --prerelease ``` The existing `v1` SDK lines will receive bug and security patches for at least six months following the final release. Backward compatibility is maintained: `v2` servers continue to accept legacy 2025–11–25 handshake requests, allowing clients to upgrade independently. ### What this means for teams running agents in production MCP is now hosted by the [Agentic AI Foundation](https://aaif.io) under the Linux Foundation. This open governance model ensures neutral stewardship while retaining a focused [specification enhancement process](https://modelcontextprotocol.io/specification/versioning) led by active maintainers. Implementing a predictable release cadence, formal deprecation timelines, and conformance testing establishes MCP as a reliable infrastructure standard rather than an unstable library. With the 2026–07–28 specification locking at the end of July and SDK betas available, you should evaluate the following points in your deployment: 1. **Identify session dependencies.** If your setup relies on sticky routing or shared session tables, migrating to the stateless core will allow you to transition to standard round-robin load balancing and simplify your infrastructure. 2. **Evaluate your onboarding flow.** If your servers currently require individual user consent redirects, verify if your IdP and client support the new enterprise token-exchange extension to enable role-based authorization. 3. **Plan your version migrations.** If your systems are pinned to the original 2025–11–25 specification, utilize the 12-month deprecation window to map out updates to your tool and resource signatures. ### [The Affordable AI Agents](https://www.fmind.dev/articles/the-affordable-ai-agents/) As AI agents transition from isolated developer experiments into production-scale workflows, engineering organizations are waking up to a sobering financial reality: the cost of running AI systems scales linearly, not logarithmically. In traditional software engineering, scaling a platform reduces the marginal cost per transaction toward zero. With agentic architectures, however, every loop, file traversal, and API call incurs a direct token cost. At one of my enterprise customers, with a distributed engineering team of 5,000 developers, we recently lived through the consequences of this shift. When [GitHub Copilot transitioned to strict usage-based billing (UBB)](https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/) on June 1, 2026, it triggered a massive 5x cost increase in a single month. Background agents got caught in unoptimized autonomous debug cycles, consuming compute at machine speed and turning a promised productivity gain into an infrastructure budget emergency. We haven’t ditched GitHub Copilot yet, but this bill shock forced our hand. We initiated a deep evaluation of alternative options, focusing heavily on self-hosted open-weights models running on our Google Cloud Platform (GCP) infrastructure. To establish a path to genuine ROI, we had to apply strict Cartesian FinOps rigor to our AI token consumption. This article is a **RETEX (Return on Experience)** on evaluating managed SaaS APIs, GCP self-hosted infrastructure, and decentralized local execution to limit the financial bleeding. ![Photo by Sasun Bughdaryan on Unsplash](/static/img/articles/the-affordable-ai-agents/cover.webp) Photo by [Sasun Bughdaryan](https://unsplash.com/@sasun1990?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### 1. The Collapse of VC-Subsidized Subscriptions Flat-rate subscriptions were venture-backed land grabs. They were never a sustainable long-term option for agentic workloads, and they are now collapsing under the computational weight of autonomous loops. The scale of this subsidization was staggering. [Leaked financial disclosures](https://startupfortune.com/openais-leaked-financials-show-who-is-actually-winning-the-ai-arms-race/) (widely discussed in developer spaces like this [Ars Technica thread](https://arstechnica.com/civis/threads/leaked-financial-docs-show-openai-is-losing-billions-of-dollars-a-year.1513551/)) show that OpenAI spent **\$34 billion against \$13 billion in revenue in 2025** — yielding an operating loss of **\$21 billion**. Most of this capital flowed directly to infrastructure providers like Microsoft and NVIDIA. **Why our bill spiked:** Previously, heavy Copilot usage was governed by a flat allowance of about 300 requests per user. Under the new UBB model, that was replaced by a pooled allowance of “AI Credits” (1 credit = \$0.01) based on raw token throughput. A massive 30,000-token codebase ingestion used to count as _one request_. Today, that single agentic run drains dollars from the shared credit pool in days. Data from the [Ramp AI Index (June 2026)](https://ramp.com/leading-indicators/the-job-market-paradox-plus-ai-spending-caps) reveals a profound polarization in corporate spend resulting from this shift: - **The Median Company:** Spends **\$11** per employee/month (mostly human-in-the-loop chat). - **The Top 10%:** Spends **\$611** per employee/month. - **The Top 1%:** Spends a massive **\$7,449** per employee/month. A human developer is restricted by biology. Autonomous agents running parallel iterations are not. ### 2. The Managed SaaS Baseline To rigorously evaluate our GCP self-hosting and local alternatives, we first needed to establish a pure API baseline. Our estimated workload for our standard agentic IDE setup is **1.85 million prompt iterations per month** _(5,000 developers × ~17 prompts per active workday)_. Our baseline utilized the highly efficient [Google Gemini 3.5 Flash](https://ai.google.dev/gemini-api/docs/pricing) API on Vertex AI. Standard pricing is **\$1.50 per 1M input tokens** and **\$9.00 per 1M output tokens**. Let’s calculate the economics for a standard codebase query (10,000-token input, 2,000-token output): 1. **Input cost:** \$0.0150 2. **Output cost:** \$0.0180 3. **Total cost per query:** **\$0.0330** At 1.85 million queries, our baseline SaaS cost is **\$61,050 per month**. This is mathematically sound and reflects a mature IDE doing continuous context injection and proactive suggestions. However, running this exact same workload on premium flagship models quickly triggers budget overruns: - [**Claude 4.8 Opus**](https://www.anthropic.com/news/claude-opus-4-8) (\$5.00/M in, \$25.00/M out): Inflates the bill to **\$185,000 / month.** - [**Claude Fable 5**](https://www.anthropic.com/pricing) (\$10.00/M in, \$50.00/M out): Balloons the cost to **\$370,000 / month.** At these prices, an unoptimized sub-agent looping indefinitely in the background ceases to be a developer friction and becomes a board-level issue, as highlighted by [Fortune reporting on Uber exhausting its entire 2026 AI budget in just four months](https://fortune.com/2026/05/26/uber-coo-ai-spending-tokens-claude-code/) on Claude coding tools. ### 3. The Challenges of Self-Hosting Confronted with the risks of runaway API bills, our infrastructure team evaluated self-hosting open-weights models on Google Kubernetes Engine (GKE). SOTA models like Zhipu AI’s [GLM-5.2](https://z.ai/blog/glm-5.2) are [rapidly closing the gap with proprietary giants](https://thezvi.wordpress.com/2026/06/22/glm-5-2-is-the-new-best-open-model/). However, a strict FinOps analysis reveals that self-hosting is not a silver bullet: it simply replaces a variable API bill with a massive, fixed infrastructure baseline. Based on a standard **730-hour cloud month**, here are the real cluster economics: - [**Gemma 4 26B MoE**](https://ai.google.dev/gemma/docs/core) **:** Served on cost-efficient GCP [g2-standard-48](https://gcloud-compute.com/g2-standard-48.html) nodes (4x NVIDIA L4, 96GB VRAM) at **\$4.00/hr**. An autoscaling cluster averaging 4 active nodes costs 4 × 730 × \$4.00 = **\$11,680 per month**. - [**Qwen3-Coder-Next**](https://huggingface.co/Qwen/Qwen3-Coder-Next-Base) **(80B MoE):** The 92GB FP8 runtime footprint causes immediate Out-Of-Memory (OOM) failures under concurrency on standard L4 nodes (which max out at 96GB of heavily fragmented VRAM). Because GCP does not offer “single-GPU” rentals for its flagship H100/H200 instances, hosting this requires falling back to an [a2-ultragpu-2g](https://cloud.google.com/compute/docs/gpus) instance (2x NVIDIA A100 80GB GPUs, 160GB total VRAM) at **\$10.14/hr**. Averaging 3 active nodes costs 3 × 730 × \$10.14 = **\$22,206 per month**. - [**GLM-5.2**](https://z.ai/blog/glm-5.2) **(744B MoE):** Its massive footprint makes it physically impossible to run on standard 40GB GPU instances. Even aggressively quantized to INT4, the weights alone consume ~~372GB of VRAM. Hosting it with a usable 1M-token KV cache demands high-end [a2-ultragpu-8g](https://cloud.google.com/compute/docs/gpus) instances (8x NVIDIA A100 80GB GPUs, 640GB total VRAM) at **~~\$40.55/hr**. Averaging 3 active nodes scales the cluster bill to 3 × 730 × \$40.55 = **\$88,804 per month** — a massive premium _over_ our managed SaaS baseline. [Committed use contracts](https://docs.cloud.google.com/docs/cuds) can significantly reduce infrastructure bills by locking in lower compute rates. However, this demands high FinOps maturity: organizations must accurately forecast long-term AI usage and strictly commit the company to specific hardware families to avoid wasting budget on unused or obsolete GPUs. **The Hidden TCO:** Managing a GKE LLM gateway requires at least 1–2 ML Platform Engineers to debug GPU drivers, handle routing, and optimize execution. Factoring in their fully loaded salaries (\$20,000–\$40,000/month) completely erases the raw compute savings of hosting mid-tier models like Gemma. ### 4. The Local Execution Fallacy If centralized GCP self-hosting carries high overhead, what about decentralizing to developer workstations? We tested running [**Gemma 4 12B**](https://developers.googleblog.com/gemma-4-12b-the-developer-guide/) and [**Qwen 3.7**](https://fireworks.ai/blog/qwen-3p7-plus) locally on Apple Silicon M4 Macs. While this successfully bypasses cloud costs, it introduces severe bottlenecks to developer velocity: **1. macOS Memory Allocation Limits** Apple Silicon shares unified memory dynamically between the CPU and GPU. By default, the Metal API restricts GPU allocations via the [recommendedMaxWorkingSetSize](https://developer.apple.com/documentation/metal/mtldevice/recommendedmaxworkingsetsize) property to ~70% of physical RAM to prevent OS starvation. On a 32GB Mac, this leaves a hard VRAM ceiling of ~22GB. Attempting to force-override this limit locks wired memory that cannot page to disk, inevitably leading to [Apple MLX kernel panics](https://github.com/ml-explore/mlx-lm/issues/883) and system crashes when developers multitask. **2. The Prefill Latency Bottleneck (The Flow Killer)** Prompt ingestion (the prefill phase) is heavily compute-bound. Because a base M4 chip lacks the compute density of a datacenter GPU, ingesting a standard 10,000-token codebase context locally takes **45 to 60 seconds**. _Compare this directly to our baseline:_ Google’s TPU infrastructure processes that exact same 10,000-token prefill via the [Gemini 3.5 Flash API](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-5/) in **roughly 1 to 2 seconds**. Waiting a minute for your local agent to “read” your code completely shatters the developer’s flow state. **3. The Bandwidth Ceiling** Once prefill completes, generation is memory bandwidth-bound. A base M4 chip has 120 GB/s of unified memory bandwidth. Pushing a 12B parameter model (quantized to roughly 8GB) across the bus establishes an absolute physical generation ceiling of **~15 tokens per second** (120/8). No software framework optimization can bypass physical hardware limits. Distributing quantized weights to endpoints and triaging local crashes adds an estimated IT maintenance overhead of **2 hours per month per developer**. Combined with the lost productivity of 60-second wait times per prompt, the decentralized savings are a total illusion. **The Harsh Reality**: With Apple [confirming price increases on its laptops](https://9to5mac.com/2026/06/17/apple-confirms-price-increases-are-coming-to-its-products-due-to-ram-shortage/) and the [global shortage of RAM](https://www.theverge.com/news/839353/pc-ram-shortage-pricing-spike-news), this hardware-dependent approach is simply not scalable. It is financially unfeasible to provision expensive, maxed-out Macs to every single developer across a 5,000-person engineering organization just to support local inference. Furthermore, moving AI execution to individual workstations creates a decentralized black box. It completely strips away the centralized telemetry, usage tracking, and auditability required for enterprise security and strict FinOps compliance. ### 5. Our Architectural Playbook Moving Forward Based on our evaluation of GCP infrastructure versus managed APIs and local execution, we stopped treating AI hosting as a binary choice. To navigate the new token economic realities, we propose a staged, hybrid roadmap: 1. **Optimize context footprint with managed APIs first:** For new use cases, we keep capital costs low by leveraging standard managed APIs. This is the easiest and most convenient way to experiment with LLMs and quickly adapt the model to the user context. 2. **Classify workloads and route queries by task complexity:** We treat LLMs like consultants. We recommend using ultra-premium flagship models (like Claude 4.8 Opus) exclusively for complex architectural design or security audits. Conversely, standard code completions, minor edits, and debugging queries are strictly routed to fast, cost-efficient mid-tier models (like Gemini 3.5 Flash or Claude 4.6 Sonnet). 3. **Leverage enterprise subscriptions cautiously with growth margins:** We haven’t migrated off Copilot yet because it still offers predictable seat-based budgeting in the short term. However, we are investing in providers willing to keep subscription-based pricing, knowing we need to have clear and long-term contracts to guarantee these costs and capacity. 4. **Host selective open-weights models for high-frequency automation:** Our GCP evaluation proved that self-hosting makes financial sense primarily for mature and predictable use cases. We can provide capacity for tasks like AI coding or day-to-day automation using models like Gemma 4. This caps our infrastructure cost on tasks running 24/7, keeping them completely off the pay-per-use meter. 5. **Factor in the hidden hardware, latency, and personnel TCO:** Before migrating any workload, we account for the full operational cost. Decoupling from SaaS APIs to run LLMs on local developer Macs introduces workflow-destroying latency bottlenecks, while GKE hosting requires specialized ML platform engineers. We only shift workloads when the ROI justifies the maintenance overhead. The era of all-you-can-eat AI and [token maxxing](https://en.wikipedia.org/wiki/Token_maxxing) is over. As open-weights models close the performance gap with proprietary APIs, designing a hybrid infrastructure is the only path to long-term model independence and predictable costs. _How is your engineering team handling the shift to usage-based AI billing? Are you sticking with GitHub Copilot, evaluating GCP self-hosting, or testing local models? Let me know in the comments below or reach out to me directly here._ ### [Agent Evolutions: Stop Guessing the Design — Evolve It](https://www.fmind.dev/articles/agent-evolutions-stop-guessing-the-design-evolve-it/) Most coding-agent tasks have one right answer hiding behind a workflow. “Add password sign-in.” “Fix the off-by-one in the pagination cursor.” “Refactor this handler to use the new client.” The job is to converge — pick the path, drive it to done, ship the diff. That is the world [**agent-levers**](https://fmind.medium.com/agent-levers-a-plan-do-check-act-loop-that-makes-coding-agents-finish-what-they-start) lives in. Some tasks aren’t shaped like that. “Which prompt wins?” “Which caching strategy is fastest?” “Which library shape is friendlier to call from a worker?” There is no single correct path. There are five plausible ones, the differences are not obvious from inspection, and the only honest way to pick is to _measure_. The shape of the work is **divergent** — branch out, evaluate, keep what works. I kept reaching for that second shape and finding nothing on the shelf. So I built [**agent-evolutions**](https://github.com/fmind/agent-evolutions) — a small set of [Agent Skills](https://github.com/anthropics/skills) that turn a coding agent into a budgeted genetic search over a design space, with verifiable scoring at every step. This is the fourth post in a thread on the shared `.agents/` ground, after [supagents](https://fmind.medium.com/supagents-one-source-to-rule-your-coding-subagents-), [agent-docs](https://fmind.medium.com/agent-docs-answer-locally-before-the-web), and [agent-levers](https://fmind.medium.com/agent-levers-a-plan-do-check-act-loop-that-makes-coding-agents-finish-what-they-start). Same bet, different question: levers ask _“did we build it right?”_, evolutions ask _“did we pick the right thing to build?”_ ![Agent Evolutions: Genetic exploration of solution spaces, with verifiable scoring — https://github.com/fmind/agent-evolutions](/static/img/articles/agent-evolutions-stop-guessing-the-design-evolve-it/cover.webp) Agent Evolutions: *Genetic exploration of solution spaces, with verifiable scoring — *[https://github.com/fmind/agent-evolutions](https://github.com/fmind/agent-evolutions) ### The design-by-intuition tax Three failure modes I keep watching teams pay for — including my own: **The five-minute eyeball test.** Two prompts on the screen, both look reasonable. One person prefers the terser one; another prefers the structured one. The team picks whichever was authored by the loudest reviewer. Three weeks later a quiet regression appears on a slice of traffic nobody A/B’d against. **Taste vs. taste.** “Which client shape should we expose — fluent builder or plain options bag?” Six people, six opinions, zero numbers. The argument lasts a week. Whatever ships is then defended for years because rolling it back is more expensive than living with it. **The benchmark that becomes a vote.** “Which caching strategy is fastest?” Real, mechanical, measurable. Then someone runs _one_ manual test, on _one_ workload, and reports a winner. Nobody asks how sensitive the result is, nobody runs the loser to confirm, nobody bothers with a second seed. These all share a common shape: a design question with multiple plausible answers and at least one mechanical measure — _and we answered it with intuition anyway_. The cost of guessing wrong is rarely visible at decision time. It shows up months later as a refactor. What I wanted was a tool that made the right move — _generate variants, measure, evolve, pick_ — cheaper than the wrong move. That is the gap agent-evolutions is aimed at. ### Convergent and divergent loops, side by side The clearest way I can frame the relationship to agent-levers is by the shape of the loop. Levers is **convergent**. One workspace, one diff, one verifier set, plan → do → check → act in a sequential chain. The question being answered is _how do we drive this single task to done_. The whole machine is tuned to finish. Evolutions is **divergent, then convergent**. N parallel workspaces (one per variant), N independent results, batched generations that learn from the previous batch, a final pick at the end. The question is _which of several plausible designs measures best_. The machine is tuned to explore first, then collapse. Both loops live on the same `.agents/` ground, both use file-based state, both let any compliant coding agent drive them. They are not competitors — they are complementary tools for different question shapes. Most non-trivial work, in my experience, is mostly levers and occasionally evolutions. The trick is knowing which kind of question you’re holding. ### Genetic search, not random sampling The first thing people picture when I say “spawn variants in parallel and pick the best” is brute-force Best-of-N: fan out thirty prompt mutations, score them, keep the top one. That is sampling. It is blind, and it scales badly — doubling the budget barely improves the answer because every sample is drawn from the same flat distribution. Agent-evolutions runs a **genetic** loop instead: - **Generation 1** seeds a small, _diverse_ set of hypotheses — qualitatively different approaches, not minor variants of one idea. - **Generation 2+** mutates the top survivors, crosses pairs over, and injects a small share of fresh exploration variants to avoid premature collapse. - **Batches are sized to `budget.parallel`**, deliberately, so the loop can _learn between batches_ instead of fanning out blindly. - **Stopping** is automatic: budget cap, wall clock, score plateau across N generations, or no survivors. Each generation conditions on what worked in the previous one. That is the qualitative difference — and the reason a small directed search routinely beats a much larger random one. The skill’s documented `optimize cli startup` walkthrough finishes in 24 evaluated variants across 4 generations, with the winner from generation 2 — not generation 1. ### Three commands, three phases The whole surface area is three slash commands, one per phase: - **Capture** — /new-agent-evolution \ discusses the brief (objective, gates, rubric, scope, budget) and commits it when you agree. - **Run** — /run-agent-evolution \ plans a batch, spawns variants in parallel, ingests each `result.json`, scores, evolves, and stops on a condition. - **Apply** — /apply-agent-evolution \ diffs the winner workspace against the live repo, confirms with you, copies it in, and optionally re-runs gates against the repo. Diff winner workspace vs the live repo → confirm → copy in → optionally re-run gates against the repo. The state machine is the one design choice I find myself most pleased with. There is no step counter, no pause flag, no “current phase” field. Phase is **derived from the presence of fields** in `evolution.yaml`: - empty `variants` → **ready** to run - has `variants`, no `winner` → **running** - has `winner`, no `applied` → **evaluated** - has `applied` → **done** Each skill reads the yaml on entry, infers what phase the evolution is in, and refuses to run outside its own. Capture won’t recreate a brief that already exists. Run won’t re-rank a finished evolution. Apply won’t run without a winner. The skills are stateless; the disk is the source of truth. Crash mid-batch, restart, and the next invocation picks up exactly where the files left off. ### The contracts that make the scoring honest A genetic loop with sloppy scoring is just an expensive random walk. Agent-evolutions leans hard on three contracts that keep selection grounded. **Gates are binary.** Each gate is a shell command. Exit 0 means pass, anything else means fail. A variant that fails any gate is _excluded from ranking entirely_ — no partial credit, no negotiation. Gates are how the framework refuses to compare “fast but broken” against “correct and slow”. **The rubric is numeric.** Each rubric axis has a `direction` (minimize or maximize) and a weight. The composite score is a rank-normalized weighted mean across axes — robust to outliers, well-defined under partial data, and crucially **recomputed on read by the run skill, never persisted** in the yaml. The yaml stores raw rubric values; the leaderboard view derives from them. You cannot accidentally enshrine a stale score because there is no stored score to enshrine. **Criteria are frozen up front.** This is the load-bearing rule. Gates and rubric axes are agreed during the Capture phase and then locked. The run skill refuses to add a new rubric axis after results come in. Without that rule, the evaluator silently turns into a rationalization engine — adding the axis on which the variant the agent likes anyway happens to win. The unstated cousin of these contracts: no silent retries, no narrative scoring. A failing gate stays failed. A missing measurement is missing, not interpolated. The leaderboard is allowed to be ugly. ### The file-based sub-agent contract Each variant runs as its own sub-agent, in its own workspace (a `git worktree` by default), starting from HEAD. The contract between the parent and each child is not chat — it is a file. ![Relationship between Agent Evolutions Concepts](/static/img/articles/agent-evolutions-stop-guessing-the-design-evolve-it/02.webp) Relationship between Agent Evolutions Concepts Every variant sub-agent writes variants/v\/result.json, validated against a [JSON Schema](https://github.com/fmind/agent-evolutions/blob/main/result.schema.json). The parent reads files, not chat output. Four properties fall out of that decision: 1. **Crash-survivable.** A child that finishes its work before the parent gets a chance to read its reply still leaves its result on disk. Partial batches don’t disappear. 2. **Audit trail.** Every variant’s exact result lives in git. You can re-rank a historical evolution from its files alone, weeks later. 3. **No JSON-in-chat parsing.** The classic failure mode where a sub-agent’s reply is “JSON-ish” — with a friendly paragraph in front, or three backticks, or a stray comma — never enters the pipeline. The schema validates a file. 4. **Parallelism stays clean.** Six sub-agents writing to six different files do not pollute one shared transcript. No interleaved thinking, no merge step. I would not bother spelling this out if I had not lost an afternoon, more than once, to the JSON-in-chat trap on a previous prototype. File-based sub-agent I/O is the kind of detail that looks pedantic on slide one and saves the whole loop by slide four. ### A walkthrough — optimize cli startup The shipped example trims a Node.js CLI’s cold-start time on — version. The brief pins one gate set (build green, tests green) and one rubric axis (median of five hyperfine runs, minimize). ```console $ /new-agent-evolution "optimize cli startup" → discusses objective, gates, rubric, scope, budget → on "looks good", commits .agents/evolutions/1-optimize_cli_startup/ $ /run-agent-evolution 1 → gen 1: seeds 6 diverse hypotheses — lazy-load, build-time inline, plugin-registry map, polyfill strip, prebundled arg-parser, dotenv short-circuit → spawns 6 sub-agents in parallel; each materializes a worktree, implements its variant, runs gates, writes result.json → ranks: v2 (build-time inline) → 142.0 ms · v1 (lazy-load) → 184.2 ms v4 fails G2 (test regression) → excluded → gen 2: mutates v2 → v7 (v2 + tree-shake help chunk), mutates v1 → v8, crosses v2×v3, adds 1 explore → ... → plateau at gen 4 — top score unchanged across 2 generations → stops at 24/30 evaluated; winner v7 at 98.6 ms $ /apply-agent-evolution 1 → diffs variants/v7/workspace vs live src/, tsdown.config.ts, package.json → "Confirm to apply, or reply abort" → "looks good" → copies files in; re-runs G1 + G2 against the live repo; both green ``` The moment I find the most instructive is the winner. v7 came from generation 2 — it is _v2 plus a tree-shake tweak that only made sense once v2 had won the first round_. Nobody on the team would have written v7 from scratch in generation 1. Evolution found it because the loop conditioned on v2’s win before composing the next batch. That is the entire pitch of genetic search over sampling, in one trace. ### Where this skill set should not run Three patterns where I deliberately do not reach for evolutions, and the skill refuses to seed a brief when it spots them: - **The task is a one-line bug fix, rename, or doc edit.** Just do it. The overhead of a genetic loop is paid in seconds; the task is paid in seconds. The ratio is wrong. - **There is no mechanical measure — not even a proxy.** “Make this prose more engaging” has no shell exit and no number. Use a normal review. - **The user has already decided the design.** “Implement X.” is a levers job, not an evolutions job. Don’t manufacture a search space just to use the tool. If a brief tries to start without a gate or a rubric axis, the Capture skill pushes back: _“we need at least one mechanical check before variants make sense.”_ Either we co-author one, or the evolution doesn’t get a folder. That refusal is on purpose — it is the load-bearing reason variants ranking _means_ something at the end. ### The leverage axis Agent-levers makes one hour of human attention move more — clearer briefs, typed verifiers, lessons that compound, the same fulcrum logic [I wrote about before](https://fmind.medium.com/agent-levers-a-plan-do-check-act-loop-that-makes-coding-agents-finish-what-they-start). The asymmetry is _human effort in, shipped work out_. Agent-evolutions sits on a different axis. The input is not human attention, it is _compute_: six or twelve or twenty-four sub-agents running in parallel, each spending tokens on a variant of the same design question. The output is a defensible answer to _which design to ship_. The fulcrum is the same shape — small effort on one end multiplies into directed work on the other — but the lever is bolted to a different wall. Most teams, today, are spending compute the way a 1990s team spent CPU time on a build: cautiously, one job at a time. The price has dropped, the parallelism has arrived, and the workflows have not caught up. Genetic search on coding tasks is one of the places that gap is widest. The practical nudge I’ll leave you with: next time a _which design wins?_ thread starts on Slack, count the design points raised against the design points measured. If the ratio is wildly skewed toward opinion — and it usually is — that is the shape of a question agent-evolutions was built to swallow. ### The three install paths mirror the three coding agents: ```text # Claude Code /plugin marketplace add fmind/agent-evolutions /plugin install agent-evolutions@agent-evolutions # Gemini CLI gemini extensions install fmind/agent-evolutions # Antigravity CLI agy plugin install https://github.com/fmind/agent-evolutions # GitHub Copilot copilot plugin marketplace add fmind/agent-evolutions copilot plugin install agent-evolutions@agent-evolutions ``` To run these skills in [OpenCode](https://opencode.ai/), place or clone the skill directories under `.agents/skills/` (for project scope) or `~/.agents/skills/` (for global scope). Then, inside a project: ```text /new-agent-evolution # capture the brief (Session 1) /run-agent-evolution <id> # drive the genetic loop (Session 2) /apply-agent-evolution <id> # land the winner (Session 3) ``` The repository is at [github.com/fmind/agent-evolutions](https://github.com/fmind/agent-evolutions). The [`examples/evolutions/`](https://github.com/fmind/agent-evolutions/tree/main/examples/evolutions) directory ships the full `optimize cli startup` walkthrough — brief, variants, scored leaderboard, applied diff — so you can read what the artifacts look like before running anything yourself. Pick a real question. Give it gates and a rubric. Let the loop run while you make coffee. The answer it comes back with will not be the one you would have written on a whiteboard — and that, more often than not, is the point. ### [Agent Levers: A Plan-Do-Check-Act Loop That Makes Coding Agents Finish What They Start](https://www.fmind.dev/articles/agent-levers-a-plan-do-check-act-loop-that-makes-coding-agents-finish-what-they-start/) A coding agent is only as useful as the spec you give it. That sentence sounds like a truism until you watch it play out for an afternoon. Type too little — “add password sign-in” — and the agent improvises. It picks a JWT library you don’t use, skips rate-limiting, ships a flow that works on the happy path and fails the moment you blow on it. Type too much — every constraint, every test name, every file path — and you’ve done the agent’s homework. You are the bottleneck, not the multiplier. I have been trying to find the shape of the work _between_ those two extremes for months. I built [**agent-levers**](https://github.com/fmind/agent-levers) — a small set of [Agent Skills](https://github.com/anthropics/skills) that give the agent a structured **plan → do → check → act** loop on disk — to do the structuring work I kept doing by hand. This is the third post in a thread about the shared `.agents/` ground, after [supagents](https://fmind.medium.com/supagents-one-source-to-rule-your-coding-subagents-) (one subagent source, many targets) and [agent-docs](https://fmind.medium.com/agent-docs-answer-locally-before-the-web) (a curated, local doc reference). Same bet, different layer: this one is about the **workflow** an agent runs _inside_ a non-trivial task. ![Agent Lever: Multiply the agent’s force, divide the human’s effort — https://github.com/fmind/agent-levers](/static/img/articles/agent-levers-a-plan-do-check-act-loop-that-makes-coding-agents-finish-what-they-start/cover.webp) Agent Lever: Multiply the agent’s force, divide the human’s effort — [https://github.com/fmind/agent-levers](https://github.com/fmind/agent-levers) ### The two failure modes of unguided agents After enough sessions, I started noticing the same two failure patterns, no matter which coding agent I drove. **The agent claims success it didn’t verify.** It says “I implemented the rate limiter,” shows a diff, and moves on. You ask “did the test pass?” and discover it never ran, or it ran in an ad-hoc shell, or — the meanest version — it passed in the agent’s session and fails the moment you re-run it from a clean terminal. The status report isn’t lying. It’s just not grounded in a verifier. **The agent runs away.** The first test fails. The second attempt fails. The third attempt fails. Forty minutes later, the diff is twice as large as the original task, and nothing is green. There is no internal “I should stop and ask” trigger because the agent is doing what it was told: keep trying until the test passes. Both failure modes have the same root cause: the loop is _implicit_. The agent makes up the loop on the fly. Each session improvises a new plan, a new sense of done, a new sense of when to give up. The fix I kept reaching for, before I built this, was structure I imposed by hand: a numbered checklist in the prompt, an explicit “run this command after each change,” a polite “stop and ask if you’ve failed three times in a row.” It worked. It also lasted exactly one session, because the next task started from a blank prompt. Agent Levers is that structure, externalized. Once. ### The lever metaphor The name is the bet. A lever multiplies force on one end while reducing effort on the other. The human pushes on the short side — a one-line brief, plus answers to a handful of clarifying questions — and the agent lifts the long side: investigation, planning, execution, verification, learning. The shape of the work is the [Deming cycle](https://en.wikipedia.org/wiki/PDCA), borrowed from process engineering and applied to a different kind of factory: **plan → do → check → act**. Every non-trivial coding task already follows that shape; it just usually lives in the agent’s head, where you can’t audit it. The point of the framework is to push the loop onto disk, where each step is an artifact that can be reviewed, paused, resumed, and continued by a different session. ![Relationship between Coding Agents and Agent Levers](/static/img/articles/agent-levers-a-plan-do-check-act-loop-that-makes-coding-agents-finish-what-they-start/02.webp) Relationship between Coding Agents and Agent Levers ### Four commands, one folder The whole surface area is four slash commands, three of them user-facing and one of them an internal dispatcher you mostly forget exists. - `/lever-init` — bootstraps the framework on a fresh repo. - `/lever-new <title>` — captures intent. The first turn is _chat-only_ — clarifying questions, proposed enhancements, a draft brief inline. No files land until you agree. - `/lever <id>` — runs the chain. Plan → do → check → act, autonomously until a pause or a terminal state. - `/lever-status [<id>]` — read-only inspector; lists levers, shows one in detail, or cancels one. Everything else lives in two files, per lever, on disk: a `lever.yaml` that holds the machine state, and a `LEVER.md` that opens with a TL;DR and grows narrative sections as each step runs. Half a day of agent work, audited in twenty lines. ### The two-session split The thing I keep coming back to is the deliberate split between `/lever-new` and /lever. `/lever-new` is high-context: you and the agent in chat, narrowing down what you actually want. The agent reads project conventions, asks clarifying questions, proposes enhancements you didn’t ask for but probably want (“you specified Google OAuth — want GitHub too?”), drafts the brief inline. No files land until you say “looks good.” `/lever` is mechanical: a fresh session, reading the state, executing the matching step, advancing the pointer. The dispatcher loads only the step it needs — never all four at once. The agent’s context stays clean across the chain. The two-session split has a property I value enough to call out: **brief-time is the only place the human is in the loop**. Once you run `/lever 1`, the chain runs to completion (or to a structural pause) without re-asking what you wanted. If the brief was crisp, you don’t see the chain again until there is a diff to review. I cannot overstate how much I prefer this to the “agent asks twenty questions while implementing” rhythm. The cost of a sharp brief is paid once, up front, in the cheapest part of the process. Everything after compounds on that investment. ### The four steps, briefly **Plan** reads the brief, investigates the codebase and external docs itself — `grep`, reading tests, pinning dependency versions, fetching authoritative docs — and writes a small set of _acceptance criteria_, each paired with a typed verifier (a shell command, a screenshot to confirm, or a manual rubric). It also sets a budget — iterations, minutes, failed-streak — so the next step has an explicit leash. The hard rule on verifiers is: **test the behavior, not the agent’s past actions**. A criterion that says “function `loginHandler` exists in `src/routes/auth.ts`” is bookkeeping — the agent can satisfy it by writing the function and then grepping its own diff. A criterion that says `POST /auth/login` returns a JWT on valid credentials is verification — it passes when the handler _works_. **Do** drives the criteria one at a time. It picks the first failing one, makes the smallest change that should flip it (TDD-style by default for shell verifiers), runs the verifier, and logs the result. Out-of-scope ideas get parked, not silently bundled into the diff. When the budget runs out, the loop stops with state on disk — raise the cap, narrow scope, or hand off; prior passes stay green. **Check** is the step that earns the framework its keep. It re-runs every verifier from a _fresh shell_ — not the agent’s session — to catch criteria that secretly relied on cached state. Then it does something I have not seen anywhere else: a **chain audit**. It walks the trail from brief to plan to events to diff, asking at each handoff whether the next step honored the previous one. The silent-killer audit is the last hop — does the diff actually deliver what the brief asked for, or did the agent build something adjacent? Tests can pass while intent quietly drifts; the chain audit is what catches that. **Act** runs only if check surfaced _hints_ — moments where a decision deserved more guidance, or where a recurring pattern started to show. It picks the right surface for each: a project rule lands in `AGENTS.md`, a skill flaw lands in the skill, work that isn’t a rule becomes a follow-up /lever-new. The edits arrive as proposals in the working tree, reviewed in the same PR as the implementation. **The next lever inherits the rule, not the mistake.** This last property is the one I find most underappreciated: the framework has a built-in mechanism for the agent’s workflow to improve. A gotcha that surfaces three times in three separate levers earns its way into `AGENTS.md` by being staged three times. You don’t have to remember to tell the agent about it next session. ### A walkthrough on a real task Two human inputs, two sessions, one finished feature. ```console $ /lever-new add password sign-in → first turn: chat-only — clarifying questions, proposed enhancements, draft brief → "rate-limit policy for /auth/login? OAuth uses 5/min/IP — match it?" → "looks good" → captures the brief on disk → "Brief captured. In a new session, run /lever 1 to start the chain." # Fresh session, hours or days later. $ /lever 1 → plan: investigates the auth + middleware code, writes 4 criteria with shell verifiers → do: C1 pass · C2 pass · C3 fail (limiter not wired) · C3 pass · C4 pass → check: rerun from fresh shell — all green chain audit: Brief→Plan · Plan→Do · Do→Result · Result→Intent — all pass 2 hints surfaced → act: hint 1 → AGENTS.md +5/-0 (explicit "attach middleware" sub-step in TDD) hint 2 → recommend /lever-new enforce_route_middleware_lint → "Ran: plan → do → check → act. 4/4 passed. 2 hints landed. Done — diff is staged." ``` The interesting moment is the limiter miss at C3 — the kind of off-by-one that on a normal “agent, implement this” run becomes a Slack thread two days later, when QA notices the rate limit doesn’t fire. Here, it surfaces as a single fail row, recovered in the next iteration, lifted into a hint by check, and codified as a rule by act. That is the whole pitch in one trace: the loop catches its own mistakes before the human sees them, and the lessons compound for next time. ### Install and try it The three install paths mirror the three coding agents: ```text # Claude Code /plugin marketplace add fmind/agent-levers /plugin install agent-levers@agent-levers # Gemini CLI gemini extensions install fmind/agent-levers # GitHub Copilot (CLI) copilot plugin marketplace add fmind/agent-levers copilot plugin install agent-levers@agent-levers ``` Then, inside a project: ```text /lever-init # bootstrap the framework /lever-new <title> # capture a new task (Session 1) /lever <id> # run the chain (Session 2) /lever-status [<id>] # inspect or cancel ``` The repository is at [github.com/fmind/agent-levers](https://github.com/fmind/agent-levers). The [`examples/levers/`](https://github.com/fmind/agent-levers/tree/main/examples/levers) directory ships two worked walkthroughs — a happy path and a scenario where check routes back to do — so you can see what the artifacts look like before you run anything. ### The lever effect A real lever’s mechanical advantage is a ratio: force out per unit of effort in. For a coding agent, the equivalent is **verified, intent-aligned work shipped per minute of human attention spent**. Clear briefs, typed verifiers, bounded loops, on-disk state, lessons that compound — each one bends that ratio in the same direction. We are not trying to build an agent that needs us less. We are building the fulcrum that lets the same hour of human attention move more. Give [**agent-levers**](https://github.com/fmind/agent-levers) a try — the lever is real when you can feel the asymmetry. ### [Agent Docs: Answer Locally Before the Web — A Shared Reference for Every Coding Agent](https://www.fmind.dev/articles/agent-docs-answer-locally-before-the-web-a-shared-reference-for-every-coding-agent/) Pair-program with a coding agent for an afternoon and you start noticing a quiet, recurring tax. Every time the agent reaches for a flag, a config field, or a CLI subcommand, it fires off a web search. Then another. Then a third when the second one returned a 2023-era blog post for a tool you upgraded last week. Each round-trip is a few seconds, a few tokens, and — worse — a chance for the agent to pick a plausible-looking answer from the wrong version of the docs. Multiply that by every session, every contributor, every agent on the team, and the tax is no longer quiet. I built [**agent-docs**](https://github.com/fmind/agent-docs) to make my agents answer locally before they answer remotely. It is a small set of file-based [Agent Skills](https://agentskills.io/home) that seed, read, and refresh a curated `.agents/docs/` reference for the tools your project actually uses. One install, three commands, three coding agents ([Claude Code](https://code.claude.com/docs/en/overview), [Gemini CLI](https://github.com/google-gemini/gemini-cli), [GitHub Copilot](https://github.com/features/copilot)) — the same answer for the same question, every run. ![Agent Docs: Project Docs that Answer Locally — https://github.com/fmind/agent-docs](/static/img/articles/agent-docs-answer-locally-before-the-web-a-shared-reference-for-every-coding-agent/cover.webp) Agent Docs: Project Docs that Answer Locally — [https://github.com/fmind/agent-docs](https://github.com/fmind/agent-docs) This is the follow-up to my [previous post on supagents](https://fmind.medium.com/supagents-one-source-to-rule-your-coding-subagents-) — same bet on the shared `.agents/` ground, applied to a different layer of the stack. ### The hallucination tax has a quieter cousin When people talk about LLM failure modes, they usually mean hallucination — confident statements that turn out to be invented. That’s real. But there’s a quieter failure mode I see far more often in practice: **lossy retrieval**. The agent doesn’t make things up. It does its honest best and lands on the wrong page. A search for “Gemini CLI hooks” returns a blog post that pre-dates the feature. A search for “Cloud Run revisions” returns the 2022 docs, before the rewrite. The agent reads what’s in front of it, summarizes correctly from the source it found, and produces an answer that is technically faithful to a stale or low-quality input. You don’t catch lossy retrieval by reading the agent’s output. You catch it three turns later, when the implementation doesn’t compile against the current SDK and you trace it back to a flag that was renamed in v0.4. The fix that occurred to me, after the third time the same agent web-searched the same `gemini settings.json` page, was almost embarrassingly simple: **stop searching for what we already know**. Pin the URLs once. Make every agent on the project read from the same shortlist before it touches the web. ### The Code Wiki nudge I won’t pretend I came up with the wiki framing in a vacuum. [Code Wiki](https://codewiki.google/) — a recent project out of Google Labs — was the nudge. The pitch resonated immediately: maintain a curated, version-aware knowledge base that a coding agent can query first, instead of asking it to re-discover the same facts every session. What I wanted, though, was different in three specific ways. I wanted it **local** — committed to the repo, diffable in a PR, not behind a service. I wanted it **scoped to the project** — only the tools and services this codebase actually depends on, nothing else. And I wanted it to work across all coding agents I use daily, not just one. So agent-docs is my take: a tiny on-disk reference set, owned by the project, read by any agent that respects the [`AGENTS.md`](https://agents.md/) convention. ### What lives in `.agents/docs/` After you run the gather skill, your project carries a small new directory: ```text .agents/docs/ ├── INDEX.md ├── gemini-cli/ │ ├── DOC.md │ └── HOOKS.md └── cloud-run/ └── DOC.md ``` `INDEX.md` is a one-line-per-topic dispatch table. Each `DOC.md` is the entry point for a topic, and large subsystems can split into sibling files (`HOOKS.md`, `EXTENSIONS.md`) with their own refresh clocks. The crucial design choice is what each `DOC.md` is — and is not. It is a **navigation map**, not a copy of the docs. Concretely, every `DOC.md` carries: 1. A one-paragraph summary of the topic and where it fits. 2. A terse list of key concepts (settings file, slash commands, hooks, MCP servers). 3. A **documentation map** — the upstream sidebar, captured as one-line link entries grouped by section. Each entry is `[Page title](url)` — short hook on what’s there. 4. An optional Patterns section with paste-ready snippets, capped at five per topic, each tied to an inline \_Source: …\_ link. The pattern I keep coming back to is: _land the agent on the right upstream URL in one hop, then fetch upstream if it needs depth_. Not every fact gets captured locally — only the navigation, plus a handful of high-frequency snippets when the team decides they’re worth it. ### Three skills, one small loop The whole thing is three skills. They are deliberately small and they compose into a loop you can reach for without thinking. ![Relationships between Coding Agents, Skills, and Local/Remote Sources](/static/img/articles/agent-docs-answer-locally-before-the-web-a-shared-reference-for-every-coding-agent/02.webp) Relationships between Coding Agents, Skills, and Local/Remote Sources ### `/gather-agent-docs` — seed the reference `/gather-agent-docs` inspects your project’s manifests (`package.json`, `pyproject.toml`, `go.mod`, `cloudbuild.yaml`, `apphosting.yaml`, `mise.toml`) and picks three to eight real topics. It does not guess at what the project “might” use; it picks what the project _declares_. For each topic, it fetches one upstream nav source — the docs sidebar, the README’s docs section, or the sitemap as a fallback — and writes a `DOC.md` with the navigation map mirroring how upstream organizes itself. `INDEX.md` gets one dated line per topic. The default capture tier is intentionally light: title + URL + one-line hook for every page in the upstream nav. That’s it. Paste-ready snippets are opt-in (`/gather-agent-docs deepen gemini-cli`), capped at five per topic, and only added when the team actually reaches for them often. ### `/use-agent-docs` — read before searching This is the skill that earns its keep on every run. Before the agent reaches for the web on a topic that’s part of the project, it reads `.agents/docs/INDEX.md`, opens the matching `DOC.md`, scans the documentation map, and fetches the right upstream URL directly. If the index is missing, it’s a no-op — the agent falls through to normal research. If a captured snippet covers the question, the agent uses it (each one carries its upstream source link, so it can verify). If the answer needs flag-level depth, it goes one hop upstream to a URL it already knows is the canonical one. Crucially, this is the skill I do not have to remember to invoke. It’s wired as a trigger: “before web-searching for a project topic, check `.agents/docs/`”. Once installed, the agent reaches for it on its own. ### `/refresh-agent-docs` — keep it honest A curated doc set that is allowed to rot is worse than no doc set, because it lies with confidence. `/refresh-agent-docs` is the gardener. Every `DOC.md` opens with YAML frontmatter: ```yaml --- last_verified: 2026-05-08 upstream_commit: 9f3c2b1 sources: docs: https://google-gemini.github.io/gemini-cli/ repo: https://github.com/google-gemini/gemini-cli changelog: https://github.com/google-gemini/gemini-cli/releases --- ``` The refresh skill reads that metadata and tries cheap paths first. If `upstream_commit` matches upstream HEAD, the page is verified by definition — bump the date, move on. If the changelog has no entries newer than `last_verified`, same outcome. Only when those short-circuits miss does it fall through to a link-rot scan, then a full re-fetch. The richer the frontmatter, the more refreshes complete in a single HTTP call. When a page can’t be re-verified this session, the skill stamps a visible `> ⚠ Stale` banner at the top so future agents see the warning before they trust the contents. This is what I mean by **honest staleness**. The doc set is allowed to age — that’s fine — but the age is always declared, never hidden. The `[YYYY-MM-DD]` next to each topic in `INDEX.md` is the _oldest_ verification date across the topic’s files, so the index shows the staleness floor at a glance. ### A walkthrough on a real project Here is what a typical session looks like, lightly compressed from a real run on a Cloud Run + Gemini CLI project: ```console $ /gather-agent-docs → inspects manifests, picks: gemini-cli, cloud-run, firebase-app-hosting → fetches one upstream nav per topic → writes .agents/docs/INDEX.md and 3 DOC.md files → "Seeded 3 topics under .agents/docs/. Review the diff before committing." $ /gather-agent-docs deepen gemini-cli → adds up to 5 Patterns snippets to gemini-cli/DOC.md, each with _Source: …_ → bumps last_verified # Later, mid-session, the agent is asked to wire up a Gemini CLI hook… $ "Can you add a pre-tool hook that blocks rm -rf?" → /use-agent-docs fires implicitly → reads INDEX.md, opens gemini-cli/DOC.md, scans the documentation map → finds gemini-cli/HOOKS.md, opens it, picks the snippet for pre-tool hooks → writes the hook against the current schema, not a 2024 blog post # A month later… $ /refresh-agent-docs gemini-cli → smart path: upstream_commit moved; scans changelog since last_verified → finds one entry touching hooks; re-fetches HOOKS.md's snippet sources → repairs two renamed links in DOC.md's documentation map → "gemini-cli: links repaired (DOC.md), snippet edits (HOOKS.md)." ``` The two things I notice consistently after running this for a few weeks: 1. **Round-trips collapse.** The “search → skim → search again” loop turns into a single fetch. The agent’s output shows up faster because there’s less network in front of it. 2. **Answers stop drifting between sessions.** Three contributors asking the agent “where is X configured?” get the same upstream page, because the agent reads the same map. ### Local-first, on purpose A few people asked me, when I described this, why I didn’t just point the agent at the upstream docs site directly and let it search. Two reasons. First, _scope_. The upstream docs for any non-trivial tool are an ocean. The right answer for a given project is a narrow shoreline — the flags this team uses, the patterns this codebase follows, the canonical URL we trust. Curating the map shrinks the search space from “every page on the site” to “the dozen pages we actually touch”. Second, _commit-ability_. `.agents/docs/` lives in git. It travels with the code. It diffs in PRs. When a contributor updates the Gemini CLI version, the PR can include the refreshed map alongside the version bump, reviewed in the same place. A hosted service can’t be diffed in your PR. There is a real cost: someone on the team has to run `/refresh-agent-docs` periodically, and someone has to decide when a snippet is worth capturing. That cost is the point. It is the same cost we pay to maintain a `README.md` that doesn’t lie — and we pay it because the alternative is worse. ### Where this sits in the `.agents/` stack If you read my [supagents post](https://fmind.medium.com/supagents-one-source-to-rule-your-coding-subagents-), the framing here will feel familiar. I keep finding the same pattern: a thing that should be one source, fragmented across N coding-agent harnesses, that we can unify in the shared `.agents/` ground. Supagents handles **subagent personas** — one markdown source compiled to `.claude/agents/`, `.gemini/agents/`, .github/agents/. Agent Docs handles **project knowledge** — one `.agents/docs/` folder, read by any agent that follows the [AGENTS.md convention](https://agents.md/) (which now includes Claude Code, Gemini CLI, and GitHub Copilot). These are two slices of the same bet: the future of coding agents is not a winner-takes-all framework. It’s a shared file-system layout, with small composable conventions, that every tool reads in its own way. The friction goes away when we stop reinventing storage formats and start respecting a common root. ### Install and try it The three install paths mirror the three coding agents: ```text # Claude Code /plugin marketplace add fmind/agent-docs /plugin install agent-docs@agent-docs # Gemini CLI gemini extensions install fmind/agent-docs # GitHub Copilot (CLI) copilot plugin marketplace add fmind/agent-docs copilot plugin install agent-docs@agent-docs ``` Then, inside a project: ```text /gather-agent-docs # seed .agents/docs/ /use-agent-docs # fires implicitly on web-search triggers /refresh-agent-docs # run periodically -- once a sprint is enough ``` The repository is at [github.com/fmind/agent-docs](https://github.com/fmind/agent-docs). The [`examples/`](https://github.com/fmind/agent-docs/tree/main/examples) directory ships a worked sample of what gather produces — two topics, one with a subtopic split — so you can see the output shape before you run it. ### Small tools, on purpose I keep coming back to the [Linux philosophy](https://fmind.medium.com/ai-agents-as-an-operating-system-rediscovering-the-linux-philosophy-f0e76f29ebdb) when I build things in this space: small tools, one job, composable. Agent Docs is three skills (gather, use, refresh) that touch one folder (`.agents/docs/`) and respect one shared root (`.agents/`). That’s the whole surface area. The bet is that this is enough. A curated map is cheaper than a clever retrieval system, the same way a README.md is cheaper than an auto-generated wiki. The reason README.md survived every documentation-platform fad for two decades is that it’s a flat file that travels with the code and reads correctly without a runtime. .agents/docs/ is the same shape, aimed at the agent instead of the human. If you are running coding agents on a real project — and tired of watching them re-search the same SDK every afternoon — give it a try. If it saves you the third web round-trip on a Friday, it has earned its keep. ### [Supagents: One Source to Rule Your Coding Subagents](https://www.fmind.dev/articles/supagents-one-source-to-rule-your-coding-subagents/) The AI providers are in a horse race. Every quarter, a new model lands, a new harness ships, and the gap between “the best tool for _this_ task” and “the best tool overall” keeps moving. As developers, we have already stopped picking sides — we pick the right tool for the job, and we switch between them constantly. I noticed this pattern hardening in my own workflow. I drive [Claude Code](https://docs.claude.com/en/docs/claude-code/overview) for deep refactors, [Gemini CLI](https://github.com/google-gemini/gemini-cli) for long-context exploration, and [GitHub Copilot](https://github.com/features/copilot) for the IDE flow. Each one has carved out a niche. Locking myself to a single provider would be a serious productivity bottleneck. The catch: every one of these tools has reinvented the same primitive — the [**subagent**](https://geminicli.com/docs/core/subagents/) — and given it a slightly different shape. I got tired of maintaining the same persona five times. So I built [**supagents**](https://github.com/fmind/agent-supagents): a single markdown source, compiled to every coding agent I use. ![Compile a single AI supagent into multiple AI subagents — https://github.com/fmind/agent-supagents](/static/img/articles/supagents-one-source-to-rule-your-coding-subagents/cover.webp) Compile a single AI supagent into multiple AI subagents — [https://github.com/fmind/agent-supagents](https://github.com/fmind/agent-supagents) ### The subagent fragmentation problem Subagents are one of the best additions to modern coding agents in the last year. They isolate workloads, keep MCP servers and tools out of the main session’s context, and let you ship richer instructions through progressive disclosure. Claude Code has [agents](https://docs.claude.com/en/docs/claude-code/sub-agents), Gemini CLI has them, [Copilot in VS Code](https://code.visualstudio.com/docs/copilot/customization/custom-instructions) has them, [Cursor](https://docs.cursor.com/) has them, [OpenCode](https://opencode.ai/) and [Kilo Code](https://kilocode.ai/) have them. The persona I want is identical across all of them: _“You are a code-investigation expert. You read first, edit never, and you cite file:line for every claim.”_ The configuration is not. Each platform wants a different YAML frontmatter shape, a different directory layout, a different file extension. Claude wants `tools: Read, Grep, Glob` and a model field. Gemini wants `kind: local` and an `mcp_servers` block. Copilot wants `target: vscode` and a list of allowed models. The bodies are roughly the same prose, but the metadata diverges. I tried hand-syncing for a while. It does not scale. You tweak the persona once on a Friday afternoon, forget to copy it across, and three weeks later you are debugging why “the same agent” behaves differently on two laptops. ### The common ground that almost works The good news is that the industry is converging on a shared root: the [`AGENTS.md`](https://agents.md/) file at the repository root, and the `.agents/` directory next to it. [GitHub now accepts `AGENTS.md`](https://docs.github.com/en/copilot/how-tos/configure-custom-instructions/add-repository-instructions) as a valid Copilot instruction file. [Gemini CLI reads skills out of `.agents/skills/`](https://geminicli.com/docs/cli/skills/). Claude Code, Cursor, and OpenCode all respect similar conventions. This is real progress. A single `AGENTS.md` and a single `.agents/skills/` tree can drive a Copilot session, a Gemini run, and a Claude Code job — without touching anything tool-specific. But subagents have not landed in that shared ground yet. Each tool still keeps its agent definitions in its own private folder: `.claude/agents/`, `.gemini/agents/`, `.github/agents/`, and so on. There is no agents.md-equivalent for subagents. That is the gap supagents fills. ### How supagents works Supagents is a small Python CLI. You write _one_ markdown file per agent in .agents/supagents/\<name\>.md (project) or ~/.agents/supagents/\<name\>.md (global). The CLI compiles that source into one output per target tool, dropping each compiled file into the location the target expects. ![One supagents source compiles into per-target agent files for Claude Code, Gemini CLI, GitHub Copilot, Cursor, OpenCode, and Kilo Code](/static/img/articles/supagents-one-source-to-rule-your-coding-subagents/02.webp) One supagents source compiles into per-target agent files for Claude Code, Gemini CLI, GitHub Copilot, Cursor, OpenCode, and Kilo Code The design is deliberately small. There are only two layers in the source file: shared frontmatter, and per-target blocks. A handful of UPPERCASE directives let you bend the output without leaving the file. ### The source file Here is a real source for an investigation agent: ```yaml --- name: code-investigator description: Explore unfamiliar codebases without editing. CLAUDE: model: sonnet tools: Read, Grep, Glob, Bash GEMINI: kind: local tools: ["*"] mcp_servers: context7: command: npx args: ["-y", "@upstash/context7-mcp@latest"] COPILOT: target: vscode model: ["gpt-5", "gpt-4.1"] APPEND_BODY: | ## Copilot-specific Reference tools with `#tool:<name>`. --- # Code Investigator You are a specialized code-investigation agent. Read the relevant files before answering. Cite `file:line` for every non-trivial claim. Never edit code; if a fix is needed, propose it as a diff in the response. When the user asks "where is X?", search broadly across naming conventions before reporting "not found". ``` The conventions are simple enough that I never need to read the docs again: - **Lowercase keys** at the top level (like `name`, `description`) are shared frontmatter — copied verbatim into every output. - **UPPERCASE keys** at the top level (`CLAUDE`, `GEMINI`, `COPILOT`, `CURSOR`, `OPENCODE`, `KILO`) are target blocks. Each block holds the frontmatter that _only that target_ understands. - **Lowercase keys inside a target block** (`model`, `tools`, …) become that target’s frontmatter, verbatim. Supagents does not translate or normalize fields. If Claude wants `tools: Read, Grep` and Gemini wants `tools: ["*"]`, you write them both — and that is fine, because they live in different blocks. - **UPPERCASE keys inside a target block** are directives — they shape the build instead of the output frontmatter. These two extra directives extend the generation process: - **`OUTPUT`**: override the destination path for one target. Useful when a tool wants the file in an unusual place. - **`APPEND_BODY`**: append target-specific markdown to the shared body. Great for tool-specific syntax (Copilot’s `#tool:` references, for example) that would confuse the other agents. The body of the markdown — everything after the frontmatter — is the persona. It is shared across all targets unless `APPEND_BODY` extends it. This is the part you write _once_ and stop hand-syncing. ### Verbatim, on purpose I made a deliberate choice to keep the per-target frontmatter verbatim. Supagents does **not** try to translate `model: sonnet` into Gemini’s equivalent, or rewrite `tools: Read, Grep` into Copilot’s `#tool:` syntax. Every coding tool moves fast, breaks frontmatter conventions on minor releases, and adds new fields I have never heard of. A schema-translation layer would be obsolete before I shipped it. Verbatim has a cost — you write each target block yourself — but it has one big payoff: when Claude Code adds a new field next week, supagents already supports it. The compiler is just a YAML splitter and a file router. ### The CLI Four commands cover the entire workflow: ```text supagents build [SCOPE] # compile sources → outputs supagents clean [SCOPE] # remove orphaned outputs supagents list [SCOPE] # show every source and its outputs supagents init NAME # scaffold a new source from template ``` `SCOPE` is either `project` (the `.agents/supagents/` next to your code) or `global` (the `~/.agents/supagents/` in your home). Skip it and supagents picks based on where you run. The flags I reach for daily: - `--dry-run` — preview every write without touching the filesystem. - `--check` — exit non-zero if any output would change. This is the CI flag. Wire it into a GitHub Action and you get a hard guarantee that committed sources match committed outputs. - `--target CLAUDE` (repeatable) — operate on a single target. Great when you are iterating on a Gemini-only tweak and don’t want to rewrite five files. - `--verbose` — print the unchanged files too. Helpful when something is mysteriously not updating and you want to confirm supagents _saw_ the source. Builds are idempotent. If a source has not changed since last build, supagents preserves the output’s mtime so file watchers and pre-commit hooks don’t re-fire on noise. ### Pre-commit and CI integration I run supagents in two places. Locally, as a pre-commit hook: ```yaml # .pre-commit-config.yaml - repo: https://github.com/fmind/agent-supagents rev: v1.0.0 hooks: - id: supagents-build ``` This catches the common mistake of editing `.agents/supagents/code-investigator.md` and forgetting to commit the regenerated `.claude/agents/code-investigator.md`. In CI, I run `supagents build --check` as a separate step. If someone edits an output by hand instead of editing the source, the check fails and the PR cannot land. Outputs are always derived; sources are always authoritative. ### A configuration file when defaults are not enough By default, supagents knows where each of the six built-in targets wants its files: - **CLAUDE** — project `.claude/agents/`, global `~/.claude/agents/`, suffix `.md` - **GEMINI** — project `.gemini/agents/`, global `~/.gemini/agents/`, suffix `.md` - **COPILOT** — project `.github/agents/`, global `~/.copilot/agents/`, suffix `.agent.md` - **CURSOR** — project `.cursor/agents/`, global `~/.cursor/agents/`, suffix `.md` - **OPENCODE** — project `.opencode/agents/`, global `~/.config/opencode/agents/`, suffix `.md` - **KILO** — project `.kilo/agents/`, global `~/.config/kilo/agents/`, suffix `.md` When a tool moves its agents folder — and it will — you don’t need to wait for a supagents release. Drop a `~/.config/supagents/config.yaml` and override only what you need: ```yaml targets: claude: global_path: ~/.config/claude-code/agents project_path: .claude-code/agents ``` Anything you omit falls back to the bundled defaults. The same mechanism lets you register a brand-new target without touching the source code, which I expect to use the moment the next coding agent ships. ### Why it matters The bigger bet is that subagents will eventually land in the shared `.agents/` ground, the way `AGENTS.md` already has. When that happens, supagents becomes a transition tool — and that is the point. I would rather build a bridge that becomes obsolete than wait for every vendor to converge on a standard while my workflow rots. In the meantime, supagents lets me do three things I could not do before: 1. **Write a persona once.** No more `cmd-find-and-replace` across five YAML files. 2. **Ship subagents in a project.** Commit the `.agents/supagents/` source, list `supagents build` in `CONTRIBUTING.md`, and every contributor — whatever coding agent they use — gets the same agents. 3. **Promote my best agents to global.** When a project-scoped agent earns its keep, I move it to `~/.agents/supagents/` and it follows me into every repo. If you are running more than one coding agent — and most of us are, whether we admit it or not — give it a try: ```bash uv tool install supagents supagents init code-investigator supagents build --dry-run ``` The repository is at [github.com/fmind/agent-supagents](https://github.com/fmind/agent-supagents). Issues and pull requests welcome — especially for new target adapters. I want this to outgrow my own workflow. The [Linux philosophy](https://fmind.medium.com/ai-agents-as-an-operating-system-rediscovering-the-linux-philosophy-f0e76f29ebdb) taught us that small tools beat big protocols. The same is becoming true for AI agents: a tiny CLI that does one thing — compile a single source to many targets — beats yet another framework that tries to own the whole stack. Supagents is my attempt at the _one thing well_ part. The “combine them to solve complex problems” part is up to you. ### [AI Agents as an Operating System: Rediscovering the Linux Philosophy](https://www.fmind.dev/articles/ai-agents-as-an-operating-system-rediscovering-the-linux-philosophy/) We are building the most advanced AI systems in history, yet the best way to control them relies on paradigms from the 1970s. I see developers wrestling with this friction daily, trying to bridge the gap between bleeding-edge models and legacy toolchains. As of 2026, a simpler and clearer trend is emerging: **the Command Line Interface (CLI) is becoming the most practical foundation for building agents**. Tools like [OpenClaw](https://openclaw.ai/) and the [Google Workspace CLI](https://github.com/googleworkspace/cli) demonstrate this well. Handing an agent a raw shell and a file system is often faster and more reliable than wrapping it in a complex protocol. As Eric Holmes noted in [_MCP is dead. Long live the CLI_](https://ejholmes.github.io/2026/02/28/mcp-is-dead-long-live-the-cli.html), modern LLMs already excel at using standard command-line utilities. These tools are lightweight, trivial to debug, and compose naturally. To understand why this approach works, we need to look at where agent protocols struggled, and why the path forward means returning to the Linux philosophy. ![Photo by Kevin Horvat on Unsplash](/static/img/articles/ai-agents-as-an-operating-system-rediscovering-the-linux-philosophy/cover.webp) Photo by [Kevin Horvat](https://unsplash.com/@hidd3n?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### The Problem with MCP (Model Context Protocol) [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) was one of the first major agent protocols to emerge. Heavily inspired by how VS Code extensions work, its goal was noble: create a standardized way to expose all available resources and tools to an LLM. In practice, however, MCP suffers from severe architectural flaws for everyday agentic workflows: - **Context Bloat:** MCP typically exposes all tools and their schemas at once. Massive tool descriptions eat up valuable context windows. On my Gemini CLI, simply booting up and saying “Hello” cost 50k input tokens. - **Deployment Mess:** The protocol tries to handle too many scenarios. It combines local runners (e.g., Docker, uvx, npx, …), remote execution, API keys, and OAuth into a single, often fragile deployment model. - **Redundant Authentication:** As Holmes pointed out, you often have to re-authenticate specifically for the MCP and define an entirely new set of tools, needlessly duplicating your existing environment. - **Verbose Returns:** Results are returned directly in the protocol’s format to the LLM, bloating the context even further with massive JSON structures rather than clean, human-readable text. ### 🛠️ MCP: Experience from the Field - **Development:** Personally, I don’t use any MCPs on my local machine anymore. It’s too bloated. I refuse to re-authenticate to yet another tool wrapper, and micromanage them by manually selecting which tools the agent should use for a given task. - **Production:** I often tell my team that MCPs are like the old regex joke: _You had a problem that required MCP, now you have two problems._ They lack maturity, and it’s rarely clear if a specific MCP was built for a slick local dev experience or for robust production. They are incredibly hard to debug, and I frequently find myself monkey-patching them just to keep pipelines running smoothly. ### A2A (Agent-to-Agent) Is Not a Tooling Protocol For a long time, I believed [Agent-to-Agent (A2A)](https://a2a-protocol.org/latest/) communication would emerge as the compelling alternative to tool-binding protocols like MCP. While A2A remains the best way to orchestrate agents across different domains, it is not a protocol meant for low-to-medium level tool execution: - **The Double-Token Tax:** If you have a client agent delegating a simple task to a remote agent via A2A, you are paying token costs and incurring latency for _two_ LLMs in the loop. For low-level execution, the cost far outweighs the benefits. - **Rigid Abstractions:** The remote agent comes with its own baked-in instructions and tools. While great for isolation, it’s highly detrimental if the local client agent needs to dynamically override or tweak how the remote task is executed. - **Network Overhead:** A2A inherently treats everything as a remote network call. This is powerful for distributed systems, but absolute overkill for local development tasks. ### 🛠️ A2A: Experience from the Field - **Development:** I don’t know anyone actually using A2A in their local development loop. Providers like GitHub tend to build their own proprietary APIs to call remote agents. When you operate inside a closed ecosystem, it’s often faster and easier to use native APIs than to force-fit an external protocol. - **Production:** I think A2A is incredibly promising here, and we already leverage it in production. It’s a fantastic approach for large organizations where you need to mirror complex corporate structures, allowing agents to delegate high-level tasks to specialized “worker” agents at scale. ### Agent Skills Are Too Abstract I previously thought [Agent Skills](https://agentskills.io/home) were going to be the golden hammer — a concept I explored in [_MLOps Coding Skills: Bridging the Gap Between Specs and Agents_](https://fmind.medium.com/mlops-coding-skills-bridging-the-gap-between-specs-and-agents-4c8170570eba). I was wrong. - **Premature Abstraction:** Skills are designed to be reusable. The trap is making them overly verbose on Day 1, trying to account for edge cases that don’t exist yet. - **Context over Skills:** On a new project, you are much better off using an [`AGENTS.md`](https://agents.md/) file, system instructions, or [context files like the 5XP framework](https://fmind.medium.com/the-5xp-framework-steering-ai-coding-agents-from-chaos-to-success-83fbdb318b2b). They are vastly easier to operate and tweak. - **Incomplete Taxonomy:** Structuring everything as a “skill” is a flawed mental model. Not everything is a skill. Way-of-working, product objectives, and business context cannot be neatly packaged into executable skills. - **The “Static Fit” Problem:** When you import open-source skills or skills written by others, they rarely align 100% with your needs. Because they are static, they either fit perfectly, or they break. There is no inherent mechanism for the skill to learn or adapt to the user’s specific workflow. ### 🛠️ Agent Skills: Experience from the Field - **Development:** I use the [5XP framework](https://fmind.medium.com/the-5xp-framework-steering-ai-coding-agents-from-chaos-to-success-83fbdb318b2b) now. I only extract a “skill” _a posteriori_ — after a major refactoring, when I actively want to reuse a specific workflow across multiple projects. - **Production:** Skills provide a clean way to formalize and version-control agent instructions to share among colleagues. They complement A2A beautifully: the A2A protocol provides the workers, while the Skills repository provides their standardized instructions. ### Rediscovering the Linux Philosophy If bloated protocols and rigid abstractions are slowing us down, what is the alternative? We need an approach to agent tooling that is dynamic, composable, and lightweight. Unsurprisingly, the industry is circling back to the [CLI (Command-Line Interface)](https://en.wikipedia.org/wiki/Command-line_interface). For developers, treating the CLI as the primary interface for agents has undeniable benefits: - **Battle-Tested Maturity:** There are thousands of mature, edge-case-tested CLI tools already on the market. Furthermore, LLMs have ingested the `man` pages for these tools; they already know exactly how to use them. - **Shared Environment:** A human can step in and run the exact same command. The environment is already authenticated, and debugging is trivial (standard error messages, exit codes, easily readable flags). - **Zero Bloat:** It’s incredibly lightweight. You don’t need to deploy a wrapper or a daemon. You just execute the command. - **Infinite Composability:** Agents can natively pipe `find`, `grep`, `jq`, and `curl` together, without passing the intermediate tool output to the LLM. - **Language Agnostic:** It doesn’t matter if the underlying tool is written in Rust, Go, Python, or Bash. To the agent, it’s just a command. What we are doing is rediscovering the [**Linux Philosophy**](https://en.wikipedia.org/wiki/Unix_philosophy): 1. Everything is a file (or a text stream). 2. Write small tools that do one thing well. 3. Combine them to solve complex problems. ### 🛠️ Experience from the Field - **Development:** During my day-to-day development, I now rely almost entirely on CLI tools combined with raw text context. It is incredibly liberating. I’m deeply curious to see how far this pure approach can scale. We are even seeing frameworks adapt to this reality; for instance, the Google ADK recently [added native bash tools](https://www.google.com/search?q=https://github.com/google/adk-python/blob/780093f389bfbffce965c89ca888d49f992219c1/src/google/adk/tools/bash_tool.py%23L62) to give agents direct shell access. _Side Note: I finally feel less guilty about the_ [_1283+ commits on my dotfiles_](https://github.com/fmind/dotfiles) _._ ### What’s Next? From Linux to Kubernetes for Agents Right now, the raw CLI is the 80/20 rule of agent development. It gives us maximum leverage with minimum setup, accomplishing 80% of what we need and making developers extremely productive locally. But local development is not the final destination. While giving an agent CLI access is like giving it a personal UNIX terminal, **security and scale are the ultimate blockers for the CLI in production.** You cannot simply hand an autonomous LLM unconstrained bash access to your AWS account or your production database. Real-world deployment requires a rigorous security layer: strict authentication, Role-Based Access Control (RBAC), sandboxing, and immutable audit logs. This is a highly complex problem that raw CLI execution cannot solve safely at an enterprise scale. In traditional software, we didn’t abandon the Linux philosophy to build the cloud; we containerized and orchestrated it. We bridged the gap between a single bash instance and a globally distributed system. We need the exact same evolution for AI. To bridge the gap between heavy, bloated agent protocols and the lightweight-but-insecure raw CLI, we need a new paradigm. We need something that adheres strictly to the composability of Linux, but is built for autonomous systems at scale. **My bet is that we don’t just need another protocol; we need an Operating System for Agents.** Just as the industry needed Kubernetes to safely orchestrate Linux containers across vast server networks, our agents will need an orchestration layer built specifically for AI workloads: - **Process Management:** Spinning agent loops up and down dynamically based on computing load. - **Granular RBAC:** Strict, declarative policies restricting exactly which binaries, data silos, and network endpoints an agent can touch. - **Dynamic Service Discovery:** An Istio-like service mesh for agents, allowing them to route tasks to specialized peers without hardcoded endpoints. - **Message Queues:** Enabling agents to share state and context asynchronously, freeing up expensive context windows. ![Istio Architecture (Source: https://istio.io/latest/docs/ops/deployment/architecture/ )](/static/img/articles/ai-agents-as-an-operating-system-rediscovering-the-linux-philosophy/02.webp) Istio Architecture (Source: [https://istio.io/latest/docs/ops/deployment/architecture/](https://istio.io/latest/docs/ops/deployment/architecture/)) The history of software is wonderfully cyclical. We spent the last few years trying to invent complex new paradigms, only to rediscover the elegance of the Linux philosophy. As we transition from local development into enterprise-grade production, it’s becoming abundantly clear: we’ve finally found the Linux for agents. **The next frontier is building the Kubernetes to run it.** ![Photo by Growtika on Unsplash](/static/img/articles/ai-agents-as-an-operating-system-rediscovering-the-linux-philosophy/03.webp) Photo by [Growtika](https://unsplash.com/@growtika?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [The 5xP Framework: Steering AI Coding Agents from Chaos to Success](https://www.fmind.dev/articles/the-5xp-framework-steering-ai-coding-agents-from-chaos-to-success/) AI Coding — or [Spec-Driven Development](https://github.blog/ai-and-ml/generative-ai/spec-driven-development-with-ai-get-started-with-a-new-open-source-toolkit/) — is a recent trend that amazes as much as it scares IT engineers. We now have access to powerful models capable of writing entire applications, but jumping from “Hello World” to a production-grade enterprise system exposes a glaring bottleneck: **Context.** The hardest challenge isn’t the coding itself; it’s explaining to the AI Coding Agent _how_ to work on the task. How do you capture your style, preferences, constraints, environment, tools, and workflow? In my previous article, [_How I Revamped My Portfolio Website in 5 Nights Using AI Agents_](https://fmind.dev), I shared how setting up explicit context was the secret to steering the AI successfully. I refused to just “[vibe code](https://x.com/karpathy/status/1886192184808149383?lang=en)” and instead relied on firm architectural guidance. In this follow-up article, I present the exact structure I use for my AI-assisted projects. After exploring many iterations and tools, I found that the key to success is providing the AI with the right context in a brutally simple, highly maintainable way. ![Photo by ZHIDA LI on Unsplash](/static/img/articles/the-5xp-framework-steering-ai-coding-agents-from-chaos-to-success/cover.webp) Photo by [ZHIDA LI](https://unsplash.com/@adam_l_ee?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) > _**Note:**_ _To make it easier to adopt, I’ve created the_ [_AI Coding 5xP Template_](https://github.com/fmind/ai-coding-5xp-template) _repository. This repository acts as the single source of truth for the framework, providing a ready-to-use directory structure you can use for your own projects._ ### Context is Everything AI Coding Agents are amazing at parsing documentation, but they lack common sense. Put yourself in the shoes of an AI agent: you are dropped into a closed room, and you don’t know the person you are assisting. You don’t know who you are working for or for what purpose. You just have access to the code and whatever you can infer from it. To steer the agent in the right direction, context is the fuel. It adapts the LLM to your skill level, your objective, and your expectations. But bringing context properly is a true challenge, and I know many engineers who struggle with it. ### The Pitfalls of Current Solutions I have experimented with several approaches to provide context, and each comes with its own trade-offs: - **Autocompletion & Inference**: Tools like [GitHub Copilot](https://github.com/features/copilot) and [Google Antigravity](https://antigravity.google/) are great for getting decent work done by inferring information from your existing codebase. However, they struggle when you start a project from scratch or when you have to rigidly copy-paste instructions over and over again to keep the agent aligned. - **Interactive Scaffolding**: Frameworks like [Spec-kit](https://github.com/spec-kit) and [Conductor](https://github.com/gemini-cli-extensions/conductor) allow you to prepare your context by answering questions. In practice, I found these tools too verbose. They infer a lot of content from your answers that you wouldn’t necessarily write yourself, leading to dilution of the core instructions. - **Protocol Integrations**: The [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is a fantastic standard for extending the capacities of your coding model by securely exposing external tools. But relying purely on MCP can bloat your LLM’s context window. Once you hit 100 tools, the model loses its focus, and you end up micromanaging tool selection instead of writing code. - [**Agent Skills**](https://agentskills.io): Dynamically loading specific skills into your LLM is amazing for quick task execution via lazy loading. However, it’s incredibly hard to generalize skills on day one. It’s often more practical to create specific rules for your current project first, and later refactor the recurring ones into standalone skills. ### My Solution: The 5xP Framework I found that the most sustainable strategy to steer an AI Coding Agent is to use flat, readable Markdown files. I organize them using what I call the **5xP Framework**: Product, Platform, Process, Profile, and Principle. Here is how to break it down. While you can customize it, the default template provides 4 files directly under a `context/` directory: 1. [**`PRODUCT.md`**](https://github.com/fmind/ai-coding-5xp-template/blob/main/context/PRODUCT.md): The objective. What are we building? Who are the users? What is the business logic? This prevents the AI from over-engineering features that don’t align with the goals. 2. [**`PLATFORM.md`**](https://github.com/fmind/ai-coding-5xp-template/blob/main/context/PLATFORM.md): The technical stack. What tools, environment, frameworks, and architectural patterns are we strictly adhering to? 3. [**`PROCESS.md`**](https://github.com/fmind/ai-coding-5xp-template/blob/main/context/PROCESS.md): The workflow. How exactly should the AI collaborate with you? What are the quality assurance standards and AI engagement rules? 4. [**`PROFILE.md`**](https://github.com/fmind/ai-coding-5xp-template/blob/main/context/PROFILE.md) _(Optional but recommended)_: Who are you? Explaining your background, communication style, and expertise prevents the agent from making wrong assumptions. The **5th xP** is the most important: **Principle**. These are your Ten Commandments, and you should put this information in an [`AGENTS.md`](https://github.com/fmind/ai-coding-5xp-template/blob/main/AGENTS.md) file at the root of your project. Acting as the master entry point for the AI, the principles sit at the top, and the file then links to the other 4xPs, explicitly instructing the agent to read them when necessary. > **Rule of Thumb:** Keep each file to **1 page maximum**. You should not have to scroll to read the content. This forces you to be concise, avoiding information overload for both you and the LLM. ### Example Project Structure Let’s look at a fictional project — an open-source AI task manager called `TaskBrain` — and see how this structure is applied in practice based on the template. The repository structure looks like this: ```text taskbrain/ ├── src/ ├── tests/ ├── context/ │ ├── PLATFORM.md │ ├── PROCESS.md │ ├── PRODUCT.md │ └── PROFILE.md └── AGENTS.md ``` Here is exactly how you would wire the [`AGENTS.md`](https://github.com/fmind/ai-coding-5xp-template/blob/main/AGENTS.md) file to act as the master prompt: ```markdown # Agent Guidelines Assume the persona of an expert AI Coding Assistant dedicated to this project. ## Principles 1. **Simplicity First**: Keep dependencies minimal and favor straightforward, readable solutions over complex alternatives. 2. **Proactive Partnership**: Anticipate potential issues and challenge decisions that contradict the project's constitution. 3. **Maintainable Code**: Prioritize code clarity, clean architecture, and consistent formatting across the entire codebase. 4. **Contextual Awareness**: Adhere to the guidelines established in the context files to ensure all contributions align with them. 5. **Security by Design**: Identify and mitigate security issues proactively. Treat data privacy and secrets handling as fundamental requirements. ## Context Navigation Read the following context files as needed to understand the project's core rules and guidelines. Suggest to the user to update the context files as needed to reflect the project's current state. - **[context/PROFILE.md](context/PROFILE.md)** (Who): My background, communication style, and expertise. - **[context/PRODUCT.md](context/PRODUCT.md)** (What): Core objectives, target audience, and business logic. - **[context/PLATFORM.md](context/PLATFORM.md)** (Where): Technology stack, infrastructure, and architectural principles. - **[context/PROCESS.md](context/PROCESS.md)** (How): Workflows, AI engagement rules, and quality assurance standards. ``` By keeping the root [`AGENTS.md`](https://github.com/fmind/ai-coding-5xp-template/blob/main/AGENTS.md) file short and linking to specific sub-files in a `context/` directory, the AI agent can dynamically lazy-load exactly what it needs, precisely when it needs it. ### Field-Tested Results I have thoroughly tested this framework on two distinct projects: [completely revamping my personal website](https://fmind.dev/) and bootstrapping a complex new project centered on AI Agents. In both cases, the results were night and day compared to previous approaches. The true power of this framework lies in its **human-friendliness**: the files are simple, focused, and I can quickly steer the model in a new direction just by tweaking a few bullet points in a Markdown file. Furthermore, it does not bloat the LLM. By providing what essentially acts as a “table of contents” in the root [`AGENTS.md`](https://github.com/fmind/ai-coding-5xp-template/blob/main/AGENTS.md) (similar to exposing agent skills), the model only pulls the context it needs for the task at hand. This naturally enforces a healthy rhythm: you spend **20% of your time thinking** and structuring the context, allowing the AI to smoothly handle the **80% implementation** burden. ### Conclusion Every expert I know keeps iterating on how to collaborate with AI Coding Agents. Many approaches are either far too verbose, drowning the LLM in noise, or far too simplistic, leading to repetitive manual prompting. The 5xP framework is the best trade-off I have found so far: it is brutally simple, easy to maintain natively in Git, and it works across almost every LLM coding environment. It does not take long to set up, but it forces you to think clearly about your project’s shape before you write a single line of code. And in the era of AI engineering, thinking clearly is your greatest superpower. ![Photo by Van Tay Media on Unsplash](/static/img/articles/the-5xp-framework-steering-ai-coding-agents-from-chaos-to-success/02.webp) Photo by [Van Tay Media](https://unsplash.com/@vantaymedia?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) Ready to start? Grab the [AI Coding 5xP Template](https://github.com/fmind/ai-coding-5xp-template) on GitHub to get up and running instantly. ### [How I Revamped My Portfolio Website in 5 Nights Using AI Agents](https://www.fmind.dev/articles/how-i-revamped-my-portfolio-website-in-5-nights-using-ai-agents/) I see many friends and acquaintances generating amazing applications in mere weeks. We are in the midst of a craze of innovation, an era where inspired people can bring their ideas to life without being strictly bounded by the usual constraints of tool mastery. This means there is no time to slack off. The go-getters will not wait for you to catch up. One representative aspect of this fast-moving landscape is your digital presence — your portfolio. As a freelance AI/ML Architect, I realized I needed a space that truly reflected my expertise. In this article, I present the revamp of my own portfolio: [fmind.dev](https://fmind.dev). ![New Homepage of Fmind.dev](/static/img/articles/how-i-revamped-my-portfolio-website-in-5-nights-using-ai-agents/cover.webp) New Homepage of Fmind.dev ### The Old and Dusty Website [My previous website](https://sites.google.com/fmind.dev/fmind-dev/home) was built with Google Sites. It was cheap, highly functional, and effortless to maintain. ![Previous version of Fmind.dev](/static/img/articles/how-i-revamped-my-portfolio-website-in-5-nights-using-ai-agents/02.webp) Previous version of Fmind.dev But functional is no longer enough in 2026, especially for a freelancer who constantly needs to be on the bleeding edge of technology. Google Sites afforded me very little control over styling, strict limitations on SEO, and — crucially — zero ability to easily evolve it using modern AI coding assistants. **Something had to be done**. ### Preparing the Field: Setting Up the Agent The secret to a successful AI-assisted project isn’t just jumping in and prompting; it’s about preparing the groundwork. I started by defining several [agent skills](https://agentskills.io/home) tailored to prepare my specific text stack. For this project, I created dedicated skills covering: - Backend Development - Frontend Development - GCP Deployment - GCP Observability - Mobile Optimization - Project Tooling - SEO Optimization Then, instead of continuously reminding the agent who I am and what I want, I explicitly created two context files: 1. **`PROFILE.md`**: Outlining my professional identity, experience, and links. # Profile - Médéric Hurier (Fmind) ## Headline Freelancer • AI/ML Architect & Engineer • AI Agents & MLOps • GCP Professional Cloud Architect • PhD in AI & Computer Security ... 2. **`DESIGN.md`**: Defining my brand identity (e.g., “Space & Tech” aesthetic). # Website Design ## Brand Identity A professional, advanced, and modern digital presence for an AI/ML Architect. Blending clean and modern style with a "Space & Tech" aesthetic to reflect deep expertise in Artificial Intelligence and MLOps. ... By establishing these foundational documents, I set the ground rules. The AI agent had immediate access to my personality and brand identity, saving me from having to explain the context over and over again. ### Building the Website: Steering the AI Rather than “vibe coding” everything and blindly compiling the results, I made a strict rule: I would review every single file generated by the agent. As an engineer, I refuse to be responsible for a codebase I haven’t read or understood. Despite the agent’s impressive capabilities, I noted several weaknesses that you must manage when using AI agents to generating code: _“Agents don’t care about your brand. They just want to get the task done.”_ — A harsh truth of AI coding It’s entirely up to you to rigorously maintain your identity and ensure the output matches your needs. > _“Code quickly degenerates. Without supervision, AI can generate piles and piles of code lacking underlying logic.”_ Agents are fantastic at refactoring, but _you_ are the architect who must point them in the right direction. - **Choosing the right tech stack is key.** Call me old school, but I demand mastery over what I produce. I decided to use **Python** as the foundational layer for my programming logic. I am confident in my ability to maintain, debug, and evolve a Python-based architecture over the long term. Once you actively tackle these limitations, the results are breathtaking. While I generally don’t enjoy writing repetitive boilerplate or templates, the AI agent excels at it and never gets tired. You can direct its focus entirely to the frontend, backend, deployment, SEO, or mobile optimization. Watching the pieces fall into place gives you an incredible dopamine rush. ### The Result I was able to finalize the website from scratch in just **5 nights**. This rapid timeline included everything: generating the base components, extensive optimizations, refactoring, and setting up proper deployment pipelines (including analytics and observability). While I could have let the agent run entirely autonomously, it is _my_ website; I wanted to be an active part of the process. ![Result of PageSpeed Insights (LightHouse)](/static/img/articles/how-i-revamped-my-portfolio-website-in-5-nights-using-ai-agents/03.webp) Result of PageSpeed Insights (LightHouse) I’m thrilled to report that the new site achieved a **100% Lighthouse score** across every single category. The site is fast, modern, and beautifully aligned with my brand. I would have never been able to produce this level of polish by myself in such a short timeframe, and I’m incredibly proud of the result. For context on the effort, here is a quick look at the codebase generated and reviewed during those 5 nights: ```text github.com/AlDanial/cloc v 1.98 T=0.10 s (915.2 files/s, 30379.3 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- HTML 57 33 40 1257 Markdown 11 277 0 580 Python 12 138 79 368 TOML 1 8 1 84 YAML 3 0 3 81 CSS 2 9 0 41 Text 4 7 0 33 JSON 1 0 0 31 Dockerfile 1 1 2 27 ------------------------------------------------------------------------------- SUM: 92 473 125 2502 ------------------------------------------------------------------------------- ``` ### Conclusions AI Coding is an absolute game-changer. I highly encourage any developer or freelancer to revamp their own website as soon as possible. It is the perfect, tightly-scoped exercise to discover the true, practical capacity of these tools for yourself. ### Key Takeaways - **Start with clear context files** to ground your agent’s understanding. - **Do not blindly trust generated code;** review everything to maintain architectural control. - **Avoid “shiny object syndrome.”** Relying entirely on AI without architectural vision can lead to generic, unmaintainable results. Find the right balance between automation and engineering rigor. Check out the final result at [fmind.dev](https://fmind.dev). ![About me on Fmind.dev](/static/img/articles/how-i-revamped-my-portfolio-website-in-5-nights-using-ai-agents/04.webp) About me on Fmind.dev If you tried ‘vibe coding’ on your own projects yet, let me know your experience in the comments! ### [Chaigent: An affordable alternative to Gemini Enterprise on Google Cloud](https://www.fmind.dev/articles/chaigent-an-affordable-alternative-to-gemini-enterprise-on-google-cloud/) The era of simple chatbots is over. Companies are now racing to build [**AI Agent platforms**](https://fmind.medium.com/architecting-the-ai-agent-platform-a-definitive-guide-405750a3de44) — systems that don’t just talk, but _act_. Whether it’s a support bot resolving Jira tickets or a data analyst agent querying BigQuery, these new digital teammates need a platform that offers more than just text generation: they require reasoning, security, and enterprise-grade observability. [**Gemini Enterprise**](https://cloud.google.com/gemini/enterprise) provides a great path to achieving this on [**Google Cloud**](https://cloud.google.com/). It offers a comprehensive set of features including agent exposition, governance, integrated knowledge search, and a visual agent builder, connecting with backends like [**Vertex AI Agent Engine**](https://cloud.google.com/products/agent-engine), [**Conversational Agent**](https://docs.cloud.google.com/dialogflow/cx/docs), or [**A2A**](https://a2aprotocol.ai/). However, for some organizations or specific use cases, the cost can be a friction point. The catalog price sits at **~\$7/user/month** for agent users and **~\$35/user/month** for visual agent builders. While this pricing is competitive for knowledge workers who gain significant productivity, it can be prohibitive for large audiences with lower usage frequency, such as field workers or occasional users. **Enter Chaigent:** [**https://github.com/fmind/chaigent**](https://github.com/fmind/chaigent) **.** ![Chaigent is an affordable alternative to Gemini Enterprise (Source: Gemini App)](/static/img/articles/chaigent-an-affordable-alternative-to-gemini-enterprise-on-google-cloud/cover.webp) Chaigent is an affordable alternative to Gemini Enterprise (Source: Gemini App) In this article, I present “Chaigent” ([**Chainlit**](https://chainlit.io/) + Agent), a cost-effective, DIY alternative to Gemini Enterprise on Google Cloud. It leverages the same powerful underlying reasoning engine but replaces the managed frontend with an open-source framework, giving you control over features and costs. ### The Architecture Chaigent enables you to build a private, secure AI agent platform by combining serverless infrastructure with open-source tooling. ![Architecture of Chaigent (Source: Fmind.dev)](/static/img/articles/chaigent-an-affordable-alternative-to-gemini-enterprise-on-google-cloud/02.webp) Architecture of Chaigent (Source: Fmind.dev) The architecture consists of three main layers: 1. **Frontend (** [**Chainlit**](https://chainlit.io/) **on** [**Cloud Run**](https://cloud.google.com/run) **)**: A Python-based UI that handles user sessions, chat history, and authentication. 2. **Backend (** [**Vertex AI Agent Engine**](https://cloud.google.com/products/agent-engine) **)**: The “brain” of the operation, capable of reasoning and tool use. 3. **Persistence & Auth**: [**Cloud SQL**](https://cloud.google.com/sql) for storing chat history and feedback, and [**OAuth**](https://oauth.net/2/) (Google, GitHub, etc.) for secure identity management. This approach allows you to pay for **consumption only** (Cloud Run CPU + Vertex AI tokens), significantly reducing costs for intermittent usage patterns compared to a flat per-seat license. ### The “Do It Yourself” Trade-off Gemini Enterprise provides a managed, “batteries-included” platform with built-in governance and visual tools. Chaigent, in contrast, offers a code-first, developer-centric approach. **What you gain:** - **Cost Efficiency**: No monthly per-seat licensing fees. - **Full Customization**: You own the code. Want to add a custom feedback mechanism or a specific UI widget? You can. - **Platform Independence**: Using [**Chainlit**](https://chainlit.io/) (frontend) and [**Google ADK**](https://google.github.io/adk-docs/) (backend) logic keeps you flexible. **What you lose (The “Subtext”):** - **No Visual Builder**: You define agents in code, not a drag-and-drop UI. - **Manual Governance**: You must implement your own permission logic per agent. - **Ops Overhead**: You are responsible for deploying, securing, and updating the application. - **Enterprise Features**: Advanced features like Model Armor ([**Prompt Security**](https://cloud.google.com/security/products/model-armor)) and integrated Knowledge Search ([**RAG**](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/rag-engine/rag-overview)) require manual implementation. ### Implementation Highlights Chaigent is surprisingly simple to set up. Here is a glimpse of the code. ### 1. Defining the Agent The agent is defined declaratively using the [**Google ADK**](https://github.com/google/adk-python). It’s just a Python object specifying the model and tools. ```python # chaigent/agent.py root_agent = agent( name="chaigent", model="gemini-2.5-flash", description="answer questions with google search.", instruction="you are an expert researcher. you always stick to the facts.", tools=[google_search], ) ``` ### 2. The Bridge (Chainlit Adapter) The `app.py` acts as the bridge. It connects the user’s chat session to the Vertex AI Agent Engine, handling the streaming response seamlessly. ```python # app.py @cl.on_message async def on_message(message: cl.Message): # Initialize response message answer = cl.Message(content="") await answer.send() # Retrieve session session = cl.user_session.get("session") user_id, session_id = session["userId"], session["id"] # Stream the query to Vertex AI response_stream = engine.async_stream_query( user_id=user_id, message=message.content, session_id=session_id ) # Stream back the tokens async for chunk in response_stream: for part in chunk.get("content", {}).get("parts", []): text = part.get("text", "") if text: await answer.stream_token(text) await answer.update() ``` ### User Experience Despite being a “DIY” solution, the user experience is premium. Chainlit provides features that users expect from modern chat apps. **Rich Chat Interface**: Supports markdown, code highlighting, and streaming responses out of the box. ![Chat interface of Chaigent](/static/img/articles/chaigent-an-affordable-alternative-to-gemini-enterprise-on-google-cloud/03.webp) Chat interface of Chaigent **Authentication & Persistence**: Secure login screens and persisted chat history allow users to resume conversations across devices. ![Login screen of Chaigent](/static/img/articles/chaigent-an-affordable-alternative-to-gemini-enterprise-on-google-cloud/04.webp) Login screen of Chaigent **Data Layer**: All interactions are stored in your own SQL database, giving you full ownership of the data for analytics or fine-tuning later. ![Home screen of Chaigent](/static/img/articles/chaigent-an-affordable-alternative-to-gemini-enterprise-on-google-cloud/05.webp) Home screen of Chaigent ### Conclusion Chaigent is an excellent solution when **cost efficiency** is the primary driver, particularly for large audiences with low individual usage. The decision comes down to ROI. At ~\$7/month/user for Gemini Enterprise, you need to save each user at least one hour of work per month to break even. For knowledge workers, this is a no-brainer. But for field workers or casual users, a consumption-based “Pay-as-you-go” model like Chaigent might be the smarter financial move. If you are ready to trade some convenience for control and cost savings, go build your own agents! ![Source: Gemini App](/static/img/articles/chaigent-an-affordable-alternative-to-gemini-enterprise-on-google-cloud/06.webp) Source: Gemini App ### [mAIdAI: Building a Personal Assistant with Google Cloud and Vertex AI](https://www.fmind.dev/articles/maidai-building-a-personal-assistant-with-google-cloud-and-vertex-ai/) As an AI Architect, I spend my days designing AI systems and agents for others. I optimize workflows, fine-tune context windows, and architect serverless solutions to solve complex business problems. But recently, I caught myself in a classic “cobbler’s children” scenario. While helpful bots supported my teams, I navigated my own workflow manually — answering the same repetitive questions, digging for the same documentation links, and context-switching constantly. I realized I needed something different. Not another generic team bot, but a **Personal AI Assistant** — one that knows _my_ specific context, _my_ preferred shortcuts, and _my_ tone. So I built **mAIdAI** (My AI Aid)**:** [https://github.com/fmind/maidai](https://github.com/fmind/maidai) ![mAIdAI Avatar (Source: Gemini App)](/static/img/articles/maidai-building-a-personal-assistant-with-google-cloud-and-vertex-ai/cover.webp) mAIdAI Avatar (Source: Gemini App) In this article, I’ll walk you through how I architected this personal agent using **Google Chat**, **Cloud Run**, and **Vertex AI**. ### The Problem: The High Cost of “Quick” Tasks We often underestimate the micro-friction in our daily work. - “Where is the design doc for Project X?” - “What’s the syntax for that specific gcloud command again?” - “Can you review this snippet?” Standard team bots are great, but they are generic. They lack the _specific_ context of your personal role and responsibilities. I wanted an agent that acts as a “Second Brain” — grounded in my personal knowledge and capable of executing my specific workflows. ### The Solution: mAIdAI Pattern mAIdAI is designed around three core interaction types: 1. **Context-Aware Chat**: A conversational flow grounded in a personal `context.md` file effective “system instructions”. 2. **Quick Commands**: Instant helpers that return static values (like commonly used links or snippets) without invoking the LLM. 3. **Slash Commands**: specialized triggers that wrap user input in a predefined prompt template (e.g., `/fix` to debug code). ![Demo of mAIdAI on Google Chat (Generic Version)](/static/img/articles/maidai-building-a-personal-assistant-with-google-cloud-and-vertex-ai/02.webp) Demo of mAIdAI on Google Chat (Generic Version) ![About mAIdAI](/static/img/articles/maidai-building-a-personal-assistant-with-google-cloud-and-vertex-ai/03.webp) About mAIdAI ### Architecture The system follows a lightweight, serverless event-driven architecture. ![Architecture Diagram of mAIdAI (Source: Fmind.dev)](/static/img/articles/maidai-building-a-personal-assistant-with-google-cloud-and-vertex-ai/04.webp) Architecture Diagram of mAIdAI (Source: Fmind.dev) ### The Flow 1. **Frontend**: usage of the **Google Chat** app interface. No custom UI to build or maintain. 2. **Transport**: Chat events are delivered via HTTP webhooks. 3. **Backend**: A **Cloud Run** service hosting a FastAPI application processes the events. 4. **Intelligence**: The backend connects to **Vertex AI** (Gemini models) for reasoning, grounded by the personal context file. ### Deep Dive: The Code The implementation is surprisingly minimal, thanks to the **Google GenAI SDK** and **FastAPI**. The entire core logic resides in a single `main.py` file. ### 1. The Setup We initialize the GenAI client using standard environment variables. This keeps the code portable and secure. ```python # main.py client = genai.Client( project=os.environ["GOOGLE_CLOUD_PROJECT"], location=os.environ["GOOGLE_CLOUD_LOCATION"], vertexai=True, ) # Loading the Second Brain MODEL_CONTEXT = (ROOT_FOLDER / "context.md").read_text() config = types.GenerateContentConfig( system_instruction=MODEL_CONTEXT, max_output_tokens=5000, ) ``` By reading `context.md` at startup and injecting it as the `system_instruction`, we ensure every interaction is grounded in my specific reality. ### 2. Handling Interaction Types The core router handles the distinction between simple commands and AI interactions. This is crucial for latency and cost — not every interaction needs a round-trip to an LLM. ```python @app.post("/") async def index(request: Request) -> dict: event = await request.json() # ... extraction logic ... if command_id := app_command_metadata.get("appCommandId"): # Handle Slash and Quick Commands if command_type == "QUICK_COMMAND": return respond(command_text) if command_type == "SLASH_COMMAND": # Contextualize the prompt prompt = f"{command_text}. USER INPUT: {user_input}" return respond(await chat(prompt)) # Fallback to standard chat return respond(await chat(user_input)) ``` This pattern allows me to have a `/links` command that returns immediately (0 latency, 0 cost), while a `/rewrite` command leverages Gemin 2.0 Flash for creative work. ### 3. Asynchrony by Default Using `async def` and `client.aio.models.generate_content` ensures the Cloud Run container can handle multiple concurrent requests efficiently, even with a single instance. ### Deployment Strategy Simplicity was the primary constraint. I didn’t want to manage infrastructure for a personal tool. - **Runtime**: Cloud Run (fully managed, scales to zero, low-cost serving). - **Configuration**: Environment variables for model selection (`gemini-3-flash`) and project details. - **Security**: IAM-based authentication ensures only verified chat events reach the service. ### Why Build This Locally? You might ask, “Why not use a standard consumer AI chat?” 1. **Privacy**: Data stays within my Google Cloud project. 2. **Context**: I control the system prompt (`context.md`) explicitly. 3. **Workflow Integration**: It lives where I work — in Google Chat — not a separate browser tab. ### Conclusion We often accept friction because “it’s just how things are.” But as engineers, we have the tools to change that. mAIdAI is a proof of concept that a highly personalized, context-aware agent doesn’t require a massive engineering team. It just requires a few hundred lines of Python and the right cloud primitives. If you find yourself copying the same text or answering the same questions repeatedly, maybe it’s time to build your own assistant. ![Source: Gemini App](/static/img/articles/maidai-building-a-personal-assistant-with-google-cloud-and-vertex-ai/05.webp) Source: Gemini App ### [Building with A2UI: Extending the Expressiveness of AI Agent Interfaces](https://www.fmind.dev/articles/building-with-a2ui-extending-the-expressiveness-of-ai-agent-interfaces/) In 2026, AI agents have become incredibly smart, yet they are often limited to simple chatbot interfaces. We have engines capable of reasoning, planning, and coding, but we force them to communicate results through text and basic markdown. To unlock the full potential of agents, we need a better language for them to express themselves. We need agents that can project rich, dynamic, and interactive user interfaces that adapt to the user’s intent. This is the promise of [**A2UI (Agent-to-User Interface)**](https://a2ui.org/): a protocol that allows agents to “speak” UI natively. In my [previous article](https://fmind.medium.com/finding-the-holy-grail-of-ai-agent-uis-from-ai-orchestrated-development-to-a2ui), I explored the landscape of AI UI solutions and explained why A2UI stands out. Now, I wanted to put it to the test. I built [Featest](https://github.com/fmind/featest), a feature request application designed to be “AI-First.” Here is the story of how it was built, the strengths of the protocols I used, and the architectural patterns that emerged. ![Source: Gemini App](/static/img/articles/building-with-a2ui-extending-the-expressiveness-of-ai-agent-interfaces/cover.webp) Source: Gemini App ### The Project: Featest The origin of this project was a simple request from my Product Manager: _“We need a way for users to vote on features.”_ I could have built a standard CRUD app. But I saw this as an opportunity. A feature request board is dynamic. Users have vague intents: _“I want something like dark mode but for audio.”_ They want to merge duplicates. They want to see trends, and admins want to automatically tag requests. This was the perfect scenario to consider an **AI-First experience**. I didn’t want a static form; I wanted an agent that users could talk to, which enhances the interactions, and uses the right UI tools when needed. **Github Repository**: [Featest](https://github.com/fmind/featest) ![Suggest new features with AI (Source: Fmind.dev)](/static/img/articles/building-with-a2ui-extending-the-expressiveness-of-ai-agent-interfaces/02.webp) Suggest new features with AI (Source: Fmind.dev) ### The Power Couple: A2UI and A2A Before diving into the code, it’s critical to understand the two pillars of this architecture. ### 1. A2UI: Agent-to-User Interface (The Content) A2UI is a **declarative protocol**. Instead of an agent writing code (which is risky and error-prone), it streams a structured JSON description of a UI. Here is what it looks like on the wire. The agent sends `SurfaceUpdate` events to render components like a card with a button: ```json { "surfaceUpdate": { "surfaceId": "main-surface", "components": [ { "id": "welcome-card", "component": { "Card": { "child": "welcome-text" } } }, { "id": "welcome-text", "component": { "Text": { "text": { "literalString": "Welcome to Featest" }, "usageHint": "h1" } } } ] } } ``` ### 2. A2A: Agent-to-Agent Communication (The Transport) [A2A](https://github.com/a2aproject/A2A) is the **transport protocol**. It standardizes how agents talk to each other and to clients over HTTP. It handles the handshake, the task lifecycle, and the message passing. In Featest, the client wraps the user’s intent in an A2A message: ```http POST /api/agents/feature_request_agent/tasks Content-Type: application/json { "task_id": "12345", "input": { "text": "I want to vote for dark mode" } } ``` Together, they create a universal language. A2A carries the envelope, and A2UI ensures the letter inside contains rich, interactive content, not just text. ### Architecture I designed the system to be modular, using the [Google Agent Development Kit (ADK)](https://google.github.io/adk-docs) and the A2A protocol effectively. ![Architecture Diagram of the Featest App (Source: Fmind.dev)](/static/img/articles/building-with-a2ui-extending-the-expressiveness-of-ai-agent-interfaces/03.webp) Architecture Diagram of the Featest App (Source: Fmind.dev) The flow is bidirectional and relies on robust open-source packages: 1. **User** interacts with the **Lit Client**, which uses the official [`@a2ui/lit`](https://github.com/google/A2UI/tree/main/renderers/lit) renderer. It acts as a state machine, processing `SurfaceUpdate` events to patch the DOM efficiently. 2. **Client** sends intent via **A2A** to the **Backend**. The `A2UIClient` wraps the user’s input (text or events) in a standard JSON-RPC envelope. 3. **Backend Agent** processes logic and streams back **A2UI** JSON instructions. 4. **Client** renders the UI components dynamically. The power of this system comes from the **Component Schema**. Featest supports a rich set of native components defined in `schemas.py`, ensuring the agent has high-level building blocks rather than raw HTML: - **Layout**: `Row`, `Column`, `List`, `Card`, `Tabs`, `Divider`, Modal. - **Input**: `Button`, `CheckBox`, `TextField`, `DateTimeInput`, `MultipleChoice`, Slider. - **Media**: `Text`, `Image`, `Icon`, `Video`, AudioPlayer. ### The AVC Pattern: Agent-View-Controller The most significant discovery I made while building Featest was an architectural one. When you start building complex agent apps, you quickly realize that a single agent doing everything (reasoning, database access, UI formatting) is a mess. It’s hard to test and hard to control. I adopted what I call the **Agent-View-Controller (AVC) Pattern** — an evolution of [Model-View-Controller (MVC)](https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller) for the agent era. ### 1. The Controller Agent (The Brain) This agent handles the business logic. It doesn’t care about pixels. It inputs a user request, decides which tool to use (e.g., `vote_feature`, `add_comment`), and outputs structured data. ```python # agent/agent.py controller_agent = agents.Agent( name="controller_agent", model=configs.AGENT_MODEL, description="Executes application logic.", instruction=prompts.CONTROLLER_INSTRUCTION, tools=[ tools.list_features, tools.add_feature, tools.upvote_feature, tools.add_comment, tools.get_feature, tools.update_feature, tools.delete_feature, ], ) ``` ### 2. The View Agent (The Renderer) This agent is the designer. It takes the data from the Controller and translates it into A2UI JSON. It cares about layout, typography, and hierarchy. ```python view_agent = agents.Agent( name="view_agent", model=configs.AGENT_MODEL, description="Formats data into A2UI schema.", instruction=prompts.VIEW_INSTRUCTION, output_schema=schemas.A2UI, ) ``` ### 3. The Sequential Pipeline I chained them together using ADK’s SequentialAgent. This simple composition gave me immense flexibility. I could swap the View Agent to change the entire look and feel of the app without touching a single line of business logic. ```python root_agent = agents.SequentialAgent( name="feature_request_agent", description="Handles feature requests from users.", sub_agents=[controller_agent, view_agent], ) ``` ### Strengths of the Protocol Working with A2UI revealed several advantages over traditional chatbot approaches. ![Leave comment, with a form rendered dynamically with A2UI (Source: Fmind.dev)](/static/img/articles/building-with-a2ui-extending-the-expressiveness-of-ai-agent-interfaces/04.webp) Leave comment, with a form rendered dynamically with A2UI (Source: Fmind.dev) ### 1. UI Language We are used to agents speaking Markdown. Markdown is fantastic for content: paragraphs, lists, and code blocks. But it fails when you need _interaction_. If an agent needs to ask for a complex set of preferences, Markdown forces it to ask one question at a time or parse a messy natural language blob. A2UI allows the agent to project a form with validation, sliders for precise values, and date pickers. It elevates the agent from a “writer” to an “interface designer,” matching the right interaction model to the user’s intent. ### 2. Security by Design This is the enterprise killer feature. Because A2UI is **data**, not code, there is no `eval()` happening on the client. The agent selects from a catalog of safe, pre-built components. You can’t inject malicious scripts via A2UI, making it safe for production environments where “generated code” is a security nightmare. ### 3. Progressive Rendering A2UI is designed to be streamed. As the LLM generates the JSON tokens, the UI builds itself on the screen. - First, the container appears. - Then, the title. - Then, the list items one by one. This makes the application feel incredibly responsive, masking some latency with visible progress. ### 4. Interoperability Because the UI is just JSON, the exact same agent response can be rendered natively on the web (via [Lit](https://lit.dev/)), on mobile (via [Flutter](https://docs.flutter.dev/ai/genui)), or on iOS (via Swift). You build the agent intelligence once, and it projects natively everywhere. ### 5. Client Style Control With A2UI, the agent is responsible for the _structure_ (intent), but the client is completely in control of the _style_. The agent says “I need a primary button.” It doesn’t say “I need a blue button with 4px border radius.” This means your application maintains perfect brand consistency. The same agent response can look like a sleek consumer app on Android or a dense dashboard on the web, simply by changing the client-side theme. ### Limitations ### 1. Maturity & Boilerplate A2UI is a new protocol. Building a custom app from scratch currently requires significant boilerplate code. For now, the best approach is to use packages like Flutter’s [GenUI SDK](https://docs.flutter.dev/ai/genui) or wait for higher-level integrations from ADK or Gemini Enterprise. ### 2. Latency vs. Smartness Another major challenge is **latency**. Generating UI tokens takes time and money. While streaming and using “Fast Planners” (like I did in Featest) mitigate this, a pure agentic experience will never beat a hand-optimized native app for core, repetitive tasks. The “smartness” of the dynamic UI must outweigh the latency cost — if it doesn’t, just use a static button. ### 3. Complementary Nature I also found that not every use case benefits from dynamic UI. For the core voting interaction in Featest, a static, predictable UI was simpler and faster. Where A2UI shines is for _augmenting_ the experience, helping users rationalize features, tag duplicates, or explore trends through conversation. In this project, it’s a powerful **complement** to a baseline UI, not necessarily a replacement ### Conclusion A2UI is a fantastic protocol, but it’s not a silver bullet. In my case, I initially thought a pure “AI-First” app would be the ideal experience. However, I learned that for basic, repetitive tasks, it’s simply too slow compared to a traditional interface. The latency of generating UI on the fly doesn’t always pay off. The ideal approach for this project is a **hybrid model**: mixing static, highly-optimized UIs for core workflows with dynamic, agentic components for complex, intent-driven tasks. It is up to the programmer to find the best trade-off for each specific use case. However, for **chatbot-focused applications**, this solution could be highly valuable. It enables the creation of much richer UIs exactly when needed, allowing the experience to go beyond simple text and adapters. One thing is sure: there will be more and more agentic features, and A2UI will be a great bridge between agent power and user needs. ![Building with A2UI: Extending the Expressiveness of AI Agent Interfaces](/static/img/articles/building-with-a2ui-extending-the-expressiveness-of-ai-agent-interfaces/05.webp) ### [MLOps Coding Skills: Bridging the Gap Between Specs and Agents](https://www.fmind.dev/articles/mlops-coding-skills-bridging-the-gap-between-specs-and-agents/) We are entering the golden age of AI Coding. Every day, I see colleagues, both technical and non-technical, marveling at how agents are rewriting the rules of software construction. The promise is intoxicating: describe what you want, and let the machine handle the rest. However, when I see my colleagues try to apply these agents to strict engineering standards, they hit a wall. On one side, you have rigorous specification tools like [spec-kit](https://github.com/github/spec-kit) or [conductor](https://github.com/gemini-cli-extensions/conductor). They are deterministic and thorough, but setting them up feels like writing a legal contract. On the other side, you have generic tools like the **Model Context Protocol (MCP)**. They act as incredible “hands” for the AI — reading databases, calling APIs — but they lack the _brain_ for your specific context. They don’t know that your team enforces [uv](https://github.com/astral-sh/uv) over [poetry](https://python-poetry.org/). They don’t know you prefer [just](https://github.com/casey/just) files for automation. They don’t know your specific flavor of “clean code.” Then I discovered [**Agent Skills**](https://agentskills.io/home), and everything clicked. I was immediately hooked. They offer the specific trade-off I had been looking for: **lightweight enough to be flexible, yet opinionated enough to be useful.** ![Source: Gemini App](/static/img/articles/mlops-coding-skills-bridging-the-gap-between-specs-and-agents/cover.webp) Source: Gemini App In this article, I want to share how I used Agent Skills to turn the theoretical “MLOps Coding Course” into a practical, actionable library: the [**MLOps Coding Skills**](https://github.com/MLOps-Courses/mlops-coding-skills) project. ### The Challenge: Making References Actionable For the past few months, I’ve been deep in the trenches writing the [**MLOps Coding Course**](https://github.com/MLOps-Courses/mlops-coding-course). It is a comprehensive curriculum teaching production-grade MLOps, from robust project initialization to advanced observability. ![MLOps Coding Skills: Bridging the Gap Between Specs and Agents](/static/img/articles/mlops-coding-skills-bridging-the-gap-between-specs-and-agents/02.webp) But as I wrote the documentation, I felt a friction point. Learning the standards is one thing; remembering to apply them in the heat of coding is another. I didn’t just want another wiki page. I wanted to make these best practices **actionable** for valid AI Agents. I wanted to move from “reading the docs” to “installing the capability.” ### The Logic: How to “Skillify” Knowledge The beauty of an Agent Skill lies in its simplicity. It is essentially a markdown file (`SKILL.md`) that functions as a context injection module. It gives the agent “muscle memory” for a specific topic. My methodology for building the **MLOps Coding Skills** repo was straightforward: 1. **Isolate a Chapter**: Take a specific section of the course (e.g., _Automation_ or _Observability_). 2. **Extract Patterns**: Use an LLM to distill the generic engineering standards from the educational content. 3. **Standardize**: Format it into a `SKILL.md` that an agent can ingest. ### A Concrete Example: Automating Ops Let’s look at the [**mlops-automation**](https://github.com/MLOps-Courses/mlops-coding-skills/tree/main/mlops-automation) skill. In our course, we have strong opinions: we use `just` for command running and [docker](https://www.docker.com/) for containerization, with very specific layer caching strategies. Here is what the skill looks like “on the wire”: ```markdown # MLOps Automation ## Goal To elevate the codebase to production standards by adding Task Automation (just), Containerization ([docker](https://www.docker.com/)), CI/CD ([github-actions](https://github.com/features/actions)), and Experiment Tracking ([mlflow](https://mlflow.org/)). ## Instructions ### 1. Task Automation Replace manual commands with a `justfile`. 1. **Tool**: `just` (modern alternative to Make). 2. **Organization**: Split tasks into `tasks/*.just` modules. 3. **Core Tasks**: - `check`: Run all linters and tests. - `package`: Build wheels. ### 2. Containerization 1. **Tool**: `docker`. 2. **Base Image**: Use `ghcr.io/astral-sh/uv:python3.1X-bookworm-slim` for minimal size. ``` When I load this skill, my agent stops guessing. It doesn’t offer me a Makefile. It doesn’t suggest a bloated Ubuntu image. It acts like a senior engineer who has been on the team for years. ### The “Senior Engineer” Injection This is the killer value proposition. Most frustrations with AI coding come from a **lack of context**. We blame the model for being “dumb,” but usually, we just haven’t told it the rules of the house. By using Agent Skills, you are effectively **injecting a Senior Engineer into your chat context**. You are giving the agent a “cheat sheet” that forces it to align with your organization’s reality. I now use these skills for every new project I touch. I don’t spend an hour setting up boilerplate. I load or create a skill, and within minutes, I had a structure that matched my most rigorous standards. ### The Friction Points Of course, no solution is perfect. There are still rough edges in this workflow: - **Local-First Friction**: Currently, skills often sit in a local `.agent/skills` folder. It works, but copying them around feels archaic. - **The Context Stack**: We are seeing a fragmentation of context. We have MCP servers for tools, `AGENTS.md` for persona, and Skills for tasks. Managing this “Context Stack” is becoming a new engineering discipline. - **Integration gaps**: I love how the **Gemini CLI** handles this via extensions, but I’m eager to see this standardized across VS Code Copilot, Cursor, and other IDEs. ### Conclusion Despite the minor friction, Agent Skills are excellent “Low Hanging Fruit” for any engineering team. The productivity gain is massive. For a few minutes of setup — writing a markdown file — you save hours of correcting boilerplate code and enforcing standards down the line. It bridges the gap between the **rigidity of a spec** and the **chaos of a raw LLM**. If you are tired of fighting your AI to follow your style, stop arguing with it. Give it a Skill. _Check out the full_ [_**MLOps Coding Skills repository**_](https://github.com/MLOps-Courses/mlops-coding-skills) _to see the library in action._ ![Source: Gemini App](/static/img/articles/mlops-coding-skills-bridging-the-gap-between-specs-and-agents/03.webp) Source: Gemini App ### [Finding the Holy Grail of AI Agent UIs: From AI-Orchestrated Development to A2UI](https://www.fmind.dev/articles/finding-the-holy-grail-of-ai-agent-uis-from-ai-orchestrated-development-to-a2ui/) In my [previous article](https://fmind.medium.com/the-real-ai-agent-bottleneck-is-the-damn-ui-90e90ee369e0), I argued that the real bottleneck for AI agents is the User Interface (UI). We are stapling rocket engines to bicycles by forcing advanced agents to communicate through basic markdown chatbots. Since then, I’ve been on a journey to find the solution. I didn’t want just a theoretical answer; I wanted to build it. I explored everything from “AI-Orchestrated Development” to Python wrappers, up to new AI protocols, searching for a scalable way to give Agents a native, rich, and dynamic interface. I dedicated time to building a concrete implementation to verify my hypotheses. Here is what I found, what failed, and why I believe [**A2UI**](https://a2ui.org/) is the protocol we’ve been waiting for to solve this problem. ![Source: Gemini App](/static/img/articles/finding-the-holy-grail-of-ai-agent-uis-from-ai-orchestrated-development-to-a2ui/cover.webp) Source: Gemini App ### The Exploration: A Graveyard of “Almost” Solutions My goal was simple: **Build a custom frontend for an agent application without spending weeks on boilerplate.** I tried multiple approaches, and most of them hit a wall. ### 1. The “Heavy” Approach: Angular & Flutter My first instinct was to build a real app. I tried both [Angular](https://angular.dev/) and [Flutter](https://flutter.dev/). These are standards for enterprise application development, offering robust ecosystems and pixel-perfect control. **The Result:** It works, but at what cost? In 2026, setting up a full frontend project is still painful. You have to configure build tools, set up linters, manage complex state stores (Redux, Bloc), and synchronize data models with your backend. This overhead is acceptable for a static, long-term product like a banking dashboard, but for a dynamic Agent? **It’s overkill**. Agents need to be able to transmit their UI and adapt on the fly. Hardcoding a heavy client defeats the purpose of an autonomous agent. If every new agent capability requires a sprint of frontend changes, the agent isn’t truly autonomous. It’s just a backend API with a very expensive chat interface. ### 2. “AI-Orchestrated Development” (AI-Generated UIs) I tried what I call “**AI-Orchestrated Development**”: a more structured approach where the AI is front and center in generating application code, popularized in early 2026 by tools like [GitHub Spec Kit](https://githubnext.com/projects/copilot-workspace), [Gemini Conductor](https://developers.googleblog.com/en/conductor-introducing-context-driven-development-for-gemini-cli/), or [Antigravity](https://blog.google/innovation-and-ai/technology/developers-tools/gemini-3-developers/). This is distinct from “vibe coding” (using AI intuitively without understanding the output). AI-Orchestrated Development aims for a systematic process where AI handles implementation under developer guidance. **The Verdict:** While promising long-term, it still generates _lots_ of code. Code that you have to maintain, test, and debug. And I’m not confident in either maintaining AI-generated codebases or letting AI be the sole responsible party for production systems. We already spend more time on application maintenance than building. AI-Orchestrated Development risks accelerating this accumulation. We need to reduce the amount of specific code generated, not increase it. ### 3. HTMX: The Backend-Driven UI I went back to my roots (PHP/AJAX) and tried [HTMX](https://htmx.org/). It’s a productive methodology that keeps logic in one place by streaming HTML fragments from the server. **The Problem:** HTMX couples the agent too tightly to a specific visual implementation. If you want to render the same agent response on a mobile app, a web dashboard, and a desktop client, you can’t reuse the HTML stream — you’re locked into one presentation layer. More fundamentally, HTML is too low-level for an agent to reason about. An agent shouldn’t be worrying about CSS classes, DOM nesting, or accessibility attributes. It should focus on _intent_ and _logic_, not pixels. Sending declarative data is more efficient, more universal, and can be consumed by different types of clients. ### 4. Python Wrappers (Streamlit, Gradio, Chainlit) These are great for prototypes. Tools like [Streamlit](https://streamlit.io/), [Gradio](https://gradio.app/), and [Chainlit](https://chainlit.io/) offer a small code surface and instant deployment. **The Flaw:** The “Glue Code” Hell. You inevitably hit a wall where the library doesn’t support the specific interaction or component you need. Maybe you need a custom drag-and-drop interface or a specific data visualization. You lose control over style, and you end up writing hacky workarounds (custom HTML injection, iframe bridges) to connect the agent’s state to the UI components. They are also not truly dynamic — they are rigid templates filled with data, not fluid interfaces generated by the agent’s needs. You are still building a form; you are just doing it in Python instead of React. ### 5. Chat Extensions (Slack/Teams/Workspace) Building into existing workflows seems smart. Why build a new UI when you can just deploy a bot to Slack or Google Chat? **The Limit:** It doesn’t scale. You end up building a specific adapter for Slack, another for Teams, another for Google Chat. Each platform has its own proprietary UI kit (Block Kit, Adaptive Cards) with different limitations. You want to build your agent _once_ and have it project its UI anywhere, not rewrite the presentation layer for every host app. This fragmentation increases the maintenance burden and prevents you from creating a consistent user experience across platforms. ### The Epiphany: Separation of Concerns I realized something fundamental during this process: **Everything is disposable.** We shouldn’t be precious about the UI code. We should focus on the **declarative** side. Just as humans use HTML not because we love drawing pixels, but because we want to say “Here is a link” or “Here is an image,” agents need a high-level language to describe _what_ needs to be shown, not _how_ to draw it. The Agent should be responsible for the **Data** and the **Logic**. The Client should be responsible for the **Style** and the **Rendering**. This separation allows the agent to be “brain-heavy” and “UI-light,” deferring the complex rendering logic to the client, which is what clients are best at. ### The Solution: A2UI (Agent-to-User Interface) Enter [**A2UI**](https://github.com/google/A2UI). I built a demo app using this protocol, and I was genuinely impressed by its elegance. A2UI is a **JSONL-based declarative protocol** that creates a standard contract between the AI and the user interface. ### How it works Instead of streaming markdown tokens like a traditional LLM, the agent streams structured JSON objects representing UI components. ![Source: https://a2ui.org/](/static/img/articles/finding-the-holy-grail-of-ai-agent-uis-from-ai-orchestrated-development-to-a2ui/02.webp) Source: [https://a2ui.org/](https://a2ui.org/) The client can use the [Lit renderer](https://github.com/google/A2UI/tree/main/renderers/lit), [Angular renderer](https://github.com/google/A2UI/tree/main/renderers/angular), or [Flutter renderer](https://docs.flutter.dev/ai/genui) to render native components progressively. ### Why it wins - **Production-Ready at Google:** A2UI isn’t vaporware — it’s already integrated into Google products like [Opal](https://labs.google/), [Gemini Enterprise](https://cloud.google.com/gemini/enterprise), and the [Flutter GenUI SDK](https://docs.flutter.dev/ai/genui). - **Transport Agnostic:** It works over HTTP (via the [A2A protocol](https://github.com/google/A2A)), WebSockets, or carrier pigeons. The protocol doesn’t care how the JSON gets there. - **Progressive Rendering:** The UI appears as the agent “thinks” it. Components stream in one by one, making the interface feel alive and responsive, much like text streaming but for rich UI elements. - **Framework Agnostic:** The client implementation (React, Angular, Lit) decides how a “Card” looks. The agent just says “I need a Card”. This means you can have a “Material Design” client and an “iOS Cupertino” client rendering the exact same agent response natively. - **Secure:** No arbitrary JavaScript execution. It’s just declarative data, mitigating injection risks. This is critical for enterprise adoption where security reviews block “dynamic code generation.” - **LLM-Friendly:** Flat, streaming JSON structure designed for easy generation. LLMs can build UIs incrementally without perfect JSON in one shot. > **Note:** A2UI is currently at [v0.8](https://a2ui.org/specification/v0.8-a2ui/) and still in active development. The protocol has some rough edges, and for production use. The best approach is to wait for native integration in tools like [Gemini Enterprise](https://cloud.google.com/gemini/enterprise) or the [Agent Development Kit (ADK)](https://google.github.io/adk-docs). ### A2UI vs AG-UI: Two Philosophies I also looked at [**AG-UI**](https://ag-ui.com/), another emerging standard in this space. - **AG-UI** aims to blend the frontend and backend deeply, creating “AI-First” apps from the ground up with a focus on real-time event loops. It’s powerful but requires you to rethink your entire application architecture regarding state synchronization and event handling. - **A2UI** focuses on _extending_ the capabilities of chat-based interaction to be richer. It’s a bridge that lets agents “speak UI” using standard components. It feels more like an evolution of the chat interface into a command center rather than a complete replacement of the application stack. I believe **A2UI** is the scalable path forward for most agent implementations. It respects the separation of concerns and integrates seamlessly with existing systems via protocols like A2A (Agent-to-Agent). ### Conclusion: The 2026 Shift We are moving towards a schism in frontend technology, and it’s happening faster than we think: 1. **Static Apps (the stock):** Dashboards, retail sites, and specialized tools. These will still be built with efficient frameworks for speed, precise control, and specific user journeys where the path is known. They represent the bulk of existing applications. 2. **Dynamic Agent Interfaces (the flow):** Powered by new protocols like **A2UI**. These will replace the “Chatbot” with something far more powerful — interactive, component-based, and generated on the fly. This is where the new growth is happening. These interfaces will emerge when the user’s intent is ambiguous or highly variable, like in [Agentic Commerce](https://cloud.google.com/transform/a-new-era-agentic-commerce-retail-ai). I am convinced that 2026 is the year we stop building UIs _for_ agents and start letting agents _project_ their UIs to us. We shouldn’t spend too much time on UI. Let it be personalized by the agent so we can focus on what truly matters: integration and instruction. _In the next article, I will share the source code and a full demo of the application I built using A2UI. Stay tuned!_ ![Source: Gemini App](/static/img/articles/finding-the-holy-grail-of-ai-agent-uis-from-ai-orchestrated-development-to-a2ui/03.webp) Source: Gemini App ### [Architecting the AI Agent Platform: A Definitive Guide](https://www.fmind.dev/articles/architecting-the-ai-agent-platform-a-definitive-guide/) The velocity of Generative AI has been nothing short of relentless. In the span of just 24 months, the industry has shifted paradigms three times. We started with the raw capability of **LLMs** (the “prompt engineering” era). We quickly moved to **RAG** (Retrieval-Augmented Generation) to ground those models in enterprise data. Now, we are at the era of **AI Agents**. We are no longer asking models to simply talk or retrieve; we are asking them to _do_. We are building systems capable of reasoning, planning, and executing actions to change the state of the world. Building a single agent in a notebook is easy. Building a system that serves, secures, and monitors thousands of autonomous agents across an enterprise is an entirely different engineering challenge. To deliver robust solutions with tangible ROI, you cannot rely on scattered Proofs of Concept. You need a factory. You need an **AI Agent Platform**. In this guide, I will deconstruct the architecture of a production-grade AI Agent Platform, breaking it down into its system context, containers, and component layers. ![Source: Gemini (Nano Banana Pro)](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/cover.webp) Source: Gemini (Nano Banana Pro) ### System Context: The PaaS Approach At its core, the AI Agent Platform is a **Platform-as-a-Service (PaaS)** designed to build, serve, and expose AI agents. Unlike **AI Agent SaaS** solutions — which lock you into a closed ecosystem and a predefined set of integrations — an AI Agent Platform is designed for **extensibility** and **control**. SaaS solutions are excellent for quick wins, but they often lack the ability to support custom logic or complex enterprise workflows. Crucially, an internal AI Agent Platform allows you to enforce **SRE (Site Reliability Engineering)** practices. If an agent fails, your Ops team can intervene. If an agent attempts an unauthorized action, your Security team has the audit trails to investigate and harden the perimeter. The platform serves two distinct types of builders: 1. **The Programmer (Code-Based):** Engineers requiring power and flexibility. 2. **The Integrator (No/Low-Code):** Business analysts requiring speed and ease of configuration. It must also be accessible to **External Systems** (Machine-to-Machine) via standard APIs like REST or gRPC. This allows other systems to offload cognitive tasks — like “analyze this log file” or “classify this ticket” — to your agent fleet programmatically. To function, the AI Agent Platform relies on five high-level systems: - **Identity & Access:** The gatekeeper for users, agents, and data. - **Foundation Models:** The cognitive “brain” (reasoning, planning, and instruction following). - **Enterprise Apps & APIs:** The “hands” of the agent (e.g., Jira, Salesforce, SAP, SQL, …). - **Information Systems:** The context providers (Operational DBs, Data Lakes, Knowledge Bases). - **Cloud Infrastructure:** The bedrock providing compute and reliability. ![Source: Fmind.dev](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/02.webp) Source: Fmind.dev ### The Container Architecture To manage complexity, we divide the AI Agent Platform into **7 Logical Containers**. This separation of concerns is vital for security auditing and independent scaling. 1. **Interaction:** The frontend where users meet agents. 2. **Development:** The workbench for building and deploying. 3. **Core:** The runtime engine that executes logic. 4. **Foundation:** The infrastructure abstraction for models and compute. 5. **Information:** The data layer managing context. 6. **Observability:** The monitoring and evaluation stack. 7. **Trust:** The security and governance control plane. ![Source: Fmind.dev](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/03.webp) Source: Fmind.dev Let’s hack through these layers one by one. ### 1. Interaction The Interaction layer is the portal. It is where the carbon lifeforms (us) communicate with the silicon. ![Source: Fmind.dev](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/04.webp) Source: Fmind.dev There are three primary ways to expose your agents: - **Standard Chatbot:** The familiar conversational interface. It is fast to ship and often requires zero frontend skills. However, it is a generic instrument; chat is not always the best interface for complex user experience. - **Custom User Interface:** Bespoke web or mobile apps. This is where the power lies. As I’ve argued before, [the UI is often the real bottleneck for agents](https://fmind.medium.com/the-real-ai-agent-bottleneck-is-the-damn-ui-90e90ee369e0). Custom UIs allow for rich interactions, but they come with a “frontend tax” — they are time-consuming to build. - **External Channels:** Extending the platform to meet users where they are — SMS, Email, Voice, or Slack. This is critical for field workers or remote teams who don’t sit in front of a dashboard all day. In the future, I expect **Generative UI** to take over by 2026. This is where the agent generates dynamic interface elements on the fly based on user intent (see [Google Research](https://research.google/blog/generative-ui-a-rich-custom-visual-interactive-user-experience-for-any-prompt/)). In the meantime, we must trade between options. ### 2. Development This is the factory floor. My experience shows a **50/50 split** between developers (code-based) and integrators (no/low code), so your platform must support both paths to avoid limiting speed or flexibility. ![Source: Fmind.dev](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/05.webp) Source: Fmind.dev #### Code-Based (The Developer Path) This path is for engineers using frameworks like [LangGraph](https://langchain-ai.github.io/langgraph/), [CrewAI](https://www.crewai.com/), or [Google ADK](https://google.github.io/adk-docs/). - **The Stack:** Code is versioned in SCM (Git), tested via CI/CD, and deployed as software artifacts. - **The Cost:** Surprisingly low. With “Model-as-a-Service,” developers can build robust agents on a laptop or Cloud Workstation for pennies per day. You don’t need a local H100 cluster. #### No-Code (The Integrator Path) This path is for business analysts using Visual Builders and iPaaS ([Integration Platform as a Service](https://cloud.google.com/application-integration)) tools. - **The Stack:** Visual designers, drag-and-drop workflows, and pre-built connectors for building AI Agents. - **The Trade-off:** Speed vs. Flexibility. It is the fastest way to prototype and connect to enterprise apps, but visual design can be less robust and more limiting than pure code. ### 3. Core The Core is the heartbeat. It houses the **Execution Engine**, the runtime responsible for the agent’s cognitive loop. ![Source: Fmind.dev](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/06.webp) Source: Fmind.dev #### The Execution Engine To be truly autonomous, the runtime needs specific capabilities that ease development: - **Session Management:** Persisting state across conversational turns. - **Memory Bank:** Handling short-term context and long-term recall. - **Code Sandbox:** A secure environment (like a micro VM) where the agent can write and execute code safely to solve math or data problems. #### Gateways & Orchestration You don’t always need a heavy Airflow setup with DAGs, but you do need: - **Task Schedulers / Event Buses:** To trigger agents asynchronously (e.g., “New Ticket Created” -\> “Wake up Triage Agent”). - **API Management:** Exposing agents via standard Gateways like [Apigee](https://cloud.google.com/apigee?authuser=1) or [Gravitee](https://www.gravitee.io/). **Standardization is Key:** Practitioners are heavily encouraged to adopt standards like [**MCP (Model Context Protocol)**](https://modelcontextprotocol.io/) and [**A2A (Agent-to-Agent)**](https://a2aprotocol.ai/) interfaces. Your platform cannot be an island; it must act as a network where your agents can call tools or even _other_ agents to complete complex tasks. ### 4. Foundation The Foundation layer is the bedrock of the AI Agent Platforms, providing both Foundation Models and Infrastructure solutions to the agents. ![Source: Fmind.dev](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/07.webp) Source: Fmind.dev #### Model Strategy - **Serving:** You will likely mix **Model-as-a-Service** (Vertex AI, Bedrock, …) for ease of use and scalability, and **Custom Model Hosting** for specific, fine-tuned, or private models that require more operational effort. - **Model Routing:** Don’t default to the most expensive model. Use a router to dispatch simple queries to cheaper/faster models and complex reasoning to “smart” models (e.g., Gemini 1.5 Pro, Claude 3.5 Sonnet, GPT-4). - **Context Caching:** A massive cost saver. Cache system instructions and heavy documents so you aren’t paying to re-tokenize your company handbook on every request. #### Infrastructure Standard cloud primitives apply here. Compute, Blob Storage, and **Artifact Management** (for abstracting the agent storage of input/output files) are essential. Treat your Agent Infrastructure as Code (IaC) to ensure reproducibility across environments (AWS, GCP, Azure, or on-premise). ### 5. Information An agent without data is a hallucination machine. The Information layer feeds the context required for decision-making. ![Source: Fmind.dev](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/08.webp) Source: Fmind.dev 1. **Knowledge (Unstructured):** Documentation and guidelines stored in shared drives or online websites. These are typically indexed by a **RAG Engine** or **Search Engine** to explain _how_ the company works. 2. **Operational (Structured):** Transactional data (SQL DBs) required to _do_ work (e.g., update a CRM record). Builders should favor APIs over direct DB access here to ensure business logic integrity. 3. **Data Lake (Analytical):** Historical data for insights and decision making. Requires a Semantic Layer and Data Catalog so the agent understands what “Revenue” actually means before running a query. **The Sync Problem:** Syncing these systems is painful. Each sync risks data duplication and inconsistency. We are moving toward a convergence of OLAP and OLTP with systems like [Google AlloyDB](https://cloud.google.com/products/alloydb?authuser=1) or [Databricks Lakebase](https://www.databricks.com/product/lakebase) to eliminate the copy/desync nightmare. ### 6. Observability If there is one thing humans must remain in control of, it is **supervising the agents.** ![Source: Fmind.dev](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/09.webp) Source: Fmind.dev - **Supervision:** The entry point. You need to collect logs, traces, and audit trails. Alerts should notify operators immediately when an agent loops or fails. - **Evaluation:** The hardest part. You need pipelines where Foundation Models (or humans) review agent traces to score them on metrics such as **Factuality, Relevance, and Accuracy**. - **Billing:** FinOps for AI. Track token usage per department. This is especially important for new architectures with less familiar cost sinks. - **Analytics:** Tracking adoption. Is the agent actually solving tickets, or are people ignoring it? This is key to reporting ROI to stakeholders. ### 7. Trust Finally, the Trust layer. Agents are high-leverage tools; without governance, they are a liability that could create havoc. ![Source: Fmind.dev](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/10.webp) Source: Fmind.dev - **IAM (Identity & Access Management):** RBAC is mandatory. Furthermore, **Tool Authentication** (OIDC/OAuth) ensures the agent only takes actions the _user_ is authorized to take (acting on behalf of the user). - **Security:** Guardrails. You need to filter content, prevent **Prompt Injection** (jailbreaks), and detect malicious content _before_ the LLM sees it. **Secret Management** is also critical to protect secrets like API keys. - **Governance:** The Registry. You need a central catalog of authorized agents, models, and tools. You don’t want to be hunting through the org chart to find out who built the “Payroll Bot” or who is responsible for a rogue agent. This can extend to a **Marketplace** for buying assets from other vendors. ### Conclusion Building an AI Agent Platform is not just about stringing together a few API calls. It is about building a scalable, secure, and observable ecosystem where code and reasoning merge to drive real business impacts. I’m really excited to build these powerhouse of automation and intelligence! Whether you are a developer writing complex orchestration logic or an integrator dragging and dropping workflows, the platform provides the stability you need to move from “demo” to “production”. The challenge will be immense, but if you have the right vision, roadmap and architecture, solutions will appear layer by layer to start addressing your use cases. Start with the core, secure the trust layer, and never underestimate the importance of observability. The agents are coming — make sure you have the platform to manage them and give them both power and control. ![Source: Gemini (Nano Banana Pro)](/static/img/articles/architecting-the-ai-agent-platform-a-definitive-guide/11.webp) Source: Gemini (Nano Banana Pro) ### [Powering Up your Agent in Production with ADK, OAuth and Gemini Enterprise](https://www.fmind.dev/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise/) The promise of AI agents is immense productivity gains. But putting them into production can be a tale of two extremes: surprisingly fast or painfully slow. The difference often hinges on the infrastructure and tooling you choose. If you attempt to build everything from scratch — creating a custom UI, managing complex authentication flows, and setting up observability — development slows down significantly. You spend more time on infrastructure than on the agent logic itself. I recently argued this point in [“The Real AI Agent Bottleneck is the Damn UI”](https://fmind.medium.com/the-real-ai-agent-bottleneck-is-the-damn-ui-90e90ee369e0). However, with the right tools, deploying an agent can be remarkably quick. To demonstrate how to achieve this fast route, we need a practical example. A while ago, I shared a [notebook built over lunch to translate Google Slides](https://fmind.medium.com/slides-to-translate-when-it-says-no-build-a-0-04-solution-on-your-lunch-break-3afa8bd9f6bb). It was effective but stuck in a notebook, inaccessible to my teammates. ![A hacker tinkering a robot (Source: Gemini App)](/static/img/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise/cover.webp) A hacker tinkering a robot (Source: Gemini App) This article details the journey of taking that “Slides Translator” and pushing it into production as a secure, scalable agent, leveraging the right stack to bypass the usual bottlenecks. We will focus on using the [Agent Development Kit (ADK)](https://google.github.io/adk-docs/), [OAuth](https://oauth.net/2/), and [Gemini Enterprise](https://cloud.google.com/gemini-enterprise?hl=en). The full code for this project is available on GitHub: [https://github.com/fmind/slides-translator-agent](https://github.com/fmind/slides-translator-agent) ### The Agentic Architecture To move from a notebook to a production agent, we need an architecture that handles security, execution, and user access robustly. ![Architecture of the Slide Translator Agent from local development to production deployment (Source: fmind.dev)](/static/img/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise/02.webp) Architecture of the Slide Translator Agent from local development to production deployment (Source: fmind.dev) The workflow is structured as follows: 1. **Local Development:** The agent logic is developed using the [Agent Development Kit (ADK)](https://google.github.io/adk-docs/) and tested locally via the [ADK Web UI](https://github.com/google/adk-web). 2. **Deployment:** The agent is deployed to the [**Vertex AI Agent Engine**](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview) on Google Cloud Platform. 3. **Production Access:** Users interact with the agent through the [**Gemini Enterprise Web UI**](https://cloud.google.com/gemini-enterprise?hl=en). 4. **Execution and Security:** The Agent Engine manages the execution. It uses [**OAuth**](https://oauth.net/2/) for secure authorization, interacts with [**Google APIs**](https://developers.google.com/apis-explorer) **(Drive and Slides)** on the user’s behalf, and utilizes [**Gemini Models**](https://ai.google.dev/gemini-api/docs/models) for the translation. ### ADK and the Power of OAuth The [Agent Development Kit (ADK)](https://github.com/google/adk-python) provides a great set of features to handle everything you need for building agents. In this specific use case, I focused on its ability to handle **OAuth**, to let the user grant access to their slides and drive. ![Overview of Google ADK and Vertex AI Agent Engine (Source: https://cloud.google.com/agent-builder/agent-engine/overview )](/static/img/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise/03.webp) Overview of Google ADK and Vertex AI Agent Engine (Source: [https://cloud.google.com/agent-builder/agent-engine/overview](https://cloud.google.com/agent-builder/agent-engine/overview)) In the notebook prototype, authentication relied on local credentials. This is not suitable for a production agent that needs to access the _user’s_ specific files. The agent must act on behalf of the user, requiring their explicit permission. ### Why OAuth? [OAuth 2.0](https://oauth.net/2/) provides excellent security guarantees and granularity. It allows users to grant specific permissions (scopes) without sharing their passwords with the agent. In this case, we need access to the [Google Drive API](https://developers.google.com/drive/api?authuser=1) (to copy the presentation) and the [Google Slides API](https://developers.google.com/slides/api?authuser=1) (to read and write slide content). While OAuth is not an easy concept to grasp for newcomers, it’s a key component to provide more security in enterprise applications. ![OAuth flow for a tool with Google ADK (Source: https://google.github.io/adk-docs/tools/authentication/ )](/static/img/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise/04.webp) OAuth flow for a tool with Google ADK (Source: [https://google.github.io/adk-docs/tools/authentication/](https://google.github.io/adk-docs/tools/authentication/)) ### Configuration To make this work, an OAuth Client ID must be configured in the Google Cloud Console: [https://console.cloud.google.com/auth/clients](https://console.cloud.google.com/auth/clients) ![Configuration of the OAuth Credentials on Google Cloud: https://console.cloud.google.com/auth/clients](/static/img/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise/05.webp) Configuration of the OAuth Credentials on Google Cloud: [https://console.cloud.google.com/auth/clients](https://console.cloud.google.com/auth/clients) Crucially, we need to define the “Authorized redirect URIs”. The `localhost` URI is used during local development with the ADK Web UI, and the `https://vertexaisearch.cloud.google.com/oauth-redirect` URI is used by the Vertex AI Agent Engine in production to securely handle the callback after the user grants consent. ### Implementation in ADK ADK simplifies the OAuth flow significantly. We define the authentication configuration and use decorators to protect the tools that require user credentials. Here is a snippet demonstrating the core authentication mechanism in the agent code: ```python """Authentication for the tools.""" # %% IMPORTS from fastapi.openapi.models import OAuth2, OAuthFlowAuthorizationCode, OAuthFlows from google.adk.auth.auth_credential import AuthCredential, AuthCredentialTypes, OAuth2Auth from google.adk.auth.auth_tool import AuthConfig from slides_translator_agent import configs # %% CONFIGS AUTHORIZATION_URL = "https://accounts.google.com/o/oauth2/auth" TOKEN_URL = "https://oauth2.googleapis.com/token" SCOPES = { "https://www.googleapis.com/auth/drive": "Google Drive API", "https://www.googleapis.com/auth/presentations": "Google Slides API", } # %% AUTHENTICATIONS AUTH_SCHEME = OAuth2( flows=OAuthFlows( authorizationCode=OAuthFlowAuthorizationCode( authorizationUrl=AUTHORIZATION_URL, tokenUrl=TOKEN_URL, scopes=SCOPES, ) ) ) AUTH_CREDENTIAL = AuthCredential( auth_type=AuthCredentialTypes.OAUTH2, oauth2=OAuth2Auth( client_id=configs.AUTHENTICATION_CLIENT_ID, client_secret=configs.AUTHENTICATION_CLIENT_SECRET, ), ) AUTH_CONFIG = AuthConfig( auth_scheme=AUTH_SCHEME, raw_auth_credential=AUTH_CREDENTIAL, ) ``` When the `translate_presentation` tool is invoked, the `negotiate_creds` function ensures that a valid token exists. If not, ADK automatically pauses the agent execution and initiates the OAuth flow with the user. ```python """Tools for the agents.""" import json from google.auth.transport.requests import Request from google.oauth2.credentials import Credentials from slides_translator_agent import auths def negotiate_creds(tool_context: ToolContext) -> Credentials | dict: """Handle the OAuth 2.0 flow to get valid credentials.""" logger.info("Negotiating credentials using oauth 2.0") # Check for cached credentials in the tool state if cached_token := tool_context.state.get(configs.TOKEN_CACHE_KEY): logger.debug("Found cached token in tool context state") if isinstance(cached_token, dict): logger.debug("Cached token is a dictionary, treating as AuthCredential.") try: creds = Credentials.from_authorized_user_info( cached_token, list(auths.SCOPES.keys()) ) if creds.valid: logger.debug("Cached credentials are valid, returning credentials") return creds if creds.expired and creds.refresh_token: logger.debug("Cached credentials expired, attempting refresh") creds.refresh(Request()) tool_context.state[configs.TOKEN_CACHE_KEY] = json.loads(creds.to_json()) logger.debug("Credentials refreshed and cached successfully") return creds except Exception as error: logger.error(f"Error loading/refreshing cached credentials: {error}") tool_context.state[configs.TOKEN_CACHE_KEY] = None # reset cache elif isinstance(cached_token, str): logger.debug("Found raw access token in tool context state.") # This creates a temporary credential object from the token # Note: This credential will not be refreshed if it expires return Credentials(token=cached_token) else: raise ValueError( f"Invalid cached token type. Expected dict or str, got {type(cached_token)}" ) # If no valid cached credentials, check for auth response logger.debug("No valid cached token. Checking for auth response") if exchanged_creds := tool_context.get_auth_response(auths.AUTH_CONFIG): logger.debug("Received auth response, creating credentials") auth_scheme = auths.AUTH_CONFIG.auth_scheme auth_credential = auths.AUTH_CONFIG.raw_auth_credential creds = Credentials( token=exchanged_creds.oauth2.access_token, refresh_token=exchanged_creds.oauth2.refresh_token, token_uri=auth_scheme.flows.authorizationCode.tokenUrl, client_id=auth_credential.oauth2.client_id, client_secret=auth_credential.oauth2.client_secret, scopes=list(auth_scheme.flows.authorizationCode.scopes.keys()), ) tool_context.state[configs.TOKEN_CACHE_KEY] = json.loads(creds.to_json()) logger.debug("New credentials created and cached successfully") return creds # If no auth response, initiate auth request logger.debug("No credentials available. Requesting user authentication") tool_context.request_credential(auths.AUTH_CONFIG) logger.info("Awaiting user authentication") return {"pending": True, "message": "Awaiting user authentication"} ``` This ensures the user explicitly consents to the agent accessing their files before any action is taken. ![OAuth is supported natively on Google ADK: when needed, ADK will prompt the user to grant more access to the agent tools](/static/img/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise/06.webp) OAuth is supported natively on Google ADK: when needed, ADK will prompt the user to grant more access to the agent tools ### Deploying with Gemini Enterprise Once the agent is developed and tested, the next step is deploying it to production. ### Configuring Production Authentication Before deploying the agent code, we need to register the OAuth configuration with the production environment. I used the following script to set this up: ```bash ./as.py create-auth \ --auth-id slides-translator-auth \ --client-id ... \ --client-secret ... \ --auth-uri "https://accounts.google.com/o/oauth2/auth?include_granted_scopes=true&response_type=code&access_type=offline&prompt=consent" \ --token-uri "https://oauth2.googleapis.com/token" \ --scope "https://www.googleapis.com/auth/drive" \ --scope "https://www.googleapis.com/auth/presentations" ``` This command links the `slides-translator-auth` ID (referenced in the Python code above as `configs.TOKEN_CACHE_KEY`) with the actual Client ID, Secret, and the required scopes. Note: As the Gemini Enterprise exposition API is still in private preview, I can’t share more details nor the deployment script yet. ### Seamless Exposition [Gemini Enterprise](https://cloud.google.com/gemini/enterprise?authuser=1) gives you a quick way to expose your agent securely and conveniently. This directly addresses the “[UI bottleneck](https://fmind.medium.com/the-real-ai-agent-bottleneck-is-the-damn-ui-90e90ee369e0)” mentioned earlier. This approach has significant advantages over deploying a separate UI (like Streamlit): - **Zero-Effort UI:** No need to design, host, or secure a separate frontend application. - **Observability:** Thanks to the underlying Agent Engine, it traces and logs the agent information automatically, providing essential observability for production monitoring and debugging. - **Core Services:** It provides more core services and integrates seamlessly within the Google Cloud security perimeter. The end result is a clean, integrated experience. Users can interact with the “Slides Translator Agent” directly within the Gemini interface. ![Slides Translator Agent deployed on Gemini Enterprise](/static/img/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise/07.webp) Slides Translator Agent deployed on Gemini Enterprise ### Conclusion This journey from a simple notebook to a production-ready agent was a great experience to see what this stack provides out of the box. The combination of ADK for development, OAuth for security, and Gemini Enterprise for deployment streamlines the entire lifecycle of an enterprise agent, allowing us to deploy quickly without compromising on security or usability. I’m eager to explore more ways to build agents. While this is a new paradigm that requires upskilling our teammates and adapting our development practices, we already see the potential from the use cases we see. The ability to rapidly deploy secure, specialized tools that act on behalf of users is a significant step forward. ![Human and Agent merging to accomplish their tasks (Source: Gemini App)](/static/img/articles/powering-up-your-agent-in-production-with-adk-oauth-and-gemini-enterprise/08.webp) Human and Agent merging to accomplish their tasks (Source: Gemini App) ### [The Real AI Agent Bottleneck is the Damn UI](https://www.fmind.dev/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/) We’re living in the golden age of AI agent development. The backend infrastructure is finally catching up to the hype. If you’ve followed my previous work on [deploying agents using ADK and Google Cloud](https://fmind.medium.com/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud-b49e7eda3b41), you know that the heavy lifting — the orchestration, the tool integration, the deployment pipelines — is becoming standardized. The major players are all in. Whether you’re using Google Cloud’s [Vertex AI Agent Engine](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview) powered by the [ADK](https://github.com/google/adk-python), [AWS AgentCore](https://aws.amazon.com/bedrock/agentcore/) with [Strands](https://docs.aws.amazon.com/prescriptive-guidance/latest/agentic-ai-frameworks/strands-agents.html), or [Databricks’ AgentBricks](https://docs.databricks.com/aws/en/generative-ai/agent-bricks/), building the _brain_ of the agent is easier than ever. But here’s the dirty secret the hype cycle isn’t talking about: **The User Interface (UI) is the real bottleneck for industrializing AI agents.** You can have the most sophisticated, multi-step reasoning agent on the planet, but if your users can’t interact with it intuitively, securely, and effectively, it’s a challenge for deploying AI agents at scale. The last mile — exposing the agent to maximize impact — is where projects go to die. In this article, we are going to explore this problem, and find the best trade-off to remove the bottlenecks and adopt AI agents full-throttle in your company! ![Source: Google AI Studio](/static/img/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/cover.webp) Source: Google AI Studio ### The Bottleneck: Why UI is the Hardest Part Building an agent requires a specific skill set: LLM understanding, backend engineering, and prompt whispering. Building a _good_ UI requires a completely different one: frontend development, UX design, and product sense. The engineers hacking together these agents are rarely UI experts. And frankly, they shouldn’t have to be. #### The Tiresome Process of Building UIs Having to spin up a new React app every time you deploy an agent is soul-crushing. It’s tedious, time-consuming, and completely unscalable. We need generalized interfaces that adapt to specific workflows, not custom code for every use case. This includes generalizing how we evaluate agent performance and collect user feedback — critical components that are often considered as an afterthought. #### The Identity Crisis User and Agent Identity is paramount. If an agent needs to access a database or pull a file from Google Drive, it must do so _on the user’s behalf,_ from the UI . We can’t have agents authenticating with god-mode service accounts, nor can we force users to re-authenticate with every single tool during an interaction. The UI must seamlessly handle delegated authority. #### Security and Governance: The Enterprise Non-Negotiables This isn’t a weekend hackathon project. In an enterprise setting, security is everything. You cannot allow loose access controls. The nightmare scenario? An agent with access to your entire data lake _and_ the ability to send emails externally. The risk of data leakage is massive. Governance requires auditing every operation, ensuring data usage is controlled, and verifying that tool access is restricted. The UI is the gateway for all of this. This technical juggle requires both admins and users to be familiar with the environment, but their interfaces must be optimized for their roles. ### Symptoms of a Broken System When the UI layer fails, the organization feels the pain. 1. **The Rise of Shadow IT:** When official tools are too hard to use or deploy, users find workarounds. We see a proliferation of quick-and-dirty solutions, like rogue [n8n](https://n8n.io/) instances deployed under someone’s desk, creating massive security vulnerabilities. 2. **Agent Silos:** Agents should be collaborative. They need to interact with each other, leveraging protocols like the emerging [A2A (Agent-to-Agent) standard](https://a2a-protocol.org/). But when agents live in isolated, collaboration is impossible. They become siloed tools rather than a cohesive intelligence layer. 3. **The “90% Done” Fallacy:** This is the classic trap. You hack together a [Streamlit](https://streamlit.io/) web app, deploy it, and declare the project 90% complete. Wrong. The real project — adoption, integration, security hardening, and UI refinement — is just beginning. ### Exploring the Approaches: The Good, The Bad, and The Ugly How are we currently trying to solve this UI challenge? Let’s break down the dominant paradigms. #### 1. The Pure Chatbot (The Terminal Approach) The idea here is that the chatbot is the _only_ interface. We see this clearly with the recent [OpenAI Apps](https://openai.com/index/introducing-apps-in-chatgpt/) or [Gemini Extensions](https://support.google.com/gemini/answer/14959807?hl=en&co=GENIE.Platform%3DAndroid). ![OpenAI Apps. Source: https://www.axios.com/2025/10/06/openai-chatgpt-app-devday](/static/img/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/02.webp) OpenAI Apps. Source: [https://www.axios.com/2025/10/06/openai-chatgpt-app-devday](https://www.axios.com/2025/10/06/openai-chatgpt-app-devday) - **Pros:** Simple, universal interface for everything. Low development overhead. - **Cons:** Incredibly restrictive. Markdown syntax is _not_ a UI framework. You can’t easily implement sliders, interactive maps, complex data visualizations, or rich editing tools. - **The Verdict:** This is like a developer terminal, but using natural language instead of Linux commands. It’s powerful for certain tasks but hits a wall quickly when complexity increases. #### 2. The Co-Pilot (The Sidecar Approach) The chatbot controls another, existing interface. The prime example is [Gemini for Workspace](https://workspace.google.com/solutions/ai/), where the chatbot sits as a widget on the right side of Docs, Sheets, or Gmail. ![Gemini for Workspace with Chat Sidecar on Google Sheets](/static/img/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/03.webp) Gemini for Workspace with Chat Sidecar on Google Sheets - **Pros:** Meets the user where they already are. Keeps the familiar interface of the host application. - **Cons:** Limited to the capabilities of the host app. Cross-application workflows (e.g., “Analyze this spreadsheet and draft a presentation based on the findings”) are difficult or impossible. - **The Verdict:** A great enhancement for existing tools, but not a solution for complex, multi-tool agentic workflows. #### 3. Generic Static UI (The Visual Workflow) This involves using a predefined visual interface, like [n8n](https://n8n.io/) or specialized agent builders. ![Example of Visual Workflow on n8n. Source: https://n8n.io/](/static/img/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/04.webp) Example of Visual Workflow on n8n. Source: [https://n8n.io/](https://n8n.io/) - **Pros:** Generic yet adaptable to specific workflows. Fast to develop and easy to interpret. - **Cons:** Visual workflows are often legacy techniques poorly suited for Generative AI. They are too rigid. How do you easily put a human in the loop? How do you give the agent more autonomy when the path is predefined? - **The Verdict:** Good for traditional automation, but stifles the potential of true AI agents. #### 4. Specific Static UI (The Artisanal Approach) Building a custom, bespoke UI for every agent, often using frameworks like [Genkit](https://genkit.dev/). ![Source: https://developers.googleblog.com/en/how-firebase-genkit-helped-add-ai-to-our-compass-app/](/static/img/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/05.webp) Source: [https://developers.googleblog.com/en/how-firebase-genkit-helped-add-ai-to-our-compass-app/](https://developers.googleblog.com/en/how-firebase-genkit-helped-add-ai-to-our-compass-app/) - **Pros:** The absolute best adaptation to the specific use case. Maximum control over the user experience. - **Cons:** Slow to develop, expensive, and completely unscalable, especially for quick agents. - **The Verdict:** Necessary for flagship products, but impossible for the rapid deployment of specialized agents. #### 5. Dynamic UI (The Shape-Shifter) The UI is fluid and generated on the spot by the AI itself. We see this with Claude generating artifacts, or experimental concepts like Google’s [Opal](https://opal.withgoogle.com/landing/) and the [AG-UI](https://github.com/ag-ui-protocol/ag-ui) protocol. ![Dynamic UI with Opal (green block). Source: https://blog.google/technology/google-labs/opal-expansion/](/static/img/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/06.webp) Dynamic UI with Opal (green block). Source: [https://blog.google/technology/google-labs/opal-expansion/](https://blog.google/technology/google-labs/opal-expansion/) - **Pros:** No need to code UI anymore. Maximum adaptation to the desired workflow. Incredibly fast development cycle. - **Cons:** Unpredictable and inconsistent. Not efficient — it feels like “vibe coding.” It’s feasible for small apps, but is it robust enough for large-scale enterprise applications? - **The Verdict:** The holy grail, but the technology isn’t mature enough for mission-critical applications. ![Summary of the UI Approaches for AI Agents](/static/img/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/07.webp) Summary of the UI Approaches for AI Agents ### The Agent Hub Imperative Regardless of the UI paradigm we choose, one thing is clear: we need an **Agent Hub**. Organizations need a centralized location to discover available agents, manage their access, orchestrate their interactions (both human-to-agent and agent-to-agent), and provide governance oversight. ### The Current Landscape: Evaluating the Options Where do today’s solutions fit in? - [**n8n**](https://n8n.io/) **/** [**OpenAI Agent Builder**](https://openai.com/index/introducing-agentkit/) **(Visual Workflow):** Familiar with organizations, which aids adoption. However, they are fundamentally restrictive and don’t allow for the autonomy and human-in-the-loop interaction that GenAI agents can leverage. - [**OpenAI Apps**](https://openai.com/index/introducing-apps-in-chatgpt/) **/** [**Gemini Extensions**](https://support.google.com/gemini/answer/13695044?hl=en&co=GENIE.Platform%3DAndroid) **(Chat-First):** The easy fix, but they lack expressiveness. If we limit agents to simple chat interfaces, we risk repeating the failures of Alexa — useful for timers, but not for complex work. - [**Opal**](https://blog.google/technology/google-labs/opal-expansion/) **/** [**AG-UI**](https://github.com/ag-ui-protocol/ag-ui) **(Dynamic UI):** Great for small, isolated apps and user autonomy, but not scalable for large, complex systems. They are hard to edit, maintain, and ensure consistency. - [**AWS QuickSuite**](https://aws.amazon.com/fr/blogs/aws/reimagine-the-way-you-work-with-ai-agents-in-amazon-quick-suite/) **(Hybrid):** A pragmatic, conservative middle ground. [QuickSuite](https://docs.aws.amazon.com/quicksuite/latest/userguide/what-is.html) offers a toolset of GenAI variants with UIs tailored for specific tasks like data analysis, deep research, or conversation. A solid, choice, especially if you are using AWS services. ![AWS Quic Suite with several experiences: Chat Agents, Flows, and Research](/static/img/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/08.webp) AWS Quic Suite with several experiences: Chat Agents, Flows, and Research - [**Gemini Enterprise**](https://cloud.google.com/gemini-enterprise?hl=en) **(Agent Hub Focus):** [Gemini Enterprise](https://cloud.google.com/gemini-enterprise) shows potential as a central hub, but it needs to deliver richer expressiveness beyond the standard chat interface to truly unlock agent potential. One solution is to control other UI (e.g., Google Sheets) from the chat app. ![Agent Gallery and Usage from Gemini Enterprise. Source: https://cloud.google.com/gemini-enterprise?hl=en](/static/img/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/09.webp) Agent Gallery and Usage from Gemini Enterprise. Source: [https://cloud.google.com/gemini-enterprise?hl=en](https://cloud.google.com/gemini-enterprise?hl=en) ### My Bet on the Future The UI bottleneck won’t be solved overnight. Here’s where I see things heading. #### Short/Medium Term: The “Hacker Terminal” Wins For the immediate future, the **Chatbot UI** will dominate. It’s the easiest to develop and gets you 80% of the way there. It’s the “hacker terminal” approach — using natural language to orchestrate complex systems — but easier to use. In addition, visual workflows will be used for deterministic applications (i.e., [agentic workflows](https://www.youtube.com/watch?v=Qd6anWv0mv0)) as a complementary solution. The key to making this work won’t be richer UIs, but better _backend_ collaboration. Agents need to be able to seamlessly call other agents ([A2A](https://a2a-protocol.org/)) behind the scenes, using the chat interface purely as the command and control layer. #### Long Term: Ambient Computing and Voice In the long term, the best UI is no UI. We will move towards **voice and ambient computing**. We will keep our existing human applications (our spreadsheets, our design tools, our CRMs), and agents will pilot them intelligently on our behalf. This is both easier to develop (no new UIs needed) and easier to adopt (users keep their existing workflows). However, this requires incredibly robust models and rigorous testing. We only adopt transformative interfaces when they are near-perfect. Think about voice translation — it only became truly useful when it crossed the 95% accuracy threshold. Ambient computing will require the same level of reliability. Until then, we need to stop treating the UI as an afterthought. It’s a critical component for unlocking the value of AI agents in the enterprise. It’s time we started engineering it with the same rigor we apply to the agents themselves. ![Photo by Anton Filatov on Unsplash](/static/img/articles/the-real-ai-agent-bottleneck-is-the-damn-ui/10.webp) Photo by [Anton Filatov](https://unsplash.com/@antony123antony?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [Da2a: The Future of Data Platforms is Agentic, Distributed, and Collaborative](https://www.fmind.dev/articles/da2a-the-future-of-data-platforms-is-agentic-distributed-and-collaborative/) For decades, the story of data platforms has been one of centralization and heavy engineering. We built massive data warehouses and data lakes, but accessing their insights required deep technical expertise. Business users couldn’t simply ask questions; they had to navigate a complex process involving specialized data engineers to build painstaking ETL pipelines, optimized queries, and specific dashboards. This highly technical approach created a rigid, monolithic source of truth that, while powerful, was slow to adapt and created significant bottlenecks. It left decision-makers waiting days or even weeks for answers, completely dependent on an over-burdened engineering team. ![Illustration of the complexity of data platforms (Source: Gemini App)](/static/img/articles/da2a-the-future-of-data-platforms-is-agentic-distributed-and-collaborative/cover.webp) Illustration of the complexity of data platforms (Source: Gemini App) What if we flipped the model on its head? Instead of a single, all-knowing monolith, imagine a collaborative ecosystem where domain experts describe their data in natural language, providing context that empowers a network of intelligent, autonomous agents. Each agent becomes an expert in its domain — sales, marketing, logistics, finance — managing its own data by combining human-provided descriptions with its own skills to answer questions. This is the future of data platforms: a system that is **agentic, distributed, and truly collaborative**. I created a new open-source project, [**Da2a**](https://github.com/fmind/da2a), to explore this paradigm. It’s a prototype that demonstrates how a multi-agent system can tackle complex data analysis by working together. ### The Old Way vs. The New Paradigm **The traditional data platform is** **engineering-focused**. The primary challenge is moving, storing, and modeling data. Answering a simple business question like, _“What’s the ROI on our latest social media campaign?”_ could involve: 1. Filing a ticket with the data engineering team. 2. Waiting for them to build a new pipeline to join marketing spend data with sales data. 3. Having an analyst write a complex SQL query across multiple massive tables. 4. Finally, getting a report back, hoping the initial question hasn’t become irrelevant. **The agentic approach is** **insight-focused**. Instead of a centralized database, you have specialized agents. For instance, the **Marketing Agent** knows everything about campaign spending and lead acquisition. On the other hand, the **E-commerce Agent** is an expert on orders, products, and revenue. To answer that same question, you simply ask a root “Orchestrator Agent.” The orchestrator understands the goal, formulates a plan, and collaborates with the specialist agents to get the answer. The focus shifts from the _how_ (engineering) to the _what_ (the business question). ### Meet Da2a: An Agentic Platform in Action Da2a implements this vision with a root orchestrator and two specialized agents: one for an **e-commerce** dataset and another for a **marketing** dataset, both based on real-world data from the [Olist store in Brazil](https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce). **GitHub Repository:** [https://github.com/fmind/da2a](https://github.com/fmind/da2a "null") You can ask the e-commerce agent, “How many orders were placed in São Paulo?” or the marketing agent, “What were our top lead sources last year?”. Better yet, you can ask the root orchestrator a question that requires both, like, _“What is the total sales revenue from sellers who were acquired via ‘Display’ advertising?”_ ![Screenshot of the Da2a User Interface](/static/img/articles/da2a-the-future-of-data-platforms-is-agentic-distributed-and-collaborative/02.webp) Screenshot of the Da2a User Interface The root agent intelligently delegates the work: first asking the marketing agent to identify the sellers from the ‘Display’ channel, then passing that list to the e-commerce agent to calculate their total sales. ### The Architecture: Collaboration via the A2A Protocol The magic that makes this collaboration possible is the [**Agent-to-Agent (A2A) protocol**](https://a2a-protocol.org/latest/). A2A provides a standardized way for agents to communicate their capabilities and call upon each other’s skills over a network. ![Architecture of the Da2a Application, with Marketing and E-Commerce agents collaborating with the Root Agent](/static/img/articles/da2a-the-future-of-data-platforms-is-agentic-distributed-and-collaborative/03.webp) Architecture of the Da2a Application, with Marketing and E-Commerce agents collaborating with the Root Agent The architecture consists of: 1. [**A Root Agent**](https://github.com/fmind/da2a/tree/main/da2a) **:** The orchestrator that receives user requests, plans the execution, and delegates tasks. 2. **Domain Agents:** The [`ecommerce_agent`](https://github.com/fmind/da2a/tree/main/ecommerce) and [`marketing_agent`](https://github.com/fmind/da2a/tree/main/marketing), each running as an independent service with its own database. 3. [**Agent Cards**](https://a2a-protocol.org/latest/tutorials/python/3-agent-skills-and-card/) **:** Each domain agent exposes a JSON “agent card” that acts like a digital business card, describing its name, capabilities, and how to communicate with it. The root agent is configured to know about these remote agents. Here is a simplified look at the code from the da2a [`agent.py`](https://github.com/fmind/da2a/blob/main/da2a/agent.py) file, which sets up the connection to the remote agents using their "agent cards." ```python import google.adk.agents.remote_a2a_agent as a2a import google.adk.tools.agent_tool as at # The URL points to the 'agent card' of the remote agent AGENT_CARD_ECOMMERCE = "[https://da2a-ecommerce.fmind.dev/a2a/ecommerce/.well-known/agent-card.json](https://da2a-ecommerce.fmind.dev/a2a/ecommerce/.well-known/agent-card.json)" AGENT_CARD_MARKETING = "[https://da2a-marketing.fmind.dev/a2a/marketing/.well-known/agent-card.json](https://da2a-marketing.fmind.dev/a2a/marketing/.well-known/agent-card.json)" # Create local proxy objects for the remote agents ecommerce_agent = a2a.RemoteA2aAgent( name="ecommerce_agent", agent_card=AGENT_CARD_ECOMMERCE, description="Answers questions about e-commerce data..." ) marketing_agent = a2a.RemoteA2aAgent( name="marketing_agent", agent_card=AGENT_CARD_MARKETING, description="Answers questions about marketing data..." ) # The root agent uses these agents as 'tools' to solve problems root_agent = LlmAgent( ... tools=[at.AgentTool(ecommerce_agent), at.AgentTool(marketing_agent)], ... ) ``` Each domain agent is served via the [Agent Development Kit’s](https://google.github.io/adk-docs/) (ADK) web server, which automatically exposes the A2A endpoints and the agent card. ```bash # Command to serve an agent and enable A2A communication adk web --a2a ``` This simple, powerful mechanism allows us to build a distributed system where components can be developed, deployed, and scaled independently. ### The Benefits of Thinking Agentically This approach unlocks several powerful advantages: - **Human-Like Task Handling:** Agents can tackle complex, multi-step tasks that require synthesizing information from different domains, much like a human analyst would. - **Scalability and Extensibility:** Adding a new data domain is as simple as building and deploying a new agent. No need to re-architect the entire platform. The system grows organically. - **Focus on High-Level Value:** It abstracts the underlying engineering complexity. Data consumers and developers can focus on defining business logic and asking high-level questions, not on writing SQL or managing data pipelines. - **Autonomous and Collaborative:** Each agent is a valuable tool on its own, but their true power is unlocked when they collaborate through an orchestrator to solve problems that no single agent could handle alone. ### The Road Ahead: Limitations and Future Work Da2a is a prototype, and building an industrial-grade agentic data platform requires solving some interesting challenges: 1. **Efficient Data Transfer:** A2A is excellent for orchestrating tasks and passing small payloads of text or JSON. It is not designed for transferring gigabytes of data between agents. For that, we’d need to integrate mechanisms that point agents to shared data storage. 2. **Dynamic Agent Discovery:** Currently, the root agent’s knowledge of other agents is hardcoded. A production system would need a discovery service or a registry where agents can dynamically register themselves and their skills. 3. **Memory and Learning:** The agents in this prototype are stateless. The next frontier is to give them memory, allowing them to learn from past interactions, recall previous results, and improve their planning and execution over time. ### Conclusion: A New Frontier for Data The agentic paradigm represents a fundamental shift in how we think about data architecture. We are moving from rigid, centralized systems to dynamic, decentralized ecosystems of intelligent specialists. This approach promises to create data platforms that are more flexible, more powerful, and more aligned with the way businesses actually work. There is still much to build, but the potential is immense. The future of data isn’t just about bigger databases or faster queries; it’s about collaboration, intelligence, and a network of agents working together to turn data into insight. ![The future is with Agentic Data Platforms (Source: Gemini App)](/static/img/articles/da2a-the-future-of-data-platforms-is-agentic-distributed-and-collaborative/04.webp) The future is with Agentic Data Platforms (Source: Gemini App) ### [Ackgent: Rapid Agent Development on GCP with ADK and Agent Config](https://www.fmind.dev/articles/ackgent-rapid-agent-development-on-gcp-with-adk-and-agent-config/) The AI agent landscape is exploding, but development speed is hitting a wall. We need a faster, more accessible way to build and iterate. In my current role, I spend my days optimizing the experience of building and deploying AI agents. I’ve witnessed firsthand the incredible use cases my customers’ developers — agents that streamline complex workflows, automate intricate decision-making, and unlock new data insights. The potential is massive, but the reality of development is often friction-filled. **Despite the advancements in foundational models, the process of taking an agent from concept to production remains too slow and overly complex.** Developers get bogged down in boilerplate code, infrastructure wrangling, and the mechanics of tool integration, rather than focusing on the actual logic and value the agent provides. We drastically need to improve the speed at which we can iterate, while simultaneously making the whole process more accessible to a broader range of builders. Enter [**Ackgent**](https://github.com/fmind/ackgent), a demonstration of how [Google ADK](https://google.github.io/adk-docs/) and [Agent Config](https://google.github.io/adk-docs/agents/config/) can be use to quickly build and deploy AI agents with a declarative approach. ![Source: Gemini App](/static/img/articles/ackgent-rapid-agent-development-on-gcp-with-adk-and-agent-config/cover.webp) Source: Gemini App ### The Shift from Imperative to Declarative The traditional approach to building agents is largely _imperative_. You write Python (or similar) code detailing exactly _how_ the agent should execute tasks, manage state, call tools, and handle errors. This offers maximum control but comes at the cost of speed and simplicity. What if we could shift to a _declarative_ approach? What if we could define _what_ the agent should do, and let a robust framework handle the execution? This is the promise of [Agent Config](https://google.github.io/adk-docs/agents/config/), a new feature of [Agent Developer Kit (ADK)](https://google.github.io/adk-docs/) introduced in [the release v.1.12.0](https://github.com/google/adk-python/releases/tag/v1.12.0). [Agent Config](https://google.github.io/adk-docs/agents/config/) allows developers to define the entire behavior of an agent — its goals, instructions, tools, and integrations — using a structured configuration file written in YAML. This addresses both the need for speed and the need for accessibility. The central insight here is that **config helps you focus on the use case, not the code.** By abstracting the underlying mechanics, developers, prompt engineers, and product managers can rapidly prototype and test different agent behaviors simply by editing a YAML file. ### Flexibility Without the Boilerplate **Crucially, adopting a declarative approach doesn’t mean sacrificing power or flexibility**. Agent Config is designed to be extensible. While the core orchestration is handled by the framework, it provides clear pathways for integrating essential components: - **External Tools:** You can easily connect your agents to real-world APIs, databases, and services. - **Callbacks:** Hooks are available to inject custom Python logic at specific points in the agent lifecycle (e.g., for pre-processing input, validating output, logging, or monitoring). - **MCP (Multi-agent Communication Protocol) Servers:** Agent Config supports integration with MCP servers, enabling sophisticated communication, governance, and orchestration in complex multi-agent systems.   ```yaml # yaml-language-server: $schema=https://raw.githubusercontent.com/google/adk-python/refs/heads/main/src/google/adk/agents/config_schemas/AgentConfig.json agent_class: LlmAgent model: gemini-2.5-flash name: prime_agent description: Handles checking if numbers are prime. instruction: | You are responsible for checking whether numbers are prime. When asked to check primes, you must call the check_prime tool with a list of integers. Never attempt to determine prime numbers manually. Return the prime number results to the root agent. tools: - name: ma_llm.check_prime ``` ### Introducing Ackgent: The Agent Config Starter Kit To help teams adopt this powerful paradigm, I’ve created a new GitHub repository: [**Ackgent**](https://github.com/fmind/ackgent). This repository is a demonstration of how to leverage ADK Agent Config within a modern, production-ready Python environment. ![Web Interface of Ackgent with the Internet and Datetime agents](/static/img/articles/ackgent-rapid-agent-development-on-gcp-with-adk-and-agent-config/02.webp) Web Interface of Ackgent with the Internet and Datetime agents This template encapsulates best practices for structuring a project where configuration is the core, supported by a suite of modern development tools. #### Repository Features: A Modern Stack The Ackgent repository is built with efficiency, robustness, and Developer Experience (DX) in mind: - **Modern Python Management with** [**`uv`**](https://www.google.com/search?q=%5Bhttps://github.com/astral-sh/uv%5D%28https://github.com/astral-sh/uv%29&authuser=1) **:** We leverage `uv` (the blazing-fast Python package manager written in Rust) to streamline dependency resolution and virtual environment management, significantly speeding up setup and CI/CD pipelines. - **Task Execution with** [**`just`**](https://www.google.com/search?q=%5Bhttps://github.com/casey/just%5D%28https://github.com/casey/just%29&authuser=1) **:** `just` serves as a convenient command runner, simplifying common tasks like installing dependencies with `just project` or deploying to the cloud with `just deploy`. - **Code Quality Tooling:** Integrated [`pre-commit`](https://www.google.com/search?q=%5Bhttps://pre-commit.com/%5D%28https://pre-commit.com/%29&authuser=1) hooks ensure code quality and consistency from the start, utilizing formatters and linters like `check-toml` , `check-yaml` , or `check-json` . - **`evalset`** **Configuration:** The repository includes ADK capability for defining and running evaluation datasets ([`evalset`](https://google.github.io/adk-docs/evaluate/)), crucial for rigorously testing and benchmarking agent performance iteratively. - **Customizable Cloud Run Deployment:** Designed for scalability, the example includes configurations and `just` recipes for deploying the agents as serverless containers on [Google Cloud Run](https://cloud.google.com/run?authuser=1). - **Advanced Agent Capabilities:** Demonstrations of **Tools**, **Callbacks**, and **MCP** integration within the Agent Config framework. ![Ackgent on Cloud Run gives you access to key metrics, logs, SLOs and Error to better observe your agent in production](/static/img/articles/ackgent-rapid-agent-development-on-gcp-with-adk-and-agent-config/03.webp) Ackgent on Cloud Run gives you access to key metrics, logs, SLOs and Error to better observe your agent in production ### Architecture Overview The Ackgent example utilizes a modular architecture centered around the ADK framework. The core concept is the separation of concerns: the agent behavior (the “what”) is defined in YAML files, while the implementations (the “how” — tools, callbacks, and MCP connections) are written in Python. The ADK framework acts as the runtime engine. It parses the Agent Config YAML, initializes the specified LLM, and orchestrates the flow of conversation. When a request is received (e.g., via the Cloud Run endpoint), the runtime identifies the target agent. When the LLM decides to use a tool or delegate to another agent, ADK handles the execution via the implementation references provided in the configuration. This separation of concerns — behavior in YAML, execution handled by ADK, and specialized logic in Python — is what enables rapid iteration. ![Architecture of the Ackgent example with 3 Agents: Root (Dispatcher), Datetime (Tools), and Internet (MCP)](/static/img/articles/ackgent-rapid-agent-development-on-gcp-with-adk-and-agent-config/04.webp) Architecture of the Ackgent example with 3 Agents: Root (Dispatcher), Datetime (Tools), and Internet (MCP) ### Under the Hood: Defining Agents with YAML Let’s look at how this works in practice. The Ackgent repository showcases three distinct agents, demonstrating the core capabilities of the Agent Config approach. Notice how minimal the Python code is, focusing mainly on tool implementation, while the behavior is entirely in YAML. #### 1. The Datetime Agent (Custom Tools) The [datetime agent](https://github.com/fmind/ackgent/blob/main/agent/datetime_agent.yaml) demonstrates how to extend an agent with external tools tools. The agent can access the current date and time defined in the [`tools.py`](https://github.com/fmind/ackgent/blob/main/agent/tools.py) of the repository, which are defined as simple functions. ```yaml # yaml-language-server: $schema=https://raw.githubusercontent.com/google/adk-python/refs/heads/main/src/google/adk/agents/config_schemas/AgentConfig.json name: datetime_agent model: gemini-2.5-flash description: A helpful assistant for datetime questions. instruction: Return the current date or time based on the user's request. generate_content_config: temperature: 0.0 tools: - name: agent.tools.now - name: agent.tools.today """Tools for agents.""" # %% IMPORTS import datetime # %% TOOLS def now() -> str: """Returns the current time. Returns: str: The current time in 'HH:MM' format. """ return datetime.datetime.now().strftime("%H:%M") def today() -> str: """Returns the current date. Returns: str: The current date in 'YYYY-MM-DD' format. """ return str(datetime.date.today()) ``` #### 2. The Internet Agent (Search Tools) The [Internet agent](https://github.com/fmind/ackgent/blob/main/agent/internet_agent.yaml) is configured to access an external MCP Server. In this case, we are using [markdown-mcp](https://github.com/microsoft/markitdown?tab=readme-ov-file), a server developed by Microsoft to quickly retrieve any source into a markdown, including external links. The MCP is started as a STDIO server, with a timeout of 10 seconds. ```yaml # yaml-language-server: $schema=https://raw.githubusercontent.com/google/adk-python/refs/heads/main/src/google/adk/agents/config_schemas/AgentConfig.json name: internet_agent model: gemini-2.5-flash description: A helpful assistant for answering questions from the Internet. instruction: Return the answer to questions using the user provided link. generate_content_config: temperature: 0.0 tools: - name: MCPToolset args: stdio_connection_params: server_params: command: "markitdown-mcp" timeout: 10 ``` #### 3. The Root Agent (Coordination and Routing) The [root agent](https://github.com/fmind/ackgent/blob/main/agent/root_agent.yaml) acts as the main entry point. It doesn't perform tasks itself; instead, its primary function is orchestration. It analyzes the user's intent and intelligently delegates the task to the most appropriate specialized agent using the `sub_agents` configuration. This pattern enables a scalable and modular multi-agent system. ```yaml # yaml-language-server: $schema=https://raw.githubusercontent.com/google/adk-python/refs/heads/main/src/google/adk/agents/config_schemas/AgentConfig.json name: root_agent model: gemini-2.5-flash description: A helpful assistant for user questions. instruction: | You are a helpful assistant that can answer questions about anything. Use the following sub-agents to answer questions: `datetime_agent` and `internet_agent`. generate_content_config: temperature: 0.0 after_model_callbacks: - name: agent.callbacks.after_model_callback sub_agents: - config_path: datetime_agent.yaml - config_path: internet_agent.yaml ``` ### Current Limitations While Agent Config is a great helper for quickly building agents, it’s important to be aware of the current constraints within the ADK framework as it evolves. One notable limitation today involves mixing different types of capabilities within a single agent definition. Currently, you cannot configure an agent that simultaneously uses “Built-In Search Tools” with `google_search` or `VertexAiSearchTool` alongside "Non-Search Tools" (like the custom Python functions) or "Sub-Agents" (like the `root` agent uses). ```yaml tools: # multiple tools are supported only when they are all search tools! - name: google_search - name: VertexAiSearchTool args: data_store_id: "projects/ackgent/locations/us/collections/default_collection/dataStores/reports_123..." ``` The ADK team is actively working on enhancing this flexibility. For now, the recommended architecture — as demonstrated in the Ackgent repository — is either to separate concerns into specialized agents, or create custom search tools (e.g., like the `markitdown-mcp` server). ### The Future: Democratizing Agent Creation **ADK Agent Config is more than just a feature; it’s a foundational shift in agent development from imperative to declarative.** Based on Agent Config, [Ackgent](https://github.com/fmind/ackgent) offers an immediate boost in productivity. It streamlines the development lifecycle, reduces boilerplate, and makes testing and deployment significantly faster. This template repository provides a concrete starting point to leverage these benefits today. But the long-term vision is even more exciting. Because the agent’s behavior is defined declaratively in a structured, human-readable format (YAML), it opens the door for non-technical users — what we might call “digital users” — to build their own agents. Imagine a future where a UI allows business analysts or domain experts to visually construct complex agents by defining instructions and plugging in tools — all powered by Agent Config under the hood. We are moving towards a future where the ability to create AI agents is truly democratized. I encourage you to explore the repository, try out the examples, and experience the speed and simplicity of declarative agent development. - **Link to GitHub Repository**: [https://github.com/fmind/ackgent](https://github.com/fmind/ackgent) ![Source: Gemini App](/static/img/articles/ackgent-rapid-agent-development-on-gcp-with-adk-and-agent-config/05.webp) Source: Gemini App ### [Combo-Banana: Building Custom Image Workflows in Record Time](https://www.fmind.dev/articles/combo-banana-building-custom-image-workflows-in-record-time/) In the fast-paced world of product retail, agility is crucial for the teams bringing products to market. Product designers at my customer handle a massive volume of images daily. Ensuring every product looks perfect across the website, mobile apps, and marketing campaigns often involves tedious, multi-step editing processes — background removal, resizing, color correction, and optimization. While essential, these repetitive tasks can consume hours, diverting designers from the creative work they do best. What if designers could automate these specific workflows themselves, without wrestling with complex software or waiting for engineering resources? ![Source: Nano Banana](/static/img/articles/combo-banana-building-custom-image-workflows-in-record-time/cover.webp) Source: Nano Banana This challenge inspired a recent project: [**Combo-Banana**](https://github.com/fmind/combo-banana). A simple open-source prototype based on [Google's Nano Banana](https://blog.google/products/gemini/updated-image-editing-model/) designed to demonstrate just how quickly we can build applications that deliver immediate value to our teammates on the field. This project is about empowering designers to create their own multi-step image editing pipelines. ### The Use Case: Beyond Manual Editing Imagine a designer preparing images for a new product line. The workflow is predictable but labor-intensive: 1. Receive raw photos from the studio. 2. Manually isolate the product from the background. 3. Adjust the lighting and contrast to meet brand guidelines. 4. Resize and crop for the product detail page (high resolution). 5. Integrate the products in several situations (e.g., on a user, in a store). When done manually across hundreds of SKUs, this process is slow and prone to inconsistencies. This prototype reimagines that process. Instead of a series of manual actions across different tools, the designer defines a “combo” — a sequence of operations executed automatically by the application. ```json { "name": "Social Media Ad Creation", "steps": [ { "title": "Place Item in Landscape", "prompt": "Integrate the product or item seamlessly into a visually stunning and appropriate landscape background, ensuring realistic lighting and perspective." }, { "title": "Add Catchy Slogan", "prompt": "Overlay a concise and catchy slogan onto the image, using a font and placement that enhances readability and visual appeal for a social media ad." } ] } ``` ### The Experience: Flexibility Meets Simplicity The prototype focuses on a streamlined experience. A user can upload an image and stack the desired operations. They define the recipe once — e.g., Step 1: Isolate Product; Step 2: Improve the Shadows; Step 3: Add a Slogan — and the application handles the rest. This transforms a 15-minute manual task into a 30-second automated process, ensuring pixel-perfect consistency across the entire product catalog and freeing up time for more creative work. #### See it in Action The prototype illustrates how an intuitive interface can abstract away the complexity running in the background. ![Combo-Banana: Workflow Definition Tab](/static/img/articles/combo-banana-building-custom-image-workflows-in-record-time/02.webp) Combo-Banana: Workflow Definition Tab On the left, the user defines the workflow with a chatbot interface based on [Gemini 2.5 Flash](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash). The chatbot extracts prompts into a series of steps that are stacked sequentially. In this example, we start with a “Place the item in a landscape” step, followed by a “Add Catchy Slogan” step, powered by [Nano Banana](https://ai.google.dev/gemini-api/docs/image-generation). ![Combo-Banana: Workflow Definition Tab](/static/img/articles/combo-banana-building-custom-image-workflows-in-record-time/03.webp) Combo-Banana: Workflow Definition Tab Once the desired “combo” is configured, the user simply uploads the source image on the top left side of the second tab. The application processes the image through the defined pipeline — the output of the first step becomes the input for the next. The final result is displayed on the right, ready for download. This visual feedback loop allows designers to quickly iterate on their workflows before applying them to large batches of images. ![Final Result of the User Combo](/static/img/articles/combo-banana-building-custom-image-workflows-in-record-time/04.webp) Final Result of the User Combo ### Under the Hood: The Tech Stack The speed of development was thanks to a modern, efficient tech stack. We focused on rapid prototyping, leveraging powerful AI, and ensuring scalability: ![Architecture of Combo-Banana](/static/img/articles/combo-banana-building-custom-image-workflows-in-record-time/05.webp) Architecture of Combo-Banana - **The Interface:** [**Gradio**](https://gradio.app/) Used to build the interactive web UI entirely in Python, avoiding the need for complex front-end development and significantly speeding up iteration. - **The Backend:** [**Python**](https://www.python.org/) The backbone of the application, handling core logic and orchestrating the sequence of image processing steps. - **The Engine:** [**Nano Banana**](https://ai.google.dev/gemini-api/docs/image-generation) The AI powerhouse driving complex tasks like high-fidelity background removal and segmentation. This project was a fantastic opportunity to leverage its impressive capabilities. In future releases, other models could with combined with Nano-Banana. - **Deployment:** [**Google Cloud Run**](https://cloud.google.com/run?authuser=1) A serverless platform ensuring the tool is accessible, cost-effective (scales to zero), and scalable on demand within an organization’s infrastructure. ### The Road Ahead: From Prototype to Platform This prototype is just the beginning. The goal is to evolve it into a robust platform that can handle the complexity of real-world production environments. Key opportunities for evolution include: - **Advanced Workflows (DAGs):** Moving beyond simple sequential pipelines (Step A -\> Step B -\> Step C) to support Directed Acyclic Graphs (DAGs). This would allow for parallel processing — for example, generating five different resolutions simultaneously after the background has been removed. - **Granular Configuration:** Providing deeper configuration options within each processing block (e.g., setting specific compression levels, defining padding for auto-crops, or choosing different AI models for specific tasks and which previous image to use). - **Ecosystem Integration:** Integrating directly with existing asset management tools. This includes pulling source files from **Google Drive** and automatically exporting the results to designated folders or downstream systems. - **User Sessions and Workflow Management:** Implementing user authentication to allow teammates to save, name, share, and reuse their custom workflows, eliminating the need to rebuild them for every session. ### The Bigger Picture: Bridging the Gap Building this prototype underscored a critical insight. We are living in a time with access to incredibly powerful technology like Nano Banana. The technology is here, and it works. However, the existence of a powerful model is not enough. The key challenge now is to **bridge the gap** between these technological capabilities and the real-world, day-to-day needs of our colleagues on the field. As this project demonstrates, we don’t need massive engineering teams or long development cycles to deliver significant value. By identifying specific pain points and leveraging modern tools like Gradio and Cloud Run, we can rapidly prototype solutions that make a difference. This is a phenomenal opportunity for builders and entrepreneurs within any organization. **The tools are ready. It’s time to build!** - **Github Repository**: [https://github.com/fmind/combo-banana](https://github.com/fmind/combo-banana) ![Source: Combo-Banana](/static/img/articles/combo-banana-building-custom-image-workflows-in-record-time/06.webp) Source: Combo-Banana ### [Vibe Youtubing with NotebookLM: The MLOps Coding Course Gets a Video Upgrade in Under 48 Hours](https://www.fmind.dev/articles/vibe-youtubing-with-notebooklm-the-mlops-coding-course-gets-a-video-upgrade-in-under-48-hours/) I’m incredibly excited to announce the launch of the first video series for my [**MLOps Coding Course**](https://mlops-coding-course.fmind.dev/), covering the introduction and the entire first chapter on [AI/ML Project Initialization](https://mlops-coding-course.fmind.dev/1.%20Initializing/index.html)! But this announcement is about more than just new content. It’s the story of how a powerful AI tool transformed my workflow, turning a mountain of a task into a manageable and even enjoyable creative process. # An error occurred. Unable to execute JavaScript. In this article, I want to share my journey of using [**NotebookLM’s new Video Overview feature**](https://blog.google/technology/google-labs/notebook-lm-audio-video-overviews-more-languages-longer-content/). I’ll take you behind the scenes of how I turned a comprehensive, text-based course into engaging video content at a speed I never thought possible, and how it has fundamentally changed my perspective on content creation you are not specialized in. ### The Challenge: A Mountain of Content to Visualize I poured my heart and soul into creating the [MLOps Coding Course](https://github.com/MLOps-Courses/mlops-coding-course), spending over six months meticulously crafting the content in Markdown and Python. My goal has always been to make MLOps accessible and practical to everybody on the Internet, but I knew that video would take the learning experience to the next level. The problem? The thought of scripting, recording, and editing videos for every single section was daunting. I estimated it would take me at least a month of dedicated work to produce the first batch alone. ![Using NotebookLM is like casting a crew of robots to record a video](/static/img/articles/vibe-youtubing-with-notebooklm-the-mlops-coding-course-gets-a-video-upgrade-in-under-48-hours/cover.webp) Using NotebookLM is like casting a crew of robots to record a video ### The Game-Changer: NotebookLM’s Video Overview This is where Google’s [**NotebookLM**](https://notebooklm.google.com/) came in and completely changed the game for me. Since my entire course was already in Markdown, the process was incredibly straightforward. I imported my content, and from there, I could generate a video for each section individually. The results were astounding! ![Capture of NotebookLM for the MLOps Coding Course](/static/img/articles/vibe-youtubing-with-notebooklm-the-mlops-coding-course-gets-a-video-upgrade-in-under-48-hours/02.webp) Capture of NotebookLM for the MLOps Coding Course The process was unbelievably fast, clocking in at about **5–10 minutes per video**. But it wasn’t just about speed. I was blown away by the quality. NotebookLM didn’t just read my text; it understood the context. It automatically generated insightful charts and tables from my data, and even grasped nuanced MLOps challenges I had written about, like the classic “but it runs on my machine” problem. It felt less like a text-to-video converter and more like a true creative partner. ### The “One-Man Army” Experience: Human Curation, AI Generation This new workflow transformed my role from a content creator buried in the mechanics of production to an _editorialist_. My focus shifted to curating and refining the AI-generated content, ensuring it perfectly matched the tone, intent, and high standards of the course. ![Content of the MLOps Coding Course as a MindMap Generated by NotebookLM](/static/img/articles/vibe-youtubing-with-notebooklm-the-mlops-coding-course-gets-a-video-upgrade-in-under-48-hours/03.webp) Content of the MLOps Coding Course as a MindMap Generated by NotebookLM It was a profound experience. [NotebookLM](https://notebooklm.google.com/) respected the six months of effort I put into writing the material, using it as a solid foundation to build upon. I was able to produce the entire first set of videos in just **two days**. This wasn’t just a productivity boost; it made me feel like a one-man army, capable of executing a project that would have otherwise required a small team or a significant budget. ### The Future is a Hybrid of Human and Machine This experience has solidified my belief that we are in a new era of content creation. Tools like NotebookLM don’t replace human expertise; they _amplify_ it. They empower us to focus on our unique strengths — our knowledge, our creative vision, our deep understanding of the subject — while automating the tedious and time-consuming tasks. ![Source: https://blog.google/technology/google-labs/notebook-lm-audio-video-overviews-more-languages-longer-content/](/static/img/articles/vibe-youtubing-with-notebooklm-the-mlops-coding-course-gets-a-video-upgrade-in-under-48-hours/04.webp) Source: [https://blog.google/technology/google-labs/notebook-lm-audio-video-overviews-more-languages-longer-content/](https://blog.google/technology/google-labs/notebook-lm-audio-video-overviews-more-languages-longer-content/) I see immense potential for this technology in both professional and personal projects. It’s a true productivity superpower. **I highly encourage you to try it out,** especially now that [NotebookLM’s video and audio overviews are available in over 80 languages](https://blog.google/technology/google-labs/notebook-lm-audio-video-overviews-more-languages-longer-content/), opening up these capabilities to a global audience. ### What’s Next? This is just the beginning! I’ll be releasing more video overviews for the MLOps Coding Course in the coming months. Stay tuned for updates. In the meantime, I’d love to hear from you. Have you tried NotebookLM’s video overview feature? What are your experiences with AI-assisted content creation? Let’s discuss in the comments below! - **MLOps Coding Course**: [https://mlops-coding-course.fmind.dev/](https://mlops-coding-course.fmind.dev/) - **GitHub (** [**CC-BY-4.0 license**](https://github.com/MLOps-Courses/mlops-coding-course#) **)**: [https://github.com/MLOps-Courses/mlops-coding-course](https://github.com/MLOps-Courses/mlops-coding-course) - **Playlist for the 1st Chapter on Initialization**: [https://www.youtube.com/playlist?list=PLPCnNL6Y2PbTxRS24rkQ7mATQXjbR0PU7](https://www.youtube.com/playlist?list=PLPCnNL6Y2PbTxRS24rkQ7mATQXjbR0PU7) ![Vibe Youtubing with NotebookLM: The MLOps Coding Course Gets a Video Upgrade in Under 48 Hours](/static/img/articles/vibe-youtubing-with-notebooklm-the-mlops-coding-course-gets-a-video-upgrade-in-under-48-hours/05.webp) ### [Deploying AI Agents in the Enterprise using ADK and Google Cloud](https://www.fmind.dev/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/) ### **Deploying AI Agents in the Enterprise without** Losing **your Humanity u**sing ADK and Google Cloud The excitement around AI agents is palpable. From automating complex workflows to providing personalized experiences, the potential is enormous. But as Data Scientists and developers rush to build proof-of-concepts, a significant hurdle emerges when it’s time to deploy these agents within a real-world organization. I’ve recently been tackling these exact challenges with my customers. The goal is clear: find solutions that are simple, powerful, and accessible for the whole team. But moving from a local demo to a production-ready agent involves navigating the labyrinth of enterprise IT. AgentOps is uniquely challenging because it combines the inherent complexity of non-deterministic systems (like Machine Learning) with the stringent requirements of enterprise security and the need to expose these applications securely to a wide audience. In this article, we’ll explore the challenges of deploying agents in a corporate environment and dive into practical strategies using [Google Cloud Platform (GCP)](https://cloud.google.com/gcp?authuser=1), based on the findings and code in my [GitHub Repository](https://github.com/fmind/search-agent/tree/main/search_agent). ![Source: Gemini App](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/cover.webp) Source: Gemini App ### The AgentOps Challenge: Bridging the Gap Between POC and Production 🛡️ Deploying an application in an organization isn’t just about making the code run on a server. It’s about integrating it seamlessly and securely into the existing IT ecosystem. This means considering: - **Networking:** Your agent likely needs to live within a [Virtual Private Cloud (VPC)](https://cloud.google.com/vpc/docs/overview?authuser=1), behind firewalls, and potentially protected by a [Web Application Firewall (WAF)](https://cloud.google.com/security/products/waf?authuser=1). - **Authentication Mechanisms:** How do users access the agent? Organizations often use complex identity providers (IdPs) leveraging protocols like [OIDC (OpenID Connect)](https://openid.net/connect/) or [SAML (Security Assertion Markup Language)](https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language). - **Security and Risk:** Exposing an unauthenticated application on the public internet is a recipe for disaster, potentially leaking sensitive data or allowing unauthorized access to internal systems. A critical aspect of AgentOps is **identity propagation**. If an agent is designed to access a user’s resources (like reading their emails or calendar), it must be able to verify _who_ the user is. This requires robust authentication that connects the user’s identity from the front-end all the way through to the agent’s backend logic, often managed through systems like [IAM (Identity and Access Management)](https://cloud.google.com/iam/docs/overview?authuser=1). Furthermore, a production-ready agent needs multiple exposition mechanisms: 1. [**A Web UI**](https://github.com/google/adk-web) **:** For quick testing and human interaction. 2. [**An API Endpoint**](https://google.github.io/adk-docs/get-started/testing/) **:** To integrate the agent into other applications and services. 3. [**Agent-to-Agent (A2A) Protocol**](https://google.github.io/adk-docs/a2a/) **:** For interoperability and complex multi-agent systems. ![Source: https://codelabs.developers.google.com/intro-a2a-purchasing-concierge#0](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/02.webp) Source: [https://codelabs.developers.google.com/intro-a2a-purchasing-concierge#0](https://codelabs.developers.google.com/intro-a2a-purchasing-concierge#0) Achieving all three securely is the core of the AgentOps exposition challenge. ### Exploring Deployment Strategies on GCP 🚀 Google Cloud provides both powerful models (like [Gemini](https://deepmind.google/technologies/gemini/), accessed via [Vertex AI](https://cloud.google.com/vertex-ai?authuser=1)) and robust infrastructure solutions. We explored several paths to deploy a sample search agent, evaluating them based on flexibility, security, and ease of use. The architecture diagram below illustrates the two main paths I evaluated for secure deployment on GCP: ![Architecture Diagram with the 2 Deployment Paths: Vertex AI Agent Engine and Cloud Run (Source: fmind.dev)](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/03.webp) Architecture Diagram with the 2 Deployment Paths: Vertex AI Agent Engine and Cloud Run (Source: fmind.dev) The diagram highlights how clients (either end-users via a browser or services/other agents via API calls) interact with the agent logic. Both paths rely on [GCP’s IAM](https://cloud.google.com/security/products/iam) to manage permissions, but they handle exposition and authentication differently. - **Path 1:** [**Cloud Run**](https://cloud.google.com/run?hl=en) **(Flexible):** This approach deploys the agent in a containerized environment. It uses [Google IAP](https://cloud.google.com/security/products/iap?hl=en) (Identity-Aware Proxy) as a gatekeeper to authenticate all incoming traffic, securing the Web UI and the API/A2A endpoints uniformly. This provides maximum control and versatility. - **Path 2:** [**Vertex AI Agent Engine**](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview) **(Opinionated):** This approach uses a managed service specifically designed for agents. Authentication is handled by the service’s API gateway, granting access to clients with the appropriate IAM roles, but it is limited to API exposition. In our example we are going to deploy [a simple search agent](https://github.com/fmind/search-agent/blob/main/search_agent/agent.py) with ADK: ```python """A simple search agent.""" # %% IMPORTS from google.adk.agents import Agent from google.adk.tools import google_search # %% AGENTS root_agent = Agent( name="search_agent", model="gemini-2.5-flash", description="Agent to answer questions using Google Search.", instruction="You are an expert researcher. You always stick to the facts.", # use the builtin google_search tool from ADK tools=[google_search], ) ``` Let’s dive into the details, pros, and cons of these approaches, as well as other alternatives. #### Option 1: The Wild West — Unauthenticated Internet Deployment 🌵 The easiest path is often the most dangerous. Cloud providers make it simple to deploy an application and expose it to the public internet without authentication. **Verdict:** While fast for development, this is generally unacceptable for enterprise deployment unless stringent security practices (both internal application security and external network security) are rigorously implemented and audited. We won’t explore this further in the code repository, as our focus is on secure deployment. #### Option 2: The Opinionated Path — Vertex AI Agent Engine 🤖 [Vertex AI Agent Engine](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview?authuser=1) offers a managed, opinionated approach to deploying agents (Path 2 in the architecture diagram). ```justfile # deploy to agent engine deploy-agent-engine: uv run adk deploy agent_engine --project=$GOOGLE_CLOUD_PROJECT --region=$GOOGLE_CLOUD_LOCATION --staging_bucket=$STAGING_BUCKET --trace_to_cloud \ --display_name={{AGENT}} --description={{AGENT}} {{env('AGENT_ENGINE_ID', '') && "--agent_engine_id=" + env('AGENT_ENGINE_ID')}} {{AGENT}} ``` ![List of Agents on Vertex AI Agent Engine](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/04.webp) List of Agents on Vertex AI Agent Engine **Pros:** - **Out-of-the-box:** Streamlines the deployment process. - **Monitoring:** Provides a dedicated console dashboard to monitor agent performance and sessions. - **Authentication:** Offers relatively easy mechanisms to [authenticate](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/manage/access?authuser=1) and use the agent from other applications via REST API or the [Python SDK](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/use/adk?authuser=1#vertex-ai-sdk-for-python). ![Search Agent running on Vertex AI Agent Engine](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/05.webp) Search Agent running on Vertex AI Agent Engine **Cons:** - **Flexibility:** You cannot deploy a custom Web UI or an A2A endpoint directly; it only exposes a REST API. - **Control:** Less control over the underlying resources. - **Pricing:** The [pricing model](https://cloud.google.com/vertex-ai/pricing?authuser=1#agent_engine) per hardware is less flexible compared to request-based solutions. **Verdict:** A great starting point for deploying API-based agents, but a lack of flexibility for UI and A2A exposition. As the product become more mature, it should become the go-to option on GCP. #### Option 3: The Sweet Spot — Cloud Run 🏃‍♀️ [Cloud Run](https://cloud.google.com/run/docs/overview/what-is-cloud-run?authuser=1) is Google Cloud’s serverless container platform, and it emerged as the best trade-off for deploying agents today (Path 1 in the architecture diagram). ```justfile # deploy to cloud run deploy-cloud-run: # when asked "Allow unauthenticated invocations to [search-agent] (y/N)?", answer "n" adk deploy cloud_run --project=$GOOGLE_CLOUD_PROJECT --region=$GOOGLE_CLOUD_LOCATION --trace_to_cloud \ --service_name={{replace(AGENT, '_', '-')}} --app_name={{AGENT}} --with_ui --a2a {{AGENT}} ``` ![List of Services deployed on Cloud Run](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/06.webp) List of Services deployed on Cloud Run **Pros:** - **Simplicity:** Cloud Run is incredibly easy and convenient to use, often requiring just a single CLI command for deployment. - **Total Control:** You manage everything, from the hardware used (CPU, GPU) to scalability settings and exposition mechanisms. - **Versatility:** You can expose a Web UI, an API server, and an A2A endpoint all from the _same_ instance, simplifying maintenance. - **Pricing:** The [pricing](https://cloud.google.com/run/pricing?hl=en&authuser=1) is highly flexible — you pay per resource consumed, and it can scale down to zero. - **Ecosystem:** Cloud Run integrates seamlessly with other GCP services and provides excellent operational features: Dashboards, Logging, Revision Management, SLOs, and Alerting. ![Agent running on Cloud Run](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/07.webp) Agent running on Cloud Run **Verdict:** Currently the best option. It provides the necessary control to implement robust security and versatile exposition while maintaining the ease of use of a serverless platform. The [ADK documentation](https://google.github.io/adk-docs/deploy/cloud-run/) also provides guidance on this path. #### Option 4: The Future — AgentSpace 🔮 The main problem of AgentOps, as discussed, is exposition and identity management. This is where [Google AgentSpace](https://cloud.google.com/products/agentspace?hl=en&authuser=1) comes in. AgentSpace is an intranet search, AI assistant, and agentic platform. On top of that, it can solve our challenges by adding an exposition and identity layer on top of Vertex AI Agent Engine, complementing it perfectly. Over time, it might become the best solution for integrating agents while developers can focus on building the backend systems. ![Integration of Agent Engine and AgentSpace: https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/08.webp) Integration of Agent Engine and AgentSpace: [https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview) **Verdict:** AgentSpace and Vertex AI Engine are the future integrated solution for streamlined, secure agent deployment on GCP. As of August 2025, these solutions need to mature a bit more to catch up with the flexibility offered by Cloud Run. ### Deep Dive: Securing Agents with IAP on Cloud Run 🔒 When choosing Cloud Run, we still need a way to secure it. We want a single solution that provides authentication for the Web UI, the API, and A2A communication, ensuring we know who (or what) is interacting with our agent. While options like implementing custom authentication within the application using OIDC or SAML exist, they add significant complexity to the codebase. For the repository, we chose Google’s [**Identity-Aware Proxy (IAP)**](https://cloud.google.com/iap/docs/concepts-overview?authuser=1). #### Why IAP? IAP is a [zero-trust](https://cloud.google.com/beyondcorp?authuser=1) access solution that allows you to manage access to applications running on GCP. It acts as a gatekeeper (as shown in Path 1 of the architecture diagram), verifying user identity and context before authorizing access to the application. - **Simplicity:** It works great out of the box without requiring complex changes to the application code. - **Versatility:** It provides authentication both for end-users (via browser redirects) and services (via headers). - **Identity Propagation:** IAP securely passes the authenticated user’s identity to the application, which is key for AgentOps to associate a user with their resources. #### End-User Authentication (Web UI) 👩‍💻 Configuring IAP for a Cloud Run service is straightforward. You enable IAP on the service and define which principals (users, groups or domains) are allowed access via IAM on the Security Tab. ![Identity Access Proxy (IAP) Management on Cloud Run](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/09.webp) Identity Access Proxy (IAP) Management on Cloud Run When a user tries to access the Web UI, IAP intercepts the request, authenticates the user (usually via Google Sign-In, though [external identities](https://cloud.google.com/iap/docs/external-identities?authuser=1) are supported), and grants access if authorized. ![Google Sign-In Page to Authenticate the End User](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/10.webp) Google Sign-In Page to Authenticate the End User ![Agent Web UI with ADK after Sign-In](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/11.webp) Agent Web UI with ADK after Sign-In You can find more details in the [Google documentation on securing Cloud Run with IAP](https://cloud.google.com/run/docs/securing/identity-aware-proxy-cloud-run?authuser=1). #### Service Authentication (API and A2A) 🤝 For [programmatic access](https://cloud.google.com/run/docs/authenticating/developers?authuser=1) (API calls or A2A communication), the authentication flow is different. Services typically use [service accounts](https://cloud.google.com/iam/docs/service-accounts?authuser=1) and must provide an OpenID Connect (OIDC) token (a [JWT](https://jwt.io/introduction)) in the `Authorization` header of the request. IAP validates this token (which must be signed by Google and have the correct audience — the Cloud Run URL or IAP Client ID) before allowing the request to reach the agent. This is crucial for secure A2A communication in the enterprise. Agents need to discover each other (often via an [Agent Card](https://github.com/fmind/search-agent/blob/main/search_agent/agent.json), as shown below) and communicate securely. ![Search Agent — Agent Card for the A2A Protocol](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/12.webp) Search Agent — Agent Card for the A2A Protocol To make this work, the calling agent needs to generate a valid JWT token. Here is a Python snippet demonstrating how to achieve this when calling an IAP-protected endpoint ([from the GitHub Repository User Agent](https://github.com/fmind/search-agent/blob/main/user_agent/agent.py)): ```python """User-facing agent that delegates search queries to a remote A2A agent.""" # %% IMPORTS import datetime import json import os import google.auth import httpx from google.adk.agents.llm_agent import Agent from google.adk.agents.remote_a2a_agent import ( AGENT_CARD_WELL_KNOWN_PATH, RemoteA2aAgent, ) from google.cloud import iam_credentials_v1 # %% ENVIRONS # URL to the Agent Card. See: https://google.github.io/adk-docs/a2a/quickstart-consuming/#how-it-works # It's the entry point for the user-facing agent to discover and interact with the remote agent. AGENT_CARD = os.getenv( "AGENT_CARD", f"http://localhost:8000/a2a/search_agent{AGENT_CARD_WELL_KNOWN_PATH}" ) # Email of the calling GCP Service Account (SA) # This is used to authenticate to the remote agent. AGENT_RUN_SA = os.environ["AGENT_RUN_SA"] # %% CLIENTS # Authenticate to Google Cloud using the default credentials. # This is necessary to use the IAM Credentials API to sign JWTs. credentials, project_id = google.auth.default() iam_client = iam_credentials_v1.IAMCredentialsClient(credentials=credentials) def get_auth_token(url: str, exp: int = 3600) -> str: """Gets an auth token for a given URL with a expiry time (in seconds). The JWT contains the following claims: - aud: The audience of the token, which is the URL of the remote agent. - iss: The issuer of the token, which is the service account. - sub: The subject of the token, which is also the service account. - iat: The time the token was issued (issued at). - exp: The time the token expires (expiration time). Args: url: The URL of the remote agent to authenticate to. exp: The expiration time of the token in seconds. Returns: The signed JWT. """ # Get the current time. iat = datetime.datetime.now(tz=datetime.timezone.utc) # Set the expiration time. exp = iat + datetime.timedelta(seconds=exp) # Create the JWT payload. jwt = { "aud": url, "iss": AGENT_RUN_SA, "sub": AGENT_RUN_SA, "iat": int(iat.timestamp()), "exp": int(exp.timestamp()), } # Convert the JWT to a JSON string. payload = json.dumps(jwt) # Get the full name of the service account. name = iam_client.service_account_path("-", AGENT_RUN_SA) # Sign the JWT using the IAM Credentials API. response = iam_client.sign_jwt(name=name, payload=payload) # Return the signed JWT. return response.signed_jwt class BearerAuth(httpx.Auth): """A custom httpx authentication class that uses a bearer token.""" def auth_flow(self, request): """Adds the Authorization header to the request. Args: request: The request to add the Authorization header to. Yields: The request with the Authorization header. """ # Get a new auth token for the request's URL. token = get_auth_token(str(request.url)) # Add the Authorization header to the request. request.headers["Authorization"] = f"Bearer {token}" # Yield the request to httpx to be sent. yield request # Create an httpx client with the custom bearer authentication. httpx_client = httpx.AsyncClient(auth=BearerAuth(), timeout=600) # %% AGENTS # Create a remote A2A agent that represents the remote search agent. # This agent will delegate calls to the remote agent's tools. search_agent = RemoteA2aAgent( name="search_agent", agent_card=AGENT_CARD, description="Google Search Agent", httpx_client=httpx_client, ) # Create a root agent that orchestrates the interaction with the user. # This agent will delegate search queries to the remote search agent. root_agent = Agent( name="root_agent", model="gemini-2.5-flash", instruction="You are a nice and polite agent. Deleguate search query to the search_agent.", sub_agents=[search_agent], ) ``` _Note: The JWT token works only for a specific audience (URL/Client ID), ensuring that a token generated for one service cannot be reused for another._ ![User Agent calling the Search Agent deployed on Cloud Run using the A2A Protocol](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/13.webp) User Agent calling the Search Agent deployed on Cloud Run using the A2A Protocol ### Conclusion: The AgentOps Gold Rush 🌟 Deploying AI agents in the enterprise is far from trivial. The intersection of machine learning’s unpredictability and the rigid requirements of corporate IT creates a unique set of challenges. We are still in the early days of **AgentOps**. There is a pressing need for new architectures, deployment patterns, and tools to facilitate this process. I believe there will be a _gold rush_ for those with the technical prowess to build robust, secure, and scalable IT systems for agents. This journey won’t be as easy as some might think, but leveraging powerful and flexible tools like Cloud Run and the broader Google Cloud ecosystem definitely provides a significant advantage. As we move forward, remember the K.I.S.S. principle: Keep It Simple (Stupid). Simple, well-understood solutions like Cloud Run combined with IAP can offer the best path to bringing the power of AI agents securely into the enterprise today. _Explore the code and deployment configurations discussed in this article on my GitHub repository:_ [https://github.com/fmind/search-agent](https://github.com/fmind/search-agent) ![Source: Gemini App](/static/img/articles/deploying-ai-agents-in-the-enterprise-using-adk-and-google-cloud/14.webp) Source: Gemini App ### [Happy Birthday XP: Celebrating Gemini Deep Think (and My Daughter’s 6th Birthday)](https://www.fmind.dev/articles/happy-birthday-xp-celebrating-gemini-deep-think-and-my-daughters-6th-birthday/) Birthdays are always milestones. But this year, my daughter Augustine’s 6th birthday coincided with another significant event: a staggering leap forward in AI capabilities: [the release of Gemini Deep Think](https://blog.google/products/gemini/gemini-2-5-deep-think/). To celebrate, I decided to combine the two. I crafted a complex coding challenge and presented it to the three tiers of Google’s Gemini 2.5 models: Flash, Pro, and the powerful new Deep Think capability. The goal was simple: Could AI create a genuinely magical, interactive experience for a six-year-old? ![Source: Gemini App](/static/img/articles/happy-birthday-xp-celebrating-gemini-deep-think-and-my-daughters-6th-birthday/cover.webp) Source: Gemini App The results were more than just a birthday gift; they were a clear demonstration of the accelerating pace of AI. As Simon Willison recently illustrated in his talk [“2025 in LLMs so far, illustrated by Pelicans on Bicycles,”](https://www.youtube.com/watch?v=YpY83-kA7Bo) we are seeing models make sudden, incredible leaps in capability. This experiment was a perfect example of that leap in action. You can explore the results yourself on the [Happy Birthday XP](https://sites.google.com/fmind.dev/happy-birthday-xp/home) site or check out the full [YouTube Playlist here](https://www.youtube.com/playlist?list=PLPCnNL6Y2PbRZCXG2miPerG75T_1uj78E). ### The Challenge: The “Magical Birthday Adventure” Prompt I didn’t ask for a simple webpage. I asked for a multi-scene, interactive experience, demanding coordination between animation, user interaction, and programmatically generated audio. The major constraint: The entire experience had to be contained within a single HTML file (HTML, CSS, JS). Here is a summary of the [prompt requirements](https://sites.google.com/fmind.dev/happy-birthday-xp/home): - **Scene 1: The Enchanted Gift.** A twilight sky and a shimmering gift box that acts as a start button. - **Scene 2: The Grand Unveiling.** The box opens, releasing particles that assemble to spell “Happy 6th Birthday, Augustine!” - **Scene 3: A Sky Full of Joy.** A fireworks spectacle where the user can click to launch more fireworks. Simultaneously, balloons float up, which the user can pop for confetti and sound. - **Scene 4: The Birthday Wish.** A cake with 6 candles slides in. The user clicks “Make a wish!” to blow them out. - **Scene 5: The Grand Finale.** Confetti falls, the “Happy Birthday” song plays, and a final message appears. This is a tough prompt. It requires understanding narrative flow, complex JavaScript interactions, CSS animation timing, and synthesizing audio. ![Source: https://developers.googleblog.com/en/start-building-with-gemini-25-flash/](/static/img/articles/happy-birthday-xp-celebrating-gemini-deep-think-and-my-daughters-6th-birthday/02.webp) Source: [https://developers.googleblog.com/en/start-building-with-gemini-25-flash/](https://developers.googleblog.com/en/start-building-with-gemini-25-flash/) Let’s see how the models did. ### Round 1: Gemini 2.5 Flash [Gemini 2.5 Flash](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash?hl=fr) is designed for speed and efficiency. It’s great for basic tasks, but complex, creative coding is not its primary strength. # An error occurred. Unable to execute JavaScript. _Watch the Flash result:_ [_https://youtu.be/g6KLIY7hy-I_](https://youtu.be/g6KLIY7hy-I) Flash grasped the very basics of the request, but not much more. **The Good:** - It created the initial scene: A blue background and a box labeled “Open Me!” (00:00). - Clicking the box triggers a transition (00:03). - The birthday message (“Happy 6th Birthday, Augustine!”) appears with a glow effect (00:06). **The Bad:** - The aesthetics are extremely basic — just a pink square on a blue background. - It missed almost all the required interactivity and complexity. - There are no fireworks, no balloons, no cake, and no multi-scene coordination. - The transition is an abrupt fade to black. **The Verdict:** Flash delivered a functional webpage, but the “magic” was missing. It followed the first two instructions and then essentially gave up. ### Round 2: Gemini 2.5 Pro [Gemini 2.5 Pro](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro?hl=fr) is the workhorse model, balancing capability with performance. I expected an improvement over Flash, and I got one. # An error occurred. Unable to execute JavaScript. _Watch the Pro result:_ [_https://youtu.be/4t0Hm_zeGqc_](https://youtu.be/4t0Hm_zeGqc) Pro provided a better aesthetic and a closer attempt at the prompt. **The Good:** - The visuals are much improved. The initial scene has a starry twilight background as requested (00:00). - The text reveal has a nice neon glow effect (00:05). - Crucially, it implemented a fireworks spectacle (00:10). The fireworks look good and are varied. **The Bad:** - The gift box CSS is slightly buggy (the lid is misplaced and the text is overlapping at the start 00:00). - It completely missed the interactive balloon popping game. - It skipped the entire birthday cake and candle sequence. - It struggled with the text formatting, displaying “Happy6thBirthday,Augustine!” **The Verdict:** A solid effort. It looks better and includes one of the major visual elements (the fireworks). However, it still failed to manage the complexity of the entire multi-stage request. ### The Ultimate Critic: A 6-Year-Old’s Review (Part 1) I showed both the Flash and Pro versions to Augustine. She clicked the button, saw the text, watched the fireworks for a moment on the Pro version, and then looked at me and asked, “Is that it?” ![Source: Gemini App](/static/img/articles/happy-birthday-xp-celebrating-gemini-deep-think-and-my-daughters-6th-birthday/03.webp) Source: Gemini App Unmoved. The magic had failed to land. ### Round 3: Gemini DeepThink Enter [Deep Think](https://blog.google/products/gemini/gemini-2-5-deep-think/). This represents the cutting edge of Gemini’s capabilities, utilizing advanced reasoning and complex instruction following. The difference was immediate and profound. # An error occurred. Unable to execute JavaScript. _Watch the DeepThink result:_ [_https://youtu.be/CA55UQ0tGrQ_](https://youtu.be/CA55UQ0tGrQ) Deep Think didn’t just execute the instructions; it understood the _intent_ of creating a magical experience. **The Execution:** - **Scene Coordination:** The flow is seamless. The initial scene is polished, with a beautiful gift box and ribbon (00:00). - **The Reveal:** When clicked, the box doesn’t just disappear; it bursts open with a flood of stars (00:03) before revealing the glowing birthday message (00:06). - **The Interactive Game:** This is where DeepThink shines. Balloons begin to float up (00:13). Not only can you pop them, but the model implemented _playful, realistic balloon physics_. They bob, they float naturally, and they even bump into the text elements (00:25). They pop with satisfying confetti bursts and synthesized “pop” sound effects (00:14 onwards). - **The Climax:** The balloons fade, and a stylized birthday cake with 6 lit candles glides in (01:07). The “Make a wish!” button appears (01:09). - **The Finale:** Clicking the button blows out the candles with a distinct “whoosh” sound effect (01:12), triggering a synthesized “Happy Birthday” melody and the final, heartwarming message: “We love you! Have the most wonderful day!” (01:14). **The Verdict:** Staggering. Deep Think nailed almost every aspect of the prompt. The attention to detail, the better sound design, the playful physics, and the timing of the animations were all far beyond what Flash and Pro could achieve. ### The Ultimate Critic: A 6-Year-Old’s Review (Part 2) When I showed the DeepThink version to Augustine, her eyes lit up. She was captivated by the opening animation, and she _loved_ the balloon popping game. She giggled as they popped, delighted by the physics and the confetti. The final message, telling her she would have a wonderful day, made her smile. ![Source: Gemini App](/static/img/articles/happy-birthday-xp-celebrating-gemini-deep-think-and-my-daughters-6th-birthday/04.webp) Source: Gemini App She immediately hit “Play Again!” and played through the entire experience four times in a row. Mission accomplished. Magic delivered. ### The Staggering Progress and the Widening Gap This simple birthday experiment highlights a critical trend in the current AI landscape. The difference between the models is not merely incremental; it is **exponential**. A year ago, getting the result that Flash produced would have been notable. Today, it’s underwhelming. While Pro provided a competent result, Deep Think delivered a _delightful_ one. It demonstrates that the frontier models are no longer just completing tasks; they are synthesizing complex requirements and producing results that feel creative and polished. The inclusion of realistic physics in a simple game, generated from a single prompt, is remarkable. We are witnessing a rapidly widening gap in two areas: 1. **The gap between people using AI and those not.** 2. **The gap between people using powerful, frontier models (like Deep Think) and those using basic models for complex tasks (like Flash).** The level of leverage that a model like Gemini Deep Think provides is astonishing. In minutes, it created an experience that would have taken a human developer significant time to code, debug, and polish. As we celebrate Augustine’s 6th birthday, I’m also celebrating the arrival of this incredible technology. The progress is staggering, and I’m thrilled to have such power at my fingertips. The future of AI-assisted creation isn’t just coming; it’s already here. ![Source: https://blog.google/products/gemini/gemini-2-5-deep-think/](/static/img/articles/happy-birthday-xp-celebrating-gemini-deep-think-and-my-daughters-6th-birthday/05.webp) Source: [https://blog.google/products/gemini/gemini-2-5-deep-think/](https://blog.google/products/gemini/gemini-2-5-deep-think/) ### [Slides-To-Translate: When IT Says No, Build a $0.04 Solution on Your Lunch Break](https://www.fmind.dev/articles/slides-to-translate-when-it-says-no-build-a-0-04-solution-on-your-lunch-break/) ### **Slides-To-Translate:** When IT Says No, Build a \$0.04 Solution on Your Lunch Break It is a classic tale of corporate life. A colleague of mine at Decathlon had a mammoth task: **translate a 155-slide presentation into several languages**. In a global company, this is a daily reality. The content was ready, the deadline was looming, but the path forward was blocked. The usual suspects — approved [Chrome](https://chromewebstore.google.com/category/extensions) and [Workspace](https://workspace.google.com/marketplace/) extensions that could automate this — were unfortunately blocked by our organization’s IT policy. Have you ever tried requesting a new tool in a large company? It often feels like sending a message in a bottle into the cosmic ocean. My colleague’s request had, predictably, vanished into a bureaucratic black hole. She was stuck. Manually translating 155 slides is not just tedious; it’s a recipe for errors and missed deadlines! We had seen several similar requests before, but no viable solution was ever found, leaving colleagues to manually copy and paste text from Google Translate … Listening to her predicament, I thought, “There has to be a faster way.” During my lunch break, I decided to take a crack at it. I figured hacking together a quick solution would be faster than waiting for a permission slip. ![Source: Gemini App](/static/img/articles/slides-to-translate-when-it-says-no-build-a-0-04-solution-on-your-lunch-break/cover.webp) Source: Gemini App **And it was. In less than an hour, I had a working solution.** ### The Power Trio: Gemini, Colab, and Vertex AI 🤖✨ My toolkit for this lunch-break challenge was simple but powerful: - 💻 [**Google Colab**](https://colab.research.google.com/): For a ready-to-go coding environment. - ☁️ [**Vertex AI**](https://cloud.google.com/vertex-ai/generative-ai/docs): To access Google’s powerful suite of AI models. - ⚡️ [**Gemini 2.5 Flash**](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash): The star of the show. A fast, multimodal, and incredibly cost-effective model perfect for a task like this. The plan was straightforward: use the [Google Slides](https://developers.google.com/workspace/slides/api/guides/overview) and [Drive APIs](https://developers.google.com/workspace/drive/api/guides/about-sdk) to read the presentation, send the text to Gemini for translation, and then write the translated text back to a new copy of the slide deck. A little bit of coding, especially when supercharged by an AI assistant, can go a very, very long way. ### How it Works: A Peek Under the Hood ⚙️ The solution is a single [Google Colab notebook](https://colab.research.google.com/drive/18FOakT2-IdhebVNPyiKYdbJWgrUyXqU-#scrollTo=ICBTF6Y_Trle) that automates the entire process. For anyone curious, here’s the gist of it. First, you need to authenticate and set up the environment. This is surprisingly easy in Colab: ```python # Authenticate with your Google Cloud Project from google.colab import auth auth.authenticate_user(project_id="your-gcp-project-id") # Build the service clients for Drive and Slides from googleapiclient.discovery import build drive_service = build('drive', 'v3') slides_service = build('slides', 'v1') # And initialize the Vertex AI client from google import genai client = genai.Client(vertexai=True) ``` After creating a safe copy of the presentation and extracting all the unique text strings, the real magic begins. To make the translation process fast, I used a [`ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor) to make concurrent API calls to the Gemini model. The core translation function is simple. It takes a piece of text and a set of instructions, then returns the translation. ```python def translate_text(text): """Translates a single text string.""" try: response = client.models.generate_content( model=MODEL_NAME, contents=text, config=types.GenerateContentConfig( system_instruction=instructions, # e.g., "Translate to German..." temperature=0.0, ) ) return text, response.text.strip() except Exception as error: print(f"Error translating '{text}': {error}") return text, None ``` This function is then called in parallel for every unique piece of text from the slides: ```python from concurrent.futures import ThreadPoolExecutor, as_completed # Use ThreadPoolExecutor for concurrent translation with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor: # Create a future for each text translation futures = [executor.submit(translate_text, text) for text in unique_texts] for future in as_completed(futures): # Collect the results as they complete text, translation = future.result() translations[text] = translation ``` Once all the text is translated, the final step is to replace the original text in the copied presentation. This is done by creating a batch of [`replaceAllText`](https://developers.google.com/workspace/slides/api/reference/rest/v1/presentations/request#replacealltextrequest) requests for the [Google Slides API](https://developers.google.com/workspace/slides/api/guides/overview). The requests are sorted from longest to shortest string to prevent issues where a shorter piece of text might be a substring of a longer one. ```python requests = [] sorted_translations = sorted(translations.items(), key=lambda item: len(item[0]), reverse=True) for text, translation in sorted_translations: if translation.strip(): requests.append({ 'replaceAllText': { 'replaceText': translation, 'pageObjectIds': list(page_ids), # The slides where the text appears 'containsText': { 'text': text, 'matchCase': True, } } }) # Execute the batch update body = {'requests': requests} response = slides_service.presentations().batchUpdate( presentationId=copied_presentation_id, body=body ).execute() ``` The result? A perfectly translated slide deck, ready to go. ![Screenshot of the Luxembourgish Version](/static/img/articles/slides-to-translate-when-it-says-no-build-a-0-04-solution-on-your-lunch-break/02.webp) Screenshot of the Luxembourgish Version ### See it in Action 🚀 Want to see the results for yourself? While I can't share corporate slides publicly, I can use some slides I wrote for an [MLOps Community Meetup](https://mlops.community/). Here are the original presentation and the versions translated by the script: - 📊 **Original Presentation**: [https://docs.google.com/presentation/d/1CvmBW0BmorHpC6NwYCh94qTAMZY9MlUx7ClcSwCg7Dg/edit?slide=id.p#slide=id.p](https://docs.google.com/presentation/d/1CvmBW0BmorHpC6NwYCh94qTAMZY9MlUx7ClcSwCg7Dg/edit?slide=id.p#slide=id.p) - 🇫🇷 **Translated to French**: [https://docs.google.com/presentation/d/1t7MqZG5BUCbTbDKlOu25e0BLi00IIIcFVWKZCCt8h2o/edit?slide=id.p#slide=id.p](https://docs.google.com/presentation/d/1t7MqZG5BUCbTbDKlOu25e0BLi00IIIcFVWKZCCt8h2o/edit?slide=id.p#slide=id.p) - 🇩🇪 **Translated to German**: [https://docs.google.com/presentation/d/1jYGc7bqus-Kluvmww5U8T3WgALDayh5cfsYgJPiA6ds/edit?slide=id.p#slide=id.p](https://docs.google.com/presentation/d/1jYGc7bqus-Kluvmww5U8T3WgALDayh5cfsYgJPiA6ds/edit?slide=id.p#slide=id.p) - 🇱🇺 **Translated to Luxembourgish**: [https://docs.google.com/presentation/d/1vcmMHp_vh1pMhNgLMlVyGC_wRJawtY9-\_g8T1df5Hfo/edit?slide=id.p#slide=id.p](https://docs.google.com/presentation/d/1vcmMHp_vh1pMhNgLMlVyGC_wRJawtY9-_g8T1df5Hfo/edit?slide=id.p#slide=id.p) ### The Jaw-Dropping Part: The Cost 💰 This is where the story goes from “cool project” to “game-changer.” **For my colleague’s 155-slide deck, the total cost of using** [**Gemini 2.5 Flash**](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash) **through Vertex AI was a minuscule \$0.04.** _Let that sink in. Four cents to solve a problem that was holding up a project and causing a major headache._ To put it in perspective, I ran the numbers on [my smaller 22-slide test presentation](https://docs.google.com/presentation/d/1CvmBW0BmorHpC6NwYCh94qTAMZY9MlUx7ClcSwCg7Dg/edit?slide=id.p#slide=id.p): - 📥 Total Input Tokens: 15,088 - 📤 Total Output Tokens: 1,687 - 💵 Input Cost: \$0.0045 - 💸 Output Cost: \$0.0042 - 🧾 **Total Cost**: \$0.0087 **That’s less than a single cent!** The cost is so low it’s practically a rounding error. ### Key Benefits of this Approach 🗝️ Beyond the speed and low cost, this DIY solution has several powerful advantages over external products: - 🔒 **Corporate Data Stays Secure**: Because the solution uses Vertex AI, which was already an approved platform within our organization, no sensitive corporate data ever leaves our GCP environment. This is a massive win for security and compliance. - 💲 **Highly Cost-Effective**: Compared to the subscription fees for many third-party translation services, this method is orders of magnitude cheaper. You only pay for what you use, which for this task is next to nothing. - 🔄 **Infinitely Reusable & Adaptable**: Now that the basic framework is in place (Colab + Vertex AI + GCP APIs), it can be adapted for [countless other use cases](https://github.com/fmind?tab=repositories). Automating reports, summarizing documents, analyzing feedback — the possibilities are endless. - 🎨 **Fully Customizable Prompts**: You have complete control over the prompt sent to the AI. This means you can guide the translation style, provide specific context about the content, and fine-tune the output in a way that black-box products simply don’t allow. ### Potential Areas for Improvement 🌱 While the solution worked, there’s always room for refinement. Here are a few ideas for taking it to the next level: - ⏩ **Optimize API Calls**: Instead of sending one translation request per text element, the code could be optimized to batch multiple requests into a single API call, which would further reduce latency and avoid repeating system instructions. - 🧠 **Provide More Context**: The model could be given even more context, such as the slide number or surrounding text, to produce even more accurate and contextually-aware translations. - 🖼️ **Translate Text in Images**: The current version only handles text boxes. A more advanced, experimental version could use a multimodal model like Gemini to identify and translate text embedded within images on the slides. ### The Real Lesson: Empowerment Through Technology 🎓 This little lunch-break project is a perfect example of the world we live in now. On one hand, we have the rigid structures of large organizations, which can be slow to adapt. On the other, we have generative AI tools that empower any individual with a bit of technical skill to build incredibly powerful and efficient solutions in minutes. We no longer have to wait for permission. With tools like Gemini and platforms like Vertex AI, we can solve our own problems, and our colleagues’ problems, with a speed and cost-efficiency that would have been unimaginable just a few years ago. So next time you’re stuck waiting on a process, maybe ask yourself: “Could I just build this myself?” You might be surprised by what you can accomplish before you’ve even finished your sandwich! _You can find the full project and notebook on GitHub here:_ [https://github.com/fmind/slides-to-translate/](https://github.com/fmind/slides-to-translate/) _._ ![Source: Gemini App](/static/img/articles/slides-to-translate-when-it-says-no-build-a-0-04-solution-on-your-lunch-break/03.webp) Source: Gemini App ### [It’s Not Artificial: Recreating a Conversational Format with Gemini’s Multi-Speaker Text to Speech](https://www.fmind.dev/articles/its-not-artificial-recreating-a-conversational-format-with-gemini-s-multi-speaker-text-to-speech/) For many who grew up in the French-speaking world, the television series “[C’est pas sorcier](https://www.youtube.com/@Cestpassorcierofficiel)” was a cultural touchstone. Its format, featuring a duo of hosts explaining complex scientific topics in an accessible and entertaining manner, was one of my favorite TV show in my childhood. This project was inspired by a desire to replicate that engaging, conversational dynamic using modern technology for my daughter. The goal was to leverage the latest advancements in generative AI — specifically the [multi-speaker text-to-speech](https://ai.google.dev/gemini-api/docs/speech-generation#multi-speaker) (TTS) capabilities of [Google’s Gemini models](https://ai.google.dev/gemini-api/docs) — to programmatically generate similar educational content. This project, titled “It’s Not Artificial,” demonstrates the power of this approach, and the results are highly promising! ![Photo by Pawel Czerwinski on Unsplash](/static/img/articles/its-not-artificial-recreating-a-conversational-format-with-gemini-s-multi-speaker-text-to-speech/cover.webp) Photo by [Pawel Czerwinski](https://unsplash.com/@pawel_czerwinski?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### The Concept: A Dynamic Duo Powered by AI 💡 The core concept is to generate a complete audio episode from a set of simple parameters. The process involves creating a script with a dialogue between two distinct personas: a curious inquirer and a subject matter expert, mirroring the dynamic of the original show. Below are two examples generated by the system, one in English and one in French, on the topic of Artificial Intelligence. - [**English Example — Theme AI**](https://drive.google.com/file/d/1pvKxShKip570-4Bpd1vXtB_G_eWAVB1m/view?usp=drive_link) - [**French Example — Theme AI**](https://drive.google.com/file/d/1q6tAHHrPtqJYnUPj_KHR7aHdD3BVnLv8/view?usp=drive_link) An examination of the underlying code reveals a two-part process. ### The Settings: Customizing the Experience ⚙️ One of the most powerful aspects of this project is its flexibility. Beyond [NotebookLM](https://notebooklm.google.com/), the underlying notebook is designed to be highly customizable, allowing anyone to tailor the generated content. Here are the key settings you can adjust: - **`AGE`**: Defines the target age for the content, which influences the complexity of the vocabulary and explanations. - **`LANG`**: Sets the language for the transcript and audio (e.g., "English" or "French", …). Gemini models support [24 languages automatically](https://ai.google.dev/gemini-api/docs/speech-generation#languages). - **`THEME`**: Determines the topic of the conversation. You could set this to anything from "Black Holes" to "The History of Pizza." - **`MINUTES`**: Specifies the desired length of the audio episode. - **`SPEAKER_1_NAME`** **&** **`SPEAKER_2_NAME`**: Customizes the names of the two hosts in the script. - **`SPEAKER_1_VOICE`** **&** **`SPEAKER_2_VOICE`**: Selects from a wide range of [30 pre-built voices](https://ai.google.dev/gemini-api/docs/speech-generation#voices) for each speaker, allowing for unique vocal combinations. This level of control makes it possible to generate a virtually unlimited variety of educational audio content. ### The Code: A Two-Step Symphony 🎼 The process is orchestrated within a Python script utilizing the [Google GenAI SDK](https://github.com/googleapis/python-genai). ### Part 1: Generating the Transcript The initial step involves generating an audio script. Rather than manual composition, a Gemini model is prompted to create a transcript. The effectiveness of this step hinges on precise prompt engineering, which defines the speaker roles, tone, topic, and target audience. ```python TRANSCRIPT_PROMPT = f""" Generate a {MINUTES} minutes transcript in {LANG} about {THEME} for a {AGE}-year-old. The speaker names are {SPEAKER_1_NAME} and {SPEAKER_2_NAME}. - {SPEAKER_1_NAME} has a curious mind and ask questions. - {SPEAKER_2_NAME} is an expert and answer questions. Follow strictly the format below for the transcript (e.g., no extra sounds, no markdown, ...): {SPEAKER_1_NAME}: So... what's on the agenda today? {SPEAKER_2_NAME}: You're never going to guess! {SPEAKER_1_NAME}: Black holes? {SPEAKER_2_NAME}: Yes! """ transcript = client.models.generate_content( model=TRANSCRIPT_MODEL, contents=TRANSCRIPT_PROMPT, # ... configuration ... ).text ``` By specifying the roles of the speakers, the model generates a natural-sounding conversation that flows logically. ```text Fred: So... what's on the agenda today? Jamy: Today, we're talking about something super smart! Fred: Ooh, like an owl? Or a dolphin? Jamy: Even smarter, in a way. We're talking about Artificial Intelligence. Fred: Arty-fish-all... what now? Jamy: Artificial Intelligence. Let's call it AI for short. It's like giving a computer or a robot a special brain so it can learn and think. Fred: A robot brain? Cool! So it can think just like me? Jamy: Almost! It can think and solve problems, but in a different way. Imagine you have a toy robot. AI is like the magic that makes the robot smart enough to play a game with you. Fred: It can play games? Like checkers? Jamy: Exactly! Or it can play chess or even video games. Some AI are so good they can beat the best players in the world. Fred: Wow! What else can this AI brain do? Jamy: Lots of things! Do you ever talk to a grown-up's phone or a smart speaker and ask it to play a song or tell you a joke? ... ``` ### Part 2: Bringing the Script to Life with Multiple Voices This is the core of the implementation. The latest Gemini models can [generate audio with multiple, distinct speakers](https://ai.google.dev/gemini-api/docs/speech-generation#multi-speaker) from a single API call. The configuration involves defining the speakers and assigning a pre-built voice to each. ```python response = client.models.generate_content( model=TEXT_TO_SPEECH_MODEL, contents=f"Read this in {LANG} with a style interesting for a {AGE}-year-old:\n\n{transcript}", config=types.GenerateContentConfig( response_modalities=["AUDIO"], speech_config=types.SpeechConfig( multi_speaker_voice_config=types.MultiSpeakerVoiceConfig( speaker_voice_configs=[ types.SpeakerVoiceConfig( speaker=SPEAKER_1_NAME, voice_config=types.VoiceConfig( prebuilt_voice_config=types.PrebuiltVoiceConfig( voice_name=SPEAKER_1_VOICE, ) ), ), types.SpeakerVoiceConfig( speaker=SPEAKER_2_NAME, voice_config=types.VoiceConfig( prebuilt_voice_config=types.PrebuiltVoiceConfig( voice_name=SPEAKER_2_VOICE, ) ), ), ] ), ) ) ) ``` The model parses the transcript, identifies the speaker tags (e.g., `Fred:` or `Jamy:)`, and applies the designated voice to the corresponding lines of dialogue. This produces a seamless, conversational audio file without the need for manual audio editing or splicing. ### The Impact: Beyond Nostalgia 📈 While this project was inspired by a classic television show, its implications extend far beyond nostalgia. **In Education:** Consider a paradigm where educational content is highly personalized. A student could select a topic, language, and the personas of their AI tutors. History could be learned through a simulated conversation with historical figures. This level of customization has the potential to make learning more accessible and engaging for a diverse range of students. **In Professional Environments:** The applications are equally compelling. Corporate training, for instance, could be transformed from static presentations into interactive, conversational modules. Employee onboarding could feature simulated dialogues with key personnel, and complex technical documentation could be elucidated through an expert-novice conversational format. ### The Future is Conversational 💬 This experiment highlights a shift towards more natural and intuitive human-computer interaction. The ability to dynamically generate multi-speaker audio opens a new frontier for automated content creation. The success of formats like “C’est pas sorcier” demonstrates the educational power of conversation. With tools like Gemini, this principle can be programmatically integrated into our digital experiences, fostering a new generation of dynamic and engaging learning tools. In the future, the project could also generate images and videos to support the episode. - **Link to the Github Repository**: [https://github.com/fmind/it-is-not-artificial/blob/main/It_is_Not_Artificial-C'est_Pas_Artificel.ipynb](https://github.com/fmind/it-is-not-artificial/blob/main/It_is_Not_Artificial-C%27est_Pas_Artificel.ipynb) ![Photo by Sander Sammy on Unsplash](/static/img/articles/its-not-artificial-recreating-a-conversational-format-with-gemini-s-multi-speaker-text-to-speech/02.webp) Photo by [Sander Sammy](https://unsplash.com/@sammywilliams?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [The Agentic Cloud: Forging the Next Era of Infrastructure](https://www.fmind.dev/articles/the-agentic-cloud-forging-the-next-era-of-infrastructure/) A new architectural pattern is going to emerge that could shift the balance of power from the cloud giants to intelligent agents, commoditizing the very services that built their empires. Today, a handful of “hyperscalers” — AWS, Azure, and GCP — hold an overwhelming majority of the market share, a dominance built on [a decade of unprecedented capital investment and engineering might](https://spacelift.io/blog/cloud-computing-statistics). They’ve constructed an [unconquerable moat](https://semianalysis.com/2023/05/04/google-we-have-no-moat-and-neither/) that has made the idea of a new competitor emerging seem not just difficult, but fundamentally impossible. ![Market Share of Cloud Providers ( Source )](/static/img/articles/the-agentic-cloud-forging-the-next-era-of-infrastructure/cover.webp) Market Share of Cloud Providers ([Source](https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/)) However, a paradigm shift is here, driven by Generative AI. This isn’t just another service to be added to a catalog of over 200 icons. It represents a new way of building, deploying, and managing software. I believe this new architectural pattern, which I call the **“Agentic Cloud,”** has the potential to dismantle the primary value proposition of the hyperscalers, creating the first genuine opportunity in a decade for new players to disrupt the market. ![Photo by Paul Melki on Unsplash](/static/img/articles/the-agentic-cloud-forging-the-next-era-of-infrastructure/02.webp) Photo by [Paul Melki](https://unsplash.com/@paulmelki?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### The Fortress 🏰: What Makes the Cloud Giants Dominate? Let’s be clear: the dominance of hyperscalers isn’t just about owning warehouses full of servers. It’s about [the incredibly sophisticated software layer](https://cloud.google.com/products?hl=en) built on top of that hardware, which creates a deep and sticky moat: - **Curated Ecosystems:** Their true value is packaging thousands of open-source and proprietary tools — from [Kubernetes](https://kubernetes.io/) to [Kafka](https://kafka.apache.org/) to specialized databases — into a single, cohesive platform where things _just work_ together. This saves enterprises untold engineering years. - **Abstraction and Usability:** They provide powerful [consoles](https://console.cloud.google.com/) and [CLIs](https://cloud.google.com/sdk/gcloud) that abstract away staggering complexity. A team can provision [a globally distributed, fault-tolerant database](https://cloud.google.com/bigtable?hl=en) with a few clicks (and a credit card), a task that would have required a specialized team just a few years ago. - **The Marketplace:** They are the ultimate distribution channel for third-party software companies like [MongoDB](https://www.mongodb.com/) and [Datadog](https://www.datadoghq.com/), creating a one-stop-shop that’s incredibly convenient and hard to leave. - **Economies of Scale:** They operate at a scale that gives them [unparalleled efficiency](https://cloud.google.com/terms/sla?hl=en) in everything from negotiating hardware prices to managing energy consumption, which they can pass on as lower costs. - **Continuous Innovation:** They are constantly pushing the boundaries with new services, from pioneering serverless with [AWS Lambda](https://aws.amazon.com/lambda/) to building globally consistent databases like [GCP Spanner](https://cloud.google.com/spanner?hl=en), all while managing the maintenance burden for their customers. ![Source: Médéric HURIER (Fmind)](/static/img/articles/the-agentic-cloud-forging-the-next-era-of-infrastructure/03.webp) Source: Médéric HURIER (Fmind) A smaller provider can lease the same data centers and buy the same servers, but they can’t compete with the tens of thousands of engineers and billions in R&D required to build and maintain this critical software layer. ### The Cracks in the Armor 🛡️: Limitations of the Catalog Model This “bigger is better” model, for all its power, has inherent weaknesses that an agent-based approach is perfectly positioned to exploit. - **Massive Code Liability:** Over time, [all code is a liability](https://www.leemeichin.com/posts/code-is-a-liability). Every new service added to the catalog is more code to maintain, patch, secure, and update. This requires a colossal, expensive, and often slow-moving engineering organization. - **The Paradox of Choice:** A catalog with [over 200 services](https://cloud.google.com/products?hl=en) is overwhelming. Developers spend precious time navigating this complexity, learning provider-specific quirks, and trying to decide between five different ways to run a simple container. - **Inflexible Architectures:** These managed services are optimized for 80% use cases. The moment your needs deviate from their “golden path,” you hit a wall of inflexibility, forcing you into awkward workarounds or abandoning the managed service entirely. - **It’s Still Just a Toolbox:** At the end of the day, the cloud is a collection of powerful tools. It still requires skilled (and expensive) human architects and engineers to select the right tools and write all the “glue code” (Terraform, CloudFormation, etc.) to assemble them into a working application. ### The Disruptor 🤖: Rise of the Intent-Driven Agentic Cloud Instead of competing on the _size of the catalog_, the new model competes on the _intelligence of the agent_. I define an **Agentic Cloud** as an AI system that translates high-level, intent-driven specifications into fully provisioned, configured, deployed, and autonomously managed infrastructure and applications. ![Source: Médéric HURIER (Fmind)](/static/img/articles/the-agentic-cloud-forging-the-next-era-of-infrastructure/04.webp) Source: Médéric HURIER (Fmind) This flips the script on how we deliver infrastructure: 1. **Specification (The “Architect” Agent):** The user specifies the _what_, not the _how_. This could be through natural language (“I need a scalable e-commerce backend with a product catalog, user auth, and a recommendation engine”), a diagram, or even a photo of a whiteboard sketch. The agent asks clarifying questions to generate a detailed architectural plan. 2. **Provisioning (The “Engineer” Agent):** The agent autonomously writes the necessary Infrastructure as Code (IaC) using best-of-breed open-source tools like [Terraform](https://developer.hashicorp.com/terraform) or [Pulumi](https://www.pulumi.com/). It configures networking, security groups, databases, and compute resources on any underlying IaaS provider, from AWS to a local bare-metal host. 3. **Deployment (The “DevOps” Agent):** The agent deploys the application code (which could also be agent-generated), runs integration tests, and performs security scans, ultimately providing the user with a ready-to-use endpoint. 4. **Autonomous Maintenance** (The “SRE” Agent): The agent continuously monitors the live system, autonomously performing tasks that are currently manual or require complex tooling: ### Is This Really Happening? 🔭 This isn’t science fiction. Some foundational pieces are already falling into place. - **Spec-to-Code (Kiro):** Projects like [Kiro](https://kiro.dev/) are already showing how high-level specifications and designs can be translated directly into functional application code. An Agentic Cloud applies this same principle to the infrastructure layer. - **Automated Design (Google Cloud ADC):** Google’s own [Application Design Center](https://cloud.google.com/application-design-center/docs/overview) is a clear bridge technology. It validates the demand for designing infrastructure with visual and natural language inputs. While it currently just assembles pre-defined GCP components, it’s a clear precursor to a dynamic, cross-cloud agent. - **Algorithmic Innovation (** [**AlphaEvolve**](https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/) **):** DeepMind’s work demonstrates that agents can go beyond just assembling known patterns; they can _innovate_ and discover novel, more efficient algorithms. An advanced agentic cloud could design entirely new network topologies or data storage strategies on the fly — a capability no static catalog can offer. ### Shifting the Value Stack 💸: The New Business Model The Agentic Cloud model could trigger a seismic shift in the tech value chain. The primary value proposition is no longer the cloud provider’s curated software catalog. It shifts to two places: 1. The raw **utility infrastructure** (compute, network, storage) at the bottom. 2. The **intelligence of the Agentic Cloud** itself at the top. The consequences are profound: - **Commoditization of Cloud Providers:** Hyperscalers are forced to compete on the metrics of a simple utility company: the price, reliability, and energy efficiency of their raw hardware. Their multi-billion-dollar investment in their proprietary software platform would be massively devalued. - **A New Hope for Open Source:** Currently, many open-source companies (like Redhat or Postgres) monetize by selling a “managed service” on cloud marketplaces. An Agentic Cloud threatens this by being able to deploy and manage the open-source version itself. A new, more direct model emerges where the agent pays a micro-transaction to a trusted registry for a verified, secure package or subscribes to an API for premium features. - **The Business to Build:** The next multi-trillion-dollar opportunity isn’t building another AWS. It’s building the definitive Agentic Cloud that sits on top of all of them. ### The Herculean Task Ahead 💪: Major Challenges Let’s be realistic. Building a true Agentic Cloud is a monumental engineering challenge. - **Reliability & Determinism:** How do you stop an agent from “hallucinating” a critically flawed or insecure architecture? Ensuring robust, predictable, and idempotent results is paramount. - **Security:** An agent with programmatic API keys to create and modify your core infrastructure is the ultimate security risk. Securing the agent, its credentials, and the code it generates is a massive undertaking — a problem space where your expertise in computer security is especially relevant. - **Complex State Management:** Real-world infrastructure is stateful. The agent must be able to understand the current state, plan complex migrations, and safely handle failures without needing a human to intervene. - **The Long Tail of Integration:** Handling the mainstream use cases is one thing. The real world is messy, full of legacy systems, obscure third-party APIs, and weird corporate requirements that will challenge agent capabilities for years. ### Conclusion: Democratizing Elite Infrastructure 🌍 The current cloud paradigm is one of centralization, concentrating capital and talent in the hands of a few. The Agentic Cloud represents a powerful force for decentralization and democratization. It marks the shift from **catalog-driven** infrastructure to **intelligence-driven** infrastructure. This technology has the power to level the playing field, enabling a small team anywhere in the world to command the same infrastructural power as a top-tier Silicon Valley company. It’s an opportunity to break the dependency on a few key players and geographies, fostering a new wave of global innovation. The race to assault the fortress is on! ![The Agentic Cloud: Forging the Next Era of Infrastructure](/static/img/articles/the-agentic-cloud-forging-the-next-era-of-infrastructure/05.webp) ### [Scaling the Summit: Challenges for Serving LLMs at Scale on GCP](https://www.fmind.dev/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/) Deploying a [Large Language Model](https://en.wikipedia.org/wiki/Large_language_model) (LLM) into a production environment can feel like transitioning from a peaceful hike to a treacherous mountain expedition. Serving LLM models at scale presents its own formidable set of challenges. It’s not just about getting a model to respond; it’s about doing so reliably, quickly, and cost-effectively under heavy user load. ![Photo by ecmadao . on Unsplash](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/cover.webp) Photo by [ecmadao .](https://unsplash.com/@ecmadao?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) On [Google Cloud Platform](https://cloud.google.com/) (GCP), you have a spectrum of solutions at your disposal, each with its own trade-offs between simplicity and control. This article dives into the practical realities of serving LLMs on GCP, reviewing the out-of-the-box options and sharing insights from a hands-on benchmark. We’ll explore why deploying even a seemingly straightforward model like [Gemma 3 12B](https://ai.google.dev/gemma/docs/core) can get tricky and help you choose the right path for your specific needs, before you even start thinking about complex optimizations. ### 🧗 The Tricky Terrain: Model and Hardware Before we dive into the platforms, let’s talk about a foundational challenge: matching your model to your hardware. For our benchmarks, I chose to serve [**Gemma 3 12B**](https://ai.google.dev/gemma/docs/core), a powerful open model from Google. I aimed to use the [**Nvidia L4 GPU**](https://www.nvidia.com/en-us/data-center/l4/), a cost-effective and widely available accelerator on GCP. Here’s the catch: **a 12-billion parameter model in its standard** **`bfloat16`** **precision requires more than 24GB of GPU memory**. The L4 GPU has exactly 24GB. This creates a tight squeeze, often leading to out-of-memory errors. To make it work, we have two primary options: 1. **Use Multiple GPUs**: Distribute the model across two or more L4 GPUs. This works but doubles your baseline hardware cost and adds configuration complexity. 2. **Use** [**Quantization**](https://ai.google.dev/gemma/docs/core#sizes): Employ techniques like 4-bit quantization to shrink the model’s memory footprint. This allows it to fit on a single L4 but can introduce a slight degradation in accuracy. An alternative path is to simplify the pipeline by using a smaller model, like [**Gemma 3 4B**](https://ai.google.dev/gemma/docs/core). It fits comfortably on a single L4 GPU, reducing complexity and cost, but at the expense of some of the model’s reasoning capability. **This trade-off between model power, hardware cost, and operational complexity is a central theme in serving LLMs.** ![Photo by Piret Ilver on Unsplash](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/02.webp) Photo by [Piret Ilver](https://unsplash.com/@saltsup?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### ⚙️ The Benchmark Setup To test these solutions under pressure, I used [**Locust**](https://locust.io/), a powerful open-source load testing tool. The goal was to simulate a real-world scenario where the number of concurrent users ramps up over time, allowing us to observe how each setup behaves under increasing strain. The benchmark was configured to run for 5 minutes for each solution, ramping up from 0 to 250 concurrent users at a rate of 1 new user per second. Each virtual user continuously sends prompts from the [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) dataset. The prompts are sent with a `temperature` of `0.0` for deterministic outputs and a `max_output_tokens` limit of 1000. This setup allows us to measure key performance indicators like failure rate, response time, and throughput as the load intensifies. ```python # Example of benchmark in locustfile.py class VertexAIMaaS(Benchmark, lc.HttpUser): """Locust user for benchmarking Vertex AI Model as a Service.""" host: str = ( f"https://{LOCATION}-aiplatform.googleapis.com" if LOCATION != "global" else "https://aiplatform.googleapis.com" ) @lc.task def predict(self): """Send a prediction request to the Vertex AI Model as a Service.""" config = { "temperature": self.options.temperature, "thinkingConfig": {"thinkingBudget": 0}, "maxOutputTokens": self.options.max_output_tokens, } headers = { "Content-Type": "application/json", "Authorization": f"Bearer {self.bearer}", } url = f"/v1/projects/{self.options.project_id}/locations/{self.options.location}/publishers/google/models/{self.options.model}:generateContent" for text in self.iter_data(): contents = [{"role": "user", "parts": [{"text": text}]}] body = {"contents": contents, "generationConfig": config} self.client.post(url=url, json=body, headers=headers) ``` ### 🗺️ GCP’s LLM Serving Options: A Review We evaluated four distinct approaches for serving LLMs on GCP, intentionally excluding the highly manual process of setting up from scratch on a [Compute Engine instance](https://cloud.google.com/products/compute?hl=en). ### 1. Vertex AI MaaS (Model-as-a-Service) 🏞️ [Vertex AI MaaS](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-partner-models) is Google’s fully-managed, serverless offering for its first-party models like [Gemini](https://cloud.google.com/vertex-ai/generative-ai/docs/models) and third-party models like [Claude](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude). It’s the easy solution for getting started. You interact with it via a simple API call, and Google handles all the underlying infrastructure for you. ![User Interface of Vertex AI Studio (MaaS)](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/03.webp) User Interface of Vertex AI Studio (MaaS) - **The Good**: It’s incredibly convenient. There’s no infrastructure to manage, it integrates seamlessly with the GCP ecosystem, and you pay per use for each input/output tokens. For building applications like RAG systems or agents where you don’t have stringent scalability requirements, this is a fantastic starting point! - **The Challenge**: This simplicity comes at the cost of control. As our benchmarks show, the [default pay-as-you-go model](https://cloud.google.com/vertex-ai/generative-ai/pricing) has [strict rate limits](https://cloud.google.com/vertex-ai/generative-ai/docs/dynamic-shared-quota). When under load, you’ll quickly run into `429 - RESOURCE_EXHAUSTED` errors. The primary lever for scaling is either [increasing your account quotas](https://cloud.google.com/vertex-ai/generative-ai/docs/dynamic-shared-quota), or [purchasing provisioned throughput](https://cloud.google.com/vertex-ai/generative-ai/docs/provisioned-throughput/error-code-429) which reserves capacity for your model but comes at a significant cost. You also have limited knobs to tune performance (e.g., disable the "thinking" mode, which can reduce latency for certain use cases). ![Code exported on the Vertex AI Studio Web Console](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/04.webp) Code exported on the Vertex AI Studio Web Console ### 2. Vertex AI Endpoint for Open Models 🏕️ [**Vertex AI Online Inference**](https://cloud.google.com/vertex-ai/docs/predictions/overview) allows you to deploy open-source models like Gemma using pre-built, optimized containers. You get a managed solution but with more control over the underlying hardware and scaling behavior than MaaS. - **The Good**: It’s a great trade-off between power and simplicity. You can [select your machine type and GPU](https://cloud.google.com/vertex-ai/docs/predictions/configure-compute) (e.g., an `g2-standard-24` with 2 x L4 GPU), [configure scaling parameters](https://cloud.google.com/vertex-ai/docs/predictions/choose-endpoint-type), and deploy. It abstracts away much of the complexity of managing infrastructure, making it ideal for teams that want to expose an LLM without a dedicated MLOps or Kubernetes expert. ![Configuration for deploying Gemma 3 12B from Vertex AI Online Inference](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/05.webp) Configuration for deploying Gemma 3 12B from Vertex AI Online Inference - **The Challenge**: While managed, it’s not free. [There’s an additional price tag for the endpoint itself on top of the compute resources](https://cloud.google.com/vertex-ai/pricing#prediction-prices). As you can see when configuring the endpoint, you need to define machine types and accelerators. [Monitoring latency and resource utilization](https://cloud.google.com/vertex-ai/docs/predictions/view-endpoint-metrics) is crucial to finding the right balance. You must carefully configure the auto-scaling settings to handle load without over-provisioning. ![Container and model deployed on Vertex AI Online Inference](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/06.webp) Container and model deployed on Vertex AI Online Inference ### 3. Cloud Run with GPU + Ollama 🚗 [**Cloud Run**](https://cloud.google.com/run?hl=en) offers a serverless approach for running custom containers, now with [GPU support](https://cloud.google.com/run/docs/configuring/services/gpu). By default, [GCP proposes to deploy Gemma 3](https://cloud.google.com/run/docs/tutorials/gpu-gemma-with-ollama) using a container running the popular [Ollama](https://ollama.com/) serving framework. ![Scaling the Summit: Challenges for Serving LLMs at Scale on GCP](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/07.webp) ![Monitoring dashboard on Cloud Run](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/08.webp) Monitoring dashboard on Cloud Run - **The Good**: This combines serverless simplicity with GPU power. It’s excellent for applications with variable traffic, as it can scale down to zero, [saving costs](https://cloud.google.com/run/pricing?hl=en). It’s a fantastic way to quickly build proofs-of-concept and applications with an embedded LLM without committing to a full-blown Kubernetes cluster. - **The Challenge**: Cloud Run’s primary limitation for large-scale serving is that [you can only attach one GPU per instance](https://cloud.google.com/run/docs/configuring/services/gpu#gpu-type). While you can scale out by adding more instances, this [concurrency-based scaling](https://cloud.google.com/run/docs/configuring/services/gpu-best-practices#autoscaling-and-gpu) can be less efficient for LLMs than having a single, powerful instance with multiple GPUs that can handle larger batches. As our results show, this approach hit its limits and produced a high number of failures under heavy load.   ```justfile # setup the cloud run ollama model serving setup-cloud-run-ollama model="gemma3-12b": gcloud run deploy {{model}}-ollama \ --cpu=8 --max-instances=2 --memory=32Gi \ --gpu=1 --gpu-type=nvidia-l4 --no-gpu-zonal-redundancy \ --image=us-docker.pkg.dev/cloudrun/container/gemma/{{model}} \ --timeout=600 --concurrency=8 --ingress=all \ --allow-unauthenticated --no-cpu-throttling \ --project=$PROJECT_ID --region=$LOCATION \ --set-env-vars OLLAMA_NUM_PARALLEL=4 \ --set-env-vars API_KEY=$API_KEY # proxy the cloud run ollama model serving proxy-cloud-run-ollama model="gemma3-12b": gcloud run services proxy {{model}}-ollama --port=8080 --region=$LOCATION --project=$PROJECT_ID ``` ### 4. GKE + vLLM (The Unclimbed Peak) 🏔️ [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine?hl=en) (GKE) is the most powerful and flexible solution. By deploying a model on GKE with a high-performance serving framework like [vLLM](https://github.com/vllm-project/vllm), you can achieve maximum performance and have fine-grained control over every aspect of the deployment. ![Configuration for deploying Gemma 3 12B on GKE](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/09.webp) Configuration for deploying Gemma 3 12B on GKE - **The Good**: Unmatched power, control, and potential for throughput. For demanding, large-scale production workloads, this is the ultimate architecture. Google has made strides in simplifying this process with a [dedicated section on AI/ML workloads](https://cloud.google.com/kubernetes-engine/docs/integrations/ai-infra), like this tutorials on [serving Gemma with vLLM on GKE](https://cloud.google.com/kubernetes-engine/docs/tutorials/serve-gemma-gpu-vllm). - **The Challenge**: Complexity. This path is not for the faint of heart. In this end, it requires [significant Kubernetes expertise](https://cloud.google.com/kubernetes-engine/docs/concepts/machine-learning/inference) to configure networking, node pools, autoscaling, and the serving framework itself. In my case, this peak remained unclimbed. Due to project quotas limiting me to a single L4 GPU, I couldn’t provision the necessary multi-GPU GKE node required to serve the Gemma 3 12B model effectively, highlighting a very real-world constraint for small teams and companies. ![No GPU for freelance developers :(](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/10.webp) No GPU for freelance developers :( ### 📊 Analyzing the Results The benchmark data paints a clear picture of the trade-offs between the different serving solutions. While some approaches appear to handle a high volume of requests, a closer look at the failure rates and response times reveals the true story of their performance under pressure. ![Scaling the Summit: Challenges for Serving LLMs at Scale on GCP](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/11.webp) ![Scaling the Summit: Challenges for Serving LLMs at Scale on GCP](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/12.webp) - **Overall Performance**: A summary of the test runs shows a stark contrast in reliability. While the **Vertex AI MaaS** and **Cloud Run Ollama** services processed a very high number of total requests (over 100,000 each), the vast majority of these were failures. For instance, `VertexAIMaaS - gemini-2.5-flash-lite-preview-06-17` had 163,993 requests but 160,879 failures. In contrast, **Vertex AI Endpoint** handled a smaller volume of total requests (1,213) but with a significantly lower failure count (311). - **Failure Rate and Types**: The reasons for failure are critical. The MaaS models consistently failed with `HTTPError: 429 Client Error: Too Many Requests`, indicating they were quickly overwhelmed by the load and hit their rate limits. CloudRun Ollama failed with a mix of server errors `500`, `503` and `429`s, suggesting the instances were completely saturated. The "Failure Rate Over Time" chart dramatically illustrates this. The MaaS and CloudRun solutions see their failure rates shoot to nearly 100% almost immediately as user load increases. - `VertexAIEndpoint`, however, shows a much more gradual and controlled increase in its failure rate, never reaching the catastrophic levels of the other services. ![Scaling the Summit: Challenges for Serving LLMs at Scale on GCP](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/13.webp) - **Throughput vs. Success**: The “Throughput vs. User Count” graph shows that several services achieved high peak “Requests/s”. However, this metric is misleading on its own. High throughput is meaningless if the requests are not successful. The “User Count vs Failures/s” plot confirms that this high throughput was accompanied by an equally high rate of failures for most services. - `VertexAIEndpoint` maintained a very low and stable failure rate throughout the test, prioritizing successful responses over raw, unsuccessful request volume. ![Scaling the Summit: Challenges for Serving LLMs at Scale on GCP](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/14.webp) - **Response Time**: The “Response Time vs. User Count” chart provides the final piece of the puzzle. - The average response time for `VertexAIEndpoint` increases steadily with the user count, which is expected behavior for a system handling progressively more load. For the other services, the response time often appears low, but this is deceptive; they are not processing requests successfully but rather "failing fast" with quick error messages like a 429 response. - `CloudRunOllama` shows an initial spike in response time before it gets overwhelmed and starts failing consistently. In summary, the **Vertex AI Endpoint** demonstrated far superior resilience and stability under load. While it processed fewer total requests, it successfully served a much higher percentage of them, proving to be the most robust solution for scaling in this benchmark. The other solutions, while easy to set up, were not able to handle the escalating user load without hitting strict rate limits or becoming overwhelmed. Still, the amount of requests handled by Vertex AI Endpoint is not high enough for scaling the LLM in this current setup. While we couldn’t test GKE in this benchmark due to GPU quotas, it should provide the most scalable and controllable solution. ![Table with the results of the test with Locust](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/15.webp) Table with the results of the test with Locust ### 🧗‍♀️ Conclusion: Choosing Your Route Serving LLMs at scale on GCP is a journey with multiple paths, none of which is a simple walk in the park. The “best” approach is not one-size-fits-all but depends entirely on your team’s expertise, budget, and application requirements. - **For rapid prototyping and internal tools** with non-critical latency needs, **Vertex AI MaaS** is an excellent, hassle-free starting point. Just be prepared to manage rate limits or pay for provisioned throughput as you scale. - **For most production use cases without a dedicated Kubernetes team**, **Vertex AI Endpoints** for open models hits the sweet spot. It provides a robust, scalable, and controllable environment without the steep learning curve of GKE. It proved to be the most resilient solution in our benchmark. - **For quick tests or applications with spiky traffic**, **Cloud Run with GPU** is a cost-effective and viable option. However, be aware of its single-GPU-per-instance limitation, which can become a bottleneck for high-throughput scenarios. - **For ultimate performance and control at a large scale**, **GKE with vLLM** is the summit. It offers the best performance but requires the most significant investment in infrastructure expertise. Deploying LLMs is a complex but navigable challenge. By understanding the landscape, benchmarking your options, and choosing a path that aligns with your resources and goals, you can successfully scale the summit and deliver powerful AI experiences to your users. - **Gemma Model**: [https://ai.google.dev/gemma/docs/core#sizes](https://ai.google.dev/gemma/docs/core#sizes) - **Link to Benchmark on Github**: [https://github.com/fmind/gcp-llm-serving-benchmarks](https://github.com/fmind/gcp-llm-serving-benchmarks) ![Photo by Connor Moynihan on Unsplash](/static/img/articles/scaling-the-summit-challenges-for-serving-llms-at-scale-on-gcp/16.webp) Photo by [Connor Moynihan](https://unsplash.com/@connor_moynihan?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [The Great Data Divergence: Why Generative AI Demands a New Approach Beyond the Data Lake](https://www.fmind.dev/articles/the-great-data-divergence-why-generative-ai-demands-a-new-approach-beyond-the-data-lake/) In the last decade, the [data lake](https://en.wikipedia.org/wiki/Data_lake) has been the undisputed heart of the enterprise data ecosystem. As an AI architect, I’ve seen firsthand how centralizing our data powered intelligent systems. It was the promised land: a single, scalable repository for all our structured and unstructured data, a single source of truth that broke down data silos. This consolidation was a massive win for business intelligence, data engineering, and, crucially, for machine learning. We built sophisticated ETL/ELT pipelines, meticulously organizing raw data into refined, queryable assets using frameworks like the [Medallion Architecture](https://www.databricks.com/glossary/medallion-architecture) (Bronze, Silver, and Gold layers). Data scientists can pull from this curated Gold layer to train forecasting models, recommendation engines, and classification systems, far from the critical path of our operational databases. The paradigm was clear: ingest, consolidate, train, and then figure out how to deploy the resulting artifact back into production. This architecture served us well. **In this article, I will explore why this established paradigm is being fundamentally challenged by the rise of Generative AI**. I’ll break down the new data requirements driven by technologies like [Retrieval-Augmented Generation](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) (RAG) and explain the core dilemma this creates for enterprise data strategy. Finally, I’ll argue that the path forward isn’t about building bigger data lakes, but about embracing a new, API-first paradigm for data access. ![Photo by Aaron Burden on Unsplash](/static/img/articles/the-great-data-divergence-why-generative-ai-demands-a-new-approach-beyond-the-data-lake/cover.webp) Photo by [Aaron Burden](https://unsplash.com/@aaronburden?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### The New Rules of the Game: Freshness, Context, and Low Latency Before Generative AI, machine learning models primarily learned from the past to predict the future. Now, we are building intelligent agents and features designed to understand and act in the _present_. Think about the new wave of “smart features” that businesses are rushing to build. We’re not just training a model on last year’s sales data. We’re building agents that can: - Summarize the latest project updates from a dozen **Jira tickets**. - Answer a specific question about a contract stored in **Google Drive**. - Generate a customer outreach email based on the most recent interactions logged in **Salesforce**. These applications don’t need a massive, historical dataset for training; they need immediate, contextual, and often unstructured data to perform their task _right now_. The data lake, with its batch-oriented ingestion and processing cycles, suddenly feels like an encyclopedia in an age that demands a real-time news feed. ### The Central Dilemma: To Lake, or Not to Lake? This new reality forces a critical question upon every data leader and architect: **Should we move all this real-time, unstructured, operational data into the data lake?** ![https://www.databricks.com/glossary/medallion-architecture](/static/img/articles/the-great-data-divergence-why-generative-ai-demands-a-new-approach-beyond-the-data-lake/02.webp) [https://www.databricks.com/glossary/medallion-architecture](https://www.databricks.com/glossary/medallion-architecture) On one hand, the traditionalist view is compelling. Moving everything to the lake promises to maintain our “single source of truth.” It allows us to apply the same governance, security, and management frameworks we’ve spent years perfecting. It seems like the safe, logical extension of our current strategy. But from my experience on the ground, this approach is fundamentally flawed for the Gen AI era. The core problem is **latency**. The data lake architecture, for all its benefits in ensuring data quality for analytics, is inherently slow. The journey from raw data (Bronze) to a clean, usable state (Gold) involves ingestion, validation, and transformation steps that can take hours or even days. A RAG application can’t wait for a Jira ticket to complete its multi-stage journey through the data lake before it can be used as context. The information would be stale on arrival. Furthermore, it introduces massive inefficiency. Why should we copy data from a perfectly good, high-performance operational system like Salesforce — which is already the system of record — only to store a second, slightly delayed copy in the data lake, just so another application can query it? ### A Paradigm Shift: From Data Warehousing to API Management I believe the solution isn’t to force new workloads into old architectures. The paradigm must shift from **data centralization** to **decentralized data access through a centralized management plane**. The future is not about moving the data; it’s about creating standardized, high-performance pathways to access the data where it lives. The new center of the Gen AI data universe will not be a storage system, but an **API Management layer**, like [Google Apigee](https://cloud.google.com/apigee). ![https://cloud.google.com/architecture/best-practices-securing-applications-and-apis-using-apigee](/static/img/articles/the-great-data-divergence-why-generative-ai-demands-a-new-approach-beyond-the-data-lake/03.webp) [https://cloud.google.com/architecture/best-practices-securing-applications-and-apis-using-apigee](https://cloud.google.com/architecture/best-practices-securing-applications-and-apis-using-apigee) In this model: 1. **Operational Systems Remain the Source of Truth:** Salesforce, Jira, and Google Drive hold the live, operational data. They are masters of their domain. 2. **Data is Exposed via APIs:** These systems expose their data through secure, well-defined, and managed APIs. These APIs become the primary way for other systems to interact with the data. 3. **Gen AI Agents are API Consumers:** Our intelligent agents and RAG systems query these APIs directly (or via an orchestration layer) to get the fresh, real-time context they need. 4. **The Data Lake Retains a Key Role:** The data lake doesn’t disappear. It remains the powerhouse for analytics, BI, and training models that rely on large-scale historical data. It becomes just another powerful data source in our ecosystem, not the mandatory hub for everything. This API-first approach solves the latency and duplication problem while allowing for robust governance. We can apply security, rate limiting, monitoring, and versioning at the API layer, ensuring controlled and observable access to our most critical data assets. This evolution is already beginning to be standardized. Emerging concepts like Google’s [**Agent-to-Agent (A2A)**](https://a2aproject.github.io/A2A/latest/) **communication** and the open-source [**Model-Context Protocol (MCP)**](https://modelcontextprotocol.io/introduction) are building the foundational grammar for this new world. They aim to create a universal language for how models and agents request and receive context, abstracting away the underlying data source and treating everything as a service to be called upon. ![https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/](/static/img/articles/the-great-data-divergence-why-generative-ai-demands-a-new-approach-beyond-the-data-lake/04.webp) [https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/](https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/) ### The Future is Distributed, Connected by APIs The data lake was the right architecture for the era of big data analytics and traditional ML. But the demands of Generative AI — speed, context, and real-time interaction — require a more dynamic, distributed, and agile approach. By shifting our focus from data ingestion pipelines to robust API management, we can unlock the full potential of Generative AI without breaking our operational systems or creating an unmanageable data swamp. For any business looking to build truly intelligent, responsive, and valuable AI features, this isn’t just an architectural choice — it’s a strategic necessity. ![A decentralized approach for building Generative AI applications](/static/img/articles/the-great-data-divergence-why-generative-ai-demands-a-new-approach-beyond-the-data-lake/05.webp) A decentralized approach for building Generative AI applications ### [Hackathon Speedrun: Build & Deploy a RAG App in Minutes with Vertex AI Studio & Vertex AI Search!](https://www.fmind.dev/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/) Hackathon clock ticking? Need to get a [Retrieval Augmented Generation](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) (RAG) app up and running _fast_ using your company’s knowledge base? Forget complex setups! Google Cloud’s [Vertex AI Studio](https://cloud.google.com/generative-ai-studio?hl=en) and [Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/enterprise-search-introduction) are your secret weapons for building and deploying a grounded LLM application in what feels like minutes. Let’s walk through how to built a Q&A assistant for “[Agentoc: A2A and MCP](https://github.com/fmind/agentoc)” based on the most well known agent protocol documentations. You can adapt this for any dataset! **The Goal:** Quickly create an AI assistant that can answer questions based on a specific set of documents, providing context from the source material. **The Stack:** - [**Google Cloud Storage (GCS)**](https://cloud.google.com/storage?hl=en) **:** To store our source documents. - [**Vertex AI Search (within AI Applications)**](https://cloud.google.com/generative-ai-app-builder/docs/enterprise-search-introduction) **:** To index our documents and make them searchable. - [**Vertex AI Studio**](https://cloud.google.com/generative-ai-studio?hl=en) **:** To create our prompt, connect it to the Vertex AI Search data store (grounding), and test. - [**Cloud Run**](https://cloud.google.com/run?hl=en) **:** To deploy our AI assistant as a web app with a single click. ![Architecture of Agentoc: Assistant for A2A and MCP protocol](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/cover.webp) Architecture of Agentoc: Assistant for A2A and MCP protocol ### Step 1: Prepare Your Data in Cloud Storage First, we need a knowledge base. For this example, we got a set of documents about our [A2A](https://github.com/google/A2A) and [MCP](https://github.com/modelcontextprotocol/modelcontextprotocol) protocols from GitHub repositories. **1. Create a GCS Bucket:** - Navigate to Cloud Storage -\> Create Bucket. - Give it a globally unique name (e.g., agentoc-data-sources like I did). - Choose your location and storage class (standard is fine for this). ![Creation of the Cloud Storage Bucket](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/02.webp) Creation of the Cloud Storage Bucket **2. Format & Upload Your Data:** - For Vertex AI Search, one of the easiest ways to provide [structured and unstructured data is using **JSON Lines (.jsonlines)** format](https://cloud.google.com/generative-ai-app-builder/docs/prepare-data#storage-unstructured). Each line is a JSON object. You can use [a script like this one to prepare the data](https://github.com/fmind/agentoc/blob/main/scripts/prepare.py). Alternatively, you can just import “unstructured documents”   ```python # %% IMPORTS import os import json from pathlib import Path # %% CONFIGS PROJECT = Path(__file__).parent.parent FOLDERS = [ PROJECT / 'A2A', PROJECT / 'modelcontextprotocol', ] BUCKET = "gs://agentoc-data-sources" BUCKET_DATA = f"{BUCKET}/data" OUTPUTS = PROJECT / 'outputs' METADATA = PROJECT / 'metadata.jsonlines' # %% PROCESSING with open(METADATA, 'w') as f: for folder in FOLDERS: for path in folder.rglob('*'): try: if not path.is_file(): # skip non files continue if path.stat().st_size == 0: # skip empty files continue # output output = OUTPUTS / str(path.relative_to(PROJECT)).replace('/', '__') output = output.with_suffix(output.suffix + '.txt') output.write_text(path.read_text()) print("File:", output) # metadata metadata = { 'id': output.name.replace('.txt', '').replace('.', '--'), "structData": { "source": folder.name, "path": str(path.relative_to(folder)), }, "content": { "mimeType": "text/plain", "uri": f"{BUCKET_DATA}/{output.name}" } } jsondata = json.dumps(metadata) f.write(jsondata + '\n') except Exception as error: print(f"Error: {path} - {error}") # %% UPLOADS os.system(f"gcloud storage cp {METADATA} {BUCKET}") os.system(f"gcloud storage cp -r {OUTPUTS}/* {BUCKET_DATA}") ``` - The resulting [metadata.jsonlines](https://github.com/fmind/agentoc/blob/main/metadata.jsonlines) file looked something like this (each document on a new line):   ```json {"id": "A2A__docs__community--md", "structData": {"source": "A2A", "path": "docs/community.md"}, "content": {"mimeType": "text/plain", "uri": "gs://agentoc-data-sources/data/A2A__docs__community.md.txt"}} {"id": "A2A__docs__documentation--md", "structData": {"source": "A2A", "path": "docs/documentation.md"}, "content": {"mimeType": "text/plain", "uri": "gs://agentoc-data-sources/data/A2A__docs__documentation.md.txt"}} {"id": "A2A__docs__index--md", "structData": {"source": "A2A", "path": "docs/index.md"}, "content": {"mimeType": "text/plain", "uri": "gs://agentoc-data-sources/data/A2A__docs__index.md.txt"}} ... ``` - Make sure the URI points to the actual files (txt, pdf, html etc.) also uploaded to your bucket. I uploaded my metadata.jsonlines and the referenced documents into my **agentoc-data-sources** bucket. ### Step 2: Create a Vertex AI Search Data Store This is where the magic of making your data “searchable” by the LLM happens. **1. Navigate to Data Stores:** - In the Google Cloud Console, go to the main navigation menu -\> “AI Applications” -\> “Data Stores”. - Click “Create Data Store”. ![Vertex AI Search under the Vertex AI Section](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/03.webp) Vertex AI Search under the Vertex AI Section **2. Select Data Source:** Choose “Cloud Storage”. ![Choice of Data Store in Vertex AI Search](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/04.webp) Choice of Data Store in Vertex AI Search **3. Configure Data Import:** ![Configuration for importing data from the Cloud Storage Bucket](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/05.webp) Configuration for importing data from the Cloud Storage Bucket - **Specify the GCS URI** of your metadata.jsonlines file (e.g., gs://agentoc-data-sources/metadata.jsonlines). - **What kind of data are you importing?** I selected “Linked unstructured documents (JSONL with metadata)” because my metadata.jsonlines file points to other documents. - **Synchronization frequency:** For a hackathon, “One time” is perfect. - Click “Continue”. **4. Configure your Data Store:** ![Configuration of the Data Store](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/06.webp) Configuration of the Data Store - **Location:** I chose “global (Global)” for wide accessibility. - **Data store name:** Give it a descriptive name, like “agentoc-data-sources”. - Click “CREATE”. **5. Wait for Indexing:** Google will now process and index your documents. You can see the status on the Data Stores page. Once it’s done, you’ll see the number of documents processed (and potential errors). ![Result of the indexing process in the Data Store](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/07.webp) Result of the indexing process in the Data Store **6. Check the list of Data Store**: you should see the name of your data store. ![List of Data Store created](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/08.webp) List of Data Store created - Note: you will need the Data Store **ID** and **Location** for the next section ### Step 3: Craft and Ground Your Prompt in Vertex AI Studio Now, let’s build the brain of our assistant. **1. Navigate to Vertex AI Studio:** ![Get Started with Vertex AI Studio and API Activation](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/09.webp) Get Started with Vertex AI Studio and API Activation - Main navigation menu -\> “Vertex AI” -\> “Vertex AI Studio” -\> “Create prompt”. - You might be asked to enable some APIs if it’s your first time. Agree and continue. **2. Select Model & Write System Instructions:** - **Model:** I chose gemini-2.0-flash-001 for its speed and capability. - **System Instructions:** Give your AI a personality and purpose. Example: “You are Agentoc, an helpful assistant that provides grounded answers about the A2A and MCP protocols from the official documentation.” **3. Enable Grounding:** ![Activation of ground with Vertex AI Search](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/10.webp) Activation of ground with Vertex AI Search - Under “Tools”, toggle ON “Grounding: Vertex AI Search”. - Click the pencil icon (Edit) to configure. **4. Configure Grounding Source:** ![Hackathon Speedrun: Build & Deploy a RAG App in Minutes with Vertex AI Studio & Vertex AI Search!](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/11.webp) - Select “Vertex AI Search”. - **Vertex AI datastore path:** This is crucial! You need the full path. Go back to your Data Stores list (AI Applications -\> Data Stores), click on your data store name, and copy the “ID”. It will look something like projects/**agentoc**/locations/**global**/collections/default_collection/dataStores/**agentoc-data-sources_1746981317464** - Paste this into the path field. Change the values in bold. - Click “Save”. **5. Test Your Prompt:** Now, ask questions related to your documents! - “What is MCP Inspector?” ![Question about MCP grounded with the protocol documentation](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/12.webp) Question about MCP grounded with the protocol documentation - “What is AgentCard?” ![Question about A2A grounded with the protocol documentation](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/13.webp) Question about A2A grounded with the protocol documentation - You should see the AI generate an answer, and importantly, **cite the “Grounding Sources”** it used from your data store! This is RAG in action. **6. Save Your Prompt (Optional but Recommended):** ![Save the prompt before deploying it on Cloud Run](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/14.webp) Save the prompt before deploying it on Cloud Run - Click “Save” in Vertex AI Studio. - Give it a name like “Agentoc: A2A and MCP Assistant”. ### Step 4: Deploy as a Web App (The 1-Click Wonder!) Ready to share this? Vertex AI Studio makes it incredibly easy with the help of [Cloud Run](https://cloud.google.com/run?hl=en). 1. **Create Web App:** - In Vertex AI Studio, with your working prompt, look for a “Build with code”. Select the option to “Deploy as app”. **2. Enable APIs (if needed):** The deployment will likely use Cloud Run, Artifact Registry, and Cloud Build. Enable these if prompted. ![Activate of the required API for Cloud Run deployment](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/15.webp) Activate of the required API for Cloud Run deployment **3. Acknowledge Public Access:** _**It is important to understand the implications of this step!!!**_ With the current version of Vertex AI Studio, **you must first deploy an unauthenticated public endpoint before converting it to a private and/or authenticated deployment**. If this is not feasible for your use case, you can instead jump to Step 5. ![Be mindful about deploying to Cloud Run as a public/unauthenticated endpoint first!](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/16.webp) Be mindful about deploying to Cloud Run as a public/unauthenticated endpoint first! **4. Test Your Live App:** You’ll get a URL for your deployed Cloud Run service. Open it up and interact with your AI assistant! - **Important note**: you must open the app from Vertex AI Studio and not Cloud Run. This is important to get the secret key required to get access to your application. - **Security note**: while the key is “secret”, it can easily be retrieved from the URL via [man-in-the-middle attack](https://en.wikipedia.org/wiki/Man-in-the-middle_attack). This is not a sufficient method. ![Pending deployment of the assistant on Cloud RUn](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/17.webp) Pending deployment of the assistant on Cloud RUn - What is MCP Inspector? ![MCP question made to the web application deployed](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/18.webp) MCP question made to the web application deployed - What is AgentCard? ![A2A question made to the web application deployed](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/19.webp) A2A question made to the web application deployed ### (Bonus) Step 5: Get the Code Vertex AI Studio also gives you the Python (and other languages) code to call your grounded model programmatically. Click “Build with code” in the prompt interface, and then “Get Code”. ![Get the code from Vertex AI Studio](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/20.webp) Get the code from Vertex AI Studio You can then [put the code on your laptop](https://github.com/fmind/agentoc/blob/main/scripts/test_get_code.py) and execute it. ![Result of the “get code” script with grounded knowledge](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/21.webp) Result of the “get code” script with grounded knowledge This is super handy for integrating into other applications! ### And That’s It! In just a few steps, we went from raw documents to a fully deployed, interactive RAG application. The key is the tight integration between Cloud Storage, Vertex AI Search, and Vertex AI Studio. **Why this is a Hackathon Winner:** - **Speed:** Minimal coding, maximum automation. - **Power:** Leverages Google’s robust LLMs and search infrastructure. - **Focus:** Lets you focus on the data and the user experience, not plumbing. Now, go build something amazing for the Hackathon! - **Github Repository**: [https://github.com/fmind/agentoc](https://github.com/fmind/agentoc) ![Hackathon Speedrun: Build & Deploy a RAG App in Minutes with Vertex AI Studio & Vertex AI Search!](/static/img/articles/hackathon-speedrun-build-deploy-a-rag-app-in-minutes-with-vertex-ai-studio-vertex-ai-search/22.webp) ### [Introducing GentWriter: Building a Multi-Agent Content Generator with Google’s ADK 👾](https://www.fmind.dev/articles/introducing-gentwriter-building-a-multi-agent-content-generator-with-googles-adk/) In the fast-paced world of content creation, efficiency is key. Manually crafting social media posts for different platforms based on a single piece of content can be time-consuming and repetitive. What if you could automate this process using the power of Generative AI? 🤔 That’s exactly what I set out to do with **GentWriter**, a project exploring the capabilities of [**Google’s Agent Development Kit (ADK)**](https://google.github.io/adk-docs/) for a practical use case: generating social media content through a multi-agent strategy. In this article, I’ll walk you through how I built GentWriter, showcasing the potential of ADK for creating specialized AI agents. We’ll look at its features, the development experience, and how it leverages a multi-agent system to streamline content creation. ![Photo by Dmytro Tolokonov on Unsplash](/static/img/articles/introducing-gentwriter-building-a-multi-agent-content-generator-with-googles-adk/cover.webp) Photo by [Dmytro Tolokonov](https://unsplash.com/@dtolokonov?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### The Challenge: Taming the Social Content Beast 🦁 Keeping multiple social media feeds updated with relevant content derived from a single source (like a blog post or article) requires significant effort. Tailoring the message, tone, and length for each platform (think X vs. SEP vs. LinkedIn) adds another layer of complexity. ![Photo by Nik on Unsplash](/static/img/articles/introducing-gentwriter-building-a-multi-agent-content-generator-with-googles-adk/02.webp) Photo by [Nik](https://unsplash.com/@helloimnik?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### Enter GentWriter: Your AI Content Assistant 🤖 GentWriter tackles this challenge. Its core mission is to take a URL of an article and automatically generate: - An SEO-friendly meta description 🔍 - A concise and engaging Twitter (X) post 🐦 - A professional LinkedIn post 👔 Example with my latest article: [https://fmind.medium.com/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application-68dcae85d028](https://fmind.medium.com/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application-68dcae85d028) **SEO Description** > CAG vs RAG for AI: Understand the key differences, latency, price, and complexity to choose the right strategy for your application. **X Post** > CAG vs RAG: Which is the best strategy for your \#GenAI app? 🤔 I break down the latency, price, & complexity trade-offs in my latest article! RAG wins for large scale & low latency, but CAG is great for quick prototypes. Check it out! 👇 \#AI \#MLOps [https://fmind.medium.com/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application-68dcae85d028](https://fmind.medium.com/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application-68dcae85d028) **LinkedIn Post** > Choosing the right strategy for your AI application can be tricky! I recently read an insightful article that dives into the comparison between Context Augmented Generation (CAG) and Retrieval Augmented Generation (RAG), focusing on latency, price, and implementation complexity. > The article highlights that RAG generally offers lower latency and cost, especially for large-scale applications with substantial knowledge bases. However, CAG, particularly with caching, can be a simpler and more efficient starting point for prototypes or when dealing with smaller context sizes. It emphasizes the importance of evaluating your specific needs and constraints before deciding which approach to use. A key takeaway for me was the reminder to estimate latency and cost for your specific use case before committing to a solution. > Check out the full article for a deeper dive into the analysis and practical recommendations: [https://fmind.medium.com/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application-68dcae85d028](https://fmind.medium.com/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application-68dcae85d028) > What are your experiences with CAG and RAG? I’m curious to hear your thoughts! \#ArtificialIntelligence \#RAG \#CAG \#GenerativeAI \#LLM \#MLOps ### Under the Hood: Leveraging Google ADK 🛠️ GentWriter is built using [Google’s Agent Development Kit](https://google.github.io/adk-docs/) (ADK), which makes orchestrating multiple AI agents surprisingly simple. Here’s how it works: 1. **Article Retrieval Agent:** This agent fetches the article’s content and title from the provided URL using a dedicated tool (`get_article_from_uri`). 2. **Parallel Processing Power:** A `parallel_writer_agent` then takes over, running three specialized writer agents simultaneously: - **SEO Writer:** Crafts that crucial, concise meta description. - **X (Twitter) Writer:** Generates a punchy tweet, complete with hashtags and link, using a specific persona. - **LinkedIn Writer:** Creates a more detailed, professional post highlighting key insights for a business audience. ![Orchestration of agent included in GentWriter](/static/img/articles/introducing-gentwriter-building-a-multi-agent-content-generator-with-googles-adk/03.webp) Orchestration of agent included in GentWriter This combination of sequential steps and parallel processing makes the whole workflow efficient. The use of Pydantic ensures the data passed between agents is well-structured and reliable. ```python class Article(pdt.BaseModel): """Schema of an online article.""" link: str = pdt.Field(description="The link of the article.") title: str = pdt.Field(description="The title of the article.") content: str = pdt.Field(description="The content of the article in markdown.") seo_writer_agent = agents.LlmAgent( name="seo_writer_agent", model="gemini-2.0-flash", description="Generates concise, keyword-focused SEO meta descriptions from article information.", instruction=tw.dedent(""" - **Role:** You are an expert SEO Analyst specializing in crafting compelling meta descriptions. - **Task:** Summarize the key information from the provided article content into a concise and informative SEO meta description. - **Input:** You will receive the article object containing the link, title, and content. - **Constraint:** The description MUST be less than 150 characters. - **Objective:** Maximize click-through rate (CTR) from search engine results pages (SERPs). Focus on accuracy, relevance, and incorporating likely search keywords naturally. - **Format:** Output only the description text. Do not include labels like "SEO Description:". """), input_schema=Article, output_key="seo_description", generate_content_config=gt.GenerateContentConfig( temperature=0.3, max_output_tokens=100, ), ) ``` ### Key Features ✨ - **Automated Content Fetching:** Grabs article content and title directly from a URL. - **Multi-Platform Generation:** Creates tailored content for SEO, Twitter (X), and LinkedIn. - **Modular Agent Design:** Built with distinct, reusable agents using the `google-adk` framework. - **Configurable:** Easy to change the underlying Google Generative AI model via environment variables. - **Developer-Friendly:** Comes with a CLI and a web UI for easy testing and interaction. ![Repository of GentWriter: https://github.com/fmind/gentwriter](/static/img/articles/introducing-gentwriter-building-a-multi-agent-content-generator-with-googles-adk/04.webp) Repository of GentWriter: [https://github.com/fmind/gentwriter](https://github.com/fmind/gentwriter) ### The Development Journey: Pros & Cons 📊 Working with Google’s ADK on the GentWriter project was a positive experience overall: **What I Liked (Pros 👍):** - **Rapid Development:** Went from idea to a working prototype in just a few hours. ADK really speeds things up! - **Clean Structure:** The SDK keeps things organized and easy to manage. - **Powerful Features:** Integrations like Vertex AI Search and MCP offer exciting possibilities. - **Multi-Agent Coordination:** Handling sequential and parallel agent flows is intuitive and supported natively. - **Reliable Outputs:** Pydantic integration is great for ensuring structured data between workflow steps. - **Helpful Tooling:** The included CLI and Web UI are fantastic for development and testing! - **Asynchronous Nature:** Built with async capabilities for better performance. - **Easy Deployment:** Simple to deploy the agent as an API on Google Cloud Run. **Minor Hiccups (Cons 👎):** - **Deployment Flexibility:** While Cloud Run API deployment is smooth, more out-of-the-box options for full web apps or chatbot integrations would be welcome (e.g., Teams, Slack, Google Chat, …). - **Documentation:** Generally good, but some niche areas (like specifying dependencies for Cloud Run via `requirements.txt`) could benefit from more examples. ### See GentWriter in Action! 👀 Want to try it yourself? I’ve set up a simple web interface: ![Demo of GentWriter](/static/img/articles/introducing-gentwriter-building-a-multi-agent-content-generator-with-googles-adk/05.webp) Demo of GentWriter ### Final Thoughts ✨ Building GentWriter with Google’s ADK was an insightful experience. It’s a potent framework for developers looking to leverage LLMs in structured, multi-agent applications. While minor improvements could be made, ADK provides a solid, efficient, and enjoyable way to build complex AI solutions. If you’re automating content workflows or exploring agent-based systems, definitely give ADK a look! **Links**: - **GitHub Repository**: [https://github.com/fmind/gentwriter](https://github.com/fmind/gentwriter) - **ADK Documentation**: [https://google.github.io/adk-docs/](https://google.github.io/adk-docs/) ![Photo by Parmanand Jagnandan on Unsplash](/static/img/articles/introducing-gentwriter-building-a-multi-agent-content-generator-with-googles-adk/06.webp) Photo by [Parmanand Jagnandan](https://unsplash.com/@parmanand?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [CAG vs. RAG: Choosing the Right Strategy for Your AI Application](https://www.fmind.dev/articles/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application/) As AI Engineers, we constantly seek the most effective ways to provide Large Language Models (LLMs) with the right context to answer questions or perform tasks accurately. Two prominent techniques have emerged: [Context Augmented Generation](https://www.anthropic.com/news/contextual-retrieval) (CAG) and [Retrieval Augmented Generation](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) (RAG). But how do they stack up, especially regarding performance and cost? And when should you choose one over the other? **In this article, we’ll break down the key differences between CAG and RAG, focusing on two critical aspects: latency and price**. We’ll explore findings from a practical benchmark to understand the trade-offs, discuss the often-overlooked factor of implementation complexity, and provide recommendations to help you decide which approach best suits your specific needs, whether you’re building a quick prototype or optimizing a production system. ![Photo by Andres Siimon on Unsplash](/static/img/articles/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application/cover.webp) Photo by [Andres Siimon](https://unsplash.com/@johnmcclane?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### 🥊The Contenders: CAG and RAG Explained **Context Augmented Generation (CAG):** This approach involves directly injecting the relevant context into the LLM prompt. For large contexts, this can be inefficient. However, modern platforms like [Google’s Gemini](https://ai.google.dev/gemini-api/docs/models) API offer [caching mechanisms](https://ai.google.dev/gemini-api/docs/caching). This allows you to pre-process and cache [large contexts](https://ai.google.dev/gemini-api/docs/long-context), sending only the cache identifier and the query in subsequent requests, significantly reducing the input tokens processed per call after the initial caching. ```python # cache creation cache = genai_client.caches.create( model=GENAI_MODEL, config=gt.CreateCachedContentConfig( display_name=CACHE_NAME, contents=[content], ttl=CACHE_TTL, ) ) # cache retrieval response = genai_client.models.generate_content( model=GENAI_MODEL, contents=query, config=gt.GenerateContentConfig( cached_content=CACHE_NAME, ), ) ``` **Retrieval Augmented Generation (RAG):** Instead of sending the entire context, RAG first retrieves the most relevant snippets or chunks of information from a larger knowledge base (often stored in a vector database like [ChromaDB](https://www.trychroma.com/), [Vertex AI Search](https://cloud.google.com/enterprise-search?hl=en), or [Cloud SQL](https://cloud.google.com/sql)) using techniques like [semantic search](https://cloud.google.com/discover/what-is-semantic-search). Then, relevant chunks are added to the prompt alongside the user’s query. ```python # database ingestion database_client = cdb.PersistentClient() collection = database_client.create_collection(name=COLLECTION_NAME, embedding_function=EMBEDDING_FUNCTION) collection.add(ids=ids, documents=docs) # database retrieval docs = collection.query( include=["documents"], query_texts=[query], n_results=5, )['documents'][0] response = genai_client.models.generate_content( model=GENAI_MODEL, contents=docs + [query], ) ``` ### ⏱ Performance Showdown: Latency Latency, the time it takes to get a response, is crucial for user experience. Our analysis, using the [new structured Wikipedia data on Kaggle](https://www.kaggle.com/datasets/wikimedia-foundation/wikipedia-structured-contents) and the [`gemini-2.0-flash-001`](https://ai.google.dev/gemini-api/docs/models#gemini-2.0-flash) model, compared three simple scenarios: - **Base:** Sending the full context with the query every time. - **CAG:** Using Gemini’s [Context Caching](https://ai.google.dev/gemini-api/docs/caching) feature with full context. - **RAG:** Retrieving the top 5 relevant chunks from a [ChromaDB](https://www.trychroma.com/) database. The results are clear (see the linked [Colab Notebook](https://colab.research.google.com/drive/1yt2rHhX9X8dD1v50Sk9_o7hF_5tAt7bH?usp=sharing) for details): ![Latency Comparison: Base vs. CAG vs. RAG](/static/img/articles/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application/02.webp) Latency Comparison: Base vs. CAG vs. RAG - **Base (full context) was the slowest**, averaging 8.34 seconds, and this latency increased significantly with larger context sizes (moving from 100k to 500k tokens). - **CAG (with caching) was the next fastest**, averaging around 6.68 seconds. The initial caching takes time, but subsequent calls are faster than the base approach. - **RAG consistently demonstrated the lowest latency**, averaging around 1.06 seconds in our tests. This is because the LLM only processes a small amount of retrieved text plus the user query. ![Latency Comparison by Context Size (100k vs. 500k input tokens)](/static/img/articles/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application/03.webp) Latency Comparison by Context Size (100k vs. 500k input tokens) **Key Takeaway:** RAG holds a distinct advantage in latency, especially as the context size grows. CAG with caching offers improvement over naive context stuffing but doesn’t match RAG’s speed. ### 💰 Performance Showdown: Price Cost is a deciding factor in deploying AI solutions. The [pricing models](https://ai.google.dev/gemini-api/docs/pricing) typically involve costs per input and output token, embedding generation (for RAG), and potentially database/cache storage. To understand how these costs differ in practice, our comparison looked at various scenarios by tweaking key variables: - **Approach**: Evaluating the cost of using no Cache (Base), a Cache (CAG), and a Vector Database like Chroma or Cloud SQL (RAG) - **LLM Used:** Comparing costs between different models (Gemini 2.0 Flash vs. Flash Lite). - **Daily Request Volume:** Estimating costs based on different usage levels (1,000 vs. 10,000 requests per day). - **Tokens per Request:** Analyzing the impact of different context sizes or retrieval amounts (e.g., ranging from 5,000 tokens for RAG to 100,000 or 500,000 for CAG). These variations help illustrate how costs scale depending on the chosen model, usage intensity, and the specific context handling strategy employed. ![Price Comparison with different Models, Approaches, Databases, \# of Requests and \# of Tokens](/static/img/articles/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application/04.webp) Price Comparison with different Models, Approaches, Databases, \# of Requests and \# of Tokens Based on estimations ([see the linked Google Sheet for details](https://docs.google.com/spreadsheets/d/1WPmoqWB5UUO2tq9-z2ng2t_a6K6fqdzJTLQZxOytP4U/edit?usp=sharing)): - **RAG tends to be more cost-effective**, particularly when dealing with large underlying datasets. While it incurs costs for embedding generation and vector database hosting, the cost per LLM call is lower because far fewer tokens are processed compared to sending the entire context. For 1,000 daily requests with a 25k token context using RAG (Chroma OSS), the estimated daily cost was ~\$4.11/day. - **CAG (without caching)** becomes expensive quickly as context size increases. Sending a 100k token context for 1,000 daily requests was estimated at ~\$15.34/day, jumping to ~\$75.34/day for a 500k context. - **CAG (with caching)** significantly mitigates the cost compared to the non-cached version but adds a cache storage cost. For 1,000 daily requests, the 100k cached context cost was estimated at ~\$6.46/day, and the 500k cached context at ~\$31.06/day. ![Price / Request Comparison with different Models, Approaches, Databases, \# of Requests and \# of Tokens](/static/img/articles/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application/05.webp) Price / Request Comparison with different Models, Approaches, Databases, \# of Requests and \# of Tokens **Key Takeaway:** RAG generally wins on price for large-scale, frequent use cases, especially with very large contexts. CAG _with caching_ can be competitive at smaller context sizes (like 100k tokens in our test) but scales less favourably than RAG as context grows. ### ⚙️ Factoring The Complexity While RAG might seem like the clear winner technically, it introduces significantly more complexity during development: - **Implementation:** Setting up a RAG pipeline involves more moving parts: data chunking strategies, choosing and implementing an embedding model, setting up and managing a vector database, and creating the retrieval logic … This is often challenging for a team starting a Gen AI prototype. - **Tuning:** Optimizing RAG performance requires careful tuning — finding the right chunk size, deciding how many chunks to retrieve (`n_results`), setting similarity thresholds, etc. This takes development time and experimentation. Building a RAG is simple, but building a good RAG is complex. **CAG, especially using platform features like Gemini’s caching, is often simpler to implement initially.** ### 🎯 The Verdict: When to Use Which? There’s no single “best” answer; it depends on your specific needs: **Use CAG (preferably with caching):** - For **prototypes and MVPs** where speed of development is crucial. - When dealing with **relatively small context sizes** (e.g., under 100k tokens, where the latency/cost difference might be negligible). - If the context is **dynamic and changes frequently**, making constant re-indexing for RAG impractical (though caching also needs refreshing). **Use RAG:** - When **optimizing for low latency and cost** is a primary goal, especially at scale. - When dealing with **very large, relatively static knowledge bases** (millions of tokens or more). - When you need the ability to **cite specific sources** for the generated answer (as retrieval tells you which chunks were used). **Recommendation:** Always perform a [quick estimation](https://docs.google.com/spreadsheets/d/1WPmoqWB5UUO2tq9-z2ng2t_a6K6fqdzJTLQZxOytP4U/edit?gid=770769719#gid=770769719) of latency and cost for your specific use case and expected load. Start simple (CAG with cache if possible) for initial development and validation. If performance or cost becomes a bottleneck as you scale, invest the time to build and tune a RAG pipeline. ### 🏁 Conclusion RAG often presents a technically superior solution for handling large contexts in terms of raw latency and cost, particularly at scale. However, this performance comes at the cost of increased implementation complexity. CAG, especially when augmented with caching, offers a simpler starting point that can be perfectly adequate for many use cases, especially during prototyping or with smaller contexts. Choose wisely based on your project’s specific constraints and goals! For a deeper dive into the analysis mentioned, check out these resources: - **Latency Benchmark Notebook:** [Comparison: CAG vs. RAG — Colab](https://colab.research.google.com/drive/1yt2rHhX9X8dD1v50Sk9_o7hF_5tAt7bH) - **Price Estimation Sheet:** [Comparison: CAG vs. RAG — Ranking](https://docs.google.com/spreadsheets/d/1WPmoqWB5UUO2tq9-z2ng2t_a6K6fqdzJTLQZxOytP4U/edit?usp=sharing&authuser=1) - **My entry to Kaggle "Gemini Long Context" competition**: [https://docs.google.com/presentation/d/1H7Kq8Ur3f76mknLcugR0ZaBFqtba6foLQS_i7KG_4Sc/edit#slide=id.g315f80125e3_1_48](https://docs.google.com/presentation/d/1H7Kq8Ur3f76mknLcugR0ZaBFqtba6foLQS_i7KG_4Sc/edit#slide=id.g315f80125e3_1_48) ![Photo by Conner Baker on Unsplash](/static/img/articles/cag-vs-rag-choosing-the-right-strategy-for-your-ai-application/06.webp) Photo by [Conner Baker](https://unsplash.com/@connerbaker?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [Structured LLM Outputs Made Easy: Building a Slack Digest Agent with Pydantic AI](https://www.fmind.dev/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/) As developers and AI practitioners, many of us belong to vibrant online communities, like the [MLOps Community on Slack](https://go.mlops.community/slack). These platforms are fantastic hubs for knowledge sharing, discussion, and collaboration. However, they can also become overwhelming sources of information. Keeping up with dozens of channels and hundreds of messages daily is a real challenge! ![Structured LLM Outputs Made Easy: Building a Slack Digest Agent with Pydantic AI](/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/cover.webp) Wouldn’t it be great to have a personal assistant that could sift through the noise, identify the most relevant conversations, and summarize them neatly for you? That’s exactly what I set out to build with the [**MLOps Digester**](https://github.com/fmind/mlops-digester) project — a simple agent designed to fetch interesting Slack threads and generate concise summaries. And the secret sauce making this manageable and reliable? [**Pydantic AI**](https://ai.pydantic.dev/). ![Agent Framework / shim to use Pydantic with LLMs: https://ai.pydantic.dev/](/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/02.webp) _Agent Framework / shim to use Pydantic with LLMs:_ [https://ai.pydantic.dev/](https://ai.pydantic.dev/) In this post, we’ll dive into how Pydantic AI helps bridge the gap between the often unstructured, free-form text generated by Large Language Models (LLMs) and the structured data formats we need for robust applications. We’ll use the [MLOps Digester](https://github.com/fmind/mlops-digester) as a practical example. ### The Challenge: Getting Structured Data from LLMs LLMs like [OpenAI’s GPT models](https://ai.pydantic.dev/api/models/openai/) or [Google’s Gemini](https://ai.pydantic.dev/api/models/gemini/) are incredibly powerful at understanding and generating human-like text. You can ask them to summarize conversations, extract key points, or classify information. However, getting the output _exactly_ in the format you need (like specific JSON fields) often requires complex prompt engineering and can still be brittle. The LLM might hallucinate fields, miss required ones, or return data in an inconsistent structure. ![Slack Channels on the MLOps Community](/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/03.webp) Slack Channels on the MLOps Community ### Enter Pydantic AI: Validation Meets Language Models If you’re a Python developer, you’ve likely encountered [Pydantic](https://docs.pydantic.dev/latest/). It’s a fantastic library for [data validation](https://docs.pydantic.dev/latest/concepts/validators/) and [settings management](https://docs.pydantic.dev/latest/concepts/pydantic_settings/) using Python [type annotations](https://docs.python.org/3/library/typing.html). You define how your data _should_ look using standard Python classes, and Pydantic handles the parsing, validation, and error handling. **Pydantic AI** cleverly extends this concept to LLM interactions. It allows you to: 1. **Define your desired output structure** using a standard Pydantic model. 2. **Provide this model alongside your prompt** to an LLM (via Pydantic AI’s interface). 3. **Pydantic AI guides the LLM** to generate output that conforms to your defined model. 4. **It automatically parses and validates** the LLM’s response against your Pydantic model. Essentially, Pydantic AI acts as a mediator, ensuring the LLM’s creativity adheres to the structural requirements you define. ![High-Level Components of the MLOps Digester](/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/04.webp) High-Level Components of the MLOps Digester ### Use Case: The MLOps Digester Agent In the [`mlops-digester`](https://github.com/fmind/mlops-digester) project, the goal is to process transcripts of Slack threads and generate a structured `Digest` containing summaries of the most relevant ones. We need a consistent output format to reliably display or further process these digests. ![Project Structure of the MLOps Digester](/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/05.webp) Project Structure of the MLOps Digester **1. Defining the Structure:** First, we define our desired output structure using Pydantic models. In [`src/mlops_digester/results.py`](https://github.com/fmind/mlops-digester/blob/main/src/mlops_digester/results.py), we have something like this: ```python import pydantic as pdt class SlackThreadDigest(pdt.BaseModel): """Digest of a Slack thread.""" title: str summary: str takeaways: list[str] tags: list[str] links: list[str] tools: list[str] class SlackWorkspaceDigest(pdt.BaseModel): """Digest of a Slack workspace.""" tops: list[str] flops: list[str] moods: list[str] topics: list[str] sharing: list[str] ``` Here, the _SlackThreadDigest_ defines the fields we want for _each_ summarized thread. On the other hand, the _SlackWorkspaceDigest_ summarizes the key points of a whole Slack Workspace. **2. Creating the Agent:** Next, in [`src/mlops_digester/agents.py`](https://github.com/fmind/mlops-digester/blob/main/src/mlops_digester/agents.py), we create an agent that uses Pydantic AI. This involves initializing `PydanticAI` with an LLM client (like OpenAI's or Gemini's) and then using it to process the input text based on our desired Pydantic model. ```python def to_slack_thread_digest_agent( slack_thread_digest_agent_settings: settings.SlackThreadDigesterAgentSettings, ) -> SlackThreadDigestAgent: """Create an agent for digesting Slack threads from settings.""" agent = pdtai.Agent( name=slack_thread_digest_agent_settings.name, model_settings=slack_thread_digest_agent_settings.model_settings, deps_type=depends.SlackThreadDepends, result_type=results.SlackThreadDigest, ) @agent.system_prompt def agent_system_prompt(ctx: pdtai.RunContext[depends.SlackThreadDepends]) -> str: """Define the system prompt for the agent.""" slack_channel_name = ctx.deps.channel_name system_prompt = f"""{slack_thread_digest_agent_settings.system_prompt} The Slack Channel Name is: {slack_channel_name} """ return system_prompt return agent ``` 3\. **Running the Main Tasks** The magic happens in [`src/mlops_digester/tasks.py`](https://github.com/fmind/mlops-digester/blob/main/src/mlops_digester/tasks.py). We provide the raw Slack transcript within a prompt and tell Pydantic AI we expect an output conforming to the `Digests` model. Pydantic AI handles the interaction with the LLM, the parsing, the validation, and potentially the retries, finally returning a validated `Digest` object (or raising an error if it fails). ```python def fetch_slack_content( fetch_slack_content_step_settings: settings.FetchSlackContentStepSettings, slack_client: slack.WebClient, ) -> SlackContent: """Fetch MLOps content from Slack channels, messages, and replies.""" # dates end_date = datetime.datetime.now() start_date = end_date - datetime.timedelta( days=fetch_slack_content_step_settings.since_last_days ) # channels slack_channels = {} if fetch_slack_content_step_settings.channels: for slack_channel_id in fetch_slack_content_step_settings.channels: slack_channel_result = slack_client.conversations_info(channel=slack_channel_id) slack_channel = slack_channel_result["channel"] # extract channel data slack_channels[slack_channel["id"]] = slack_channel else: slack_conversations = slack_client.conversations_list( limit=fetch_slack_content_step_settings.max_channels_per_workspace, exclude_archived=fetch_slack_content_step_settings.exclude_archived_channels, ) for slack_channel in slack_conversations["channels"]: slack_channels[slack_channel["id"]] = slack_channel # messages for slack_channel_id, slack_channel in slack_channels.items(): slack_channel_messages = slack_channel.setdefault("messages", {}) try: slack_conversation_history = slack_client.conversations_history( channel=slack_channel_id, oldest=str(start_date.timestamp()), limit=fetch_slack_content_step_settings.max_messages_per_channel, ) for slack_channel_message in slack_conversation_history["messages"]: slack_channel_messages[slack_channel_message["ts"]] = slack_channel_message except slack_errors.SlackApiError as slack_api_error: logger.warning(f"Error while Fetching Slack Channel Messages: {slack_api_error}") # replies for slack_channel_id, slack_channel in slack_channels.items(): for slack_message_ts, slack_message in slack_channel["messages"].items(): slack_message_replies = slack_message.setdefault("replies", {}) slack_message_thread = slack_client.conversations_replies( ts=slack_message_ts, channel=slack_channel_id, limit=fetch_slack_content_step_settings.max_replies_per_message, ) for slack_message_reply in slack_message_thread["messages"]: slack_message_replies[slack_message_reply["ts"]] = slack_message_reply return slack_channels ``` This gives us confidence that `digest_output` will have all the required fields (`title`, `summary`, `takeaways`) and nested structures (SlackThreadDigest and SlackWorkspaceDigest) we defined, ready for use in our application. ![User Interface of the MLOps Digester showing the key information of a MLOps Community channel](/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/06.webp) User Interface of the MLOps Digester showing the key information of a MLOps Community channel ### Pydantic AI: A Quick Review Based on my experience using it in this project and exploring its capabilities: **Strengths:** - **Reliable Structured Output:** This is its killer feature. Significantly increases the reliability of getting JSON or other structured data from LLMs compared to prompt engineering alone. - **Leverages Pydantic’s Power:** Uses the familiar and powerful validation features of Pydantic. If you already use Pydantic, the learning curve is gentle. - **Improved Developer Experience:** Reduces the need for complex output parsing and validation logic in your application code. Type hints provide great editor support. - **LLM Agnostic:** Designed to work with various LLM providers (OpenAI, Gemini, Bedrock, etc.), giving you flexibility. **Weaknesses:** - **Abstraction Leak:** Pydantic AI relies on the underlying LLM’s ability to understand the request and the structure. If the LLM fundamentally struggles with the task complexity or the data, Pydantic AI can only enforce the _format_, not guarantee the _semantic correctness_ beyond what the LLM provides. - **Overhead:** Adds another dependency to your project. For _extremely_ simple, single-field extractions, it might feel like overkill (though still often worth it for the validation). - **Prompting Still Matters:** While it handles _structure_, you still need to write clear prompts to guide the LLM on _what_ content to generate for the fields. The descriptions in your Pydantic models become part of this implicit prompt. ### Conclusion [**Pydantic AI**](https://ai.pydantic.dev/) is a powerful tool for any developer working with LLMs who needs reliable, structured output. By combining the robust data validation of Pydantic with intelligent LLM interaction, it significantly simplifies the process of integrating generative AI into applications that require predictable data formats. The [MLOps Digester](https://github.com/fmind/mlops-digester) project demonstrates how Pydantic AI can turn a potentially messy task — summarizing noisy Slack conversations — into a structured, manageable output. It streamlines development, improves reliability, and lets you focus more on the application logic rather than fighting with inconsistent LLM responses. If you’re building applications that consume LLM output, especially for data extraction, classification, or structured content generation, I highly recommend giving Pydantic AI a try. **Link to the GitHub Repository**: [https://github.com/fmind/mlops-digester/tree/main](https://github.com/fmind/mlops-digester/tree/main) # An error occurred. Unable to execute JavaScript. Build a Slack Agent with Pydantic AI \[Step-by-Step Tutorial\] ### [GitWorks: an AI Agent for Enforcing GitHub Repository Standards](https://www.fmind.dev/articles/gitworks-an-ai-agent-for-enforcing-github-repository-standards/) Maintaining consistency and quality across software projects, especially in collaborative environments, is a significant challenge. As projects grow and teams evolve, ensuring adherence to established coding standards, best practices, and [MLOps maturity levels](https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning) becomes increasingly complex. Manually reviewing repositories against detailed checklists is time-consuming, prone to inconsistency, and often falls behind the pace of development. As an AI Engineer passionate about practical AI applications and MLOps, I frequently encounter this challenge. How can we automate the review process to ensure our GitHub repositories consistently meet defined guidelines, from basic structure to [advanced MLOps practices](https://mlops-coding-course.fmind.dev/)? This question led me to develop [**GitWorks**](https://github.com/fmind/GitWorks), a [Python notebook agent](https://github.com/fmind/GitWorks/blob/main/GitWorks_Automatically_Review_GitHub_Projects_with_Your_Guidelines.ipynb) designed to automatically review GitHub projects against your custom guidelines using the power of Generative AI. Similar to my previous projects, [BKFC for Google Chat knowledge extraction](https://fmind.medium.com/bkfc-an-agentic-workflow-for-gathering-knowledge-from-google-chat-b521cba535d7) and [GenV for Google Meet video analysis](https://fmind.medium.com/genv-an-agentic-workflow-for-actionable-insights-from-google-meet-recordings-746d465fb827), GitWorks leverages accessible tools like [Google Colab](https://colab.research.google.com/), the [GitHub API](https://docs.github.com/en/rest), and [Google’s Gemini models](https://ai.google.dev/gemini-api/docs/models) to create a focused, [agentic workflow](https://www.youtube.com/watch?v=Qd6anWv0mv0) solving a specific, real-world problem. ![GitWorks: an AI Agent for Enforcing GitHub Repository Standards](/static/img/articles/gitworks-an-ai-agent-for-enforcing-github-repository-standards/cover.webp) ### The Motivation: From Manual Checks to Automated Compliance In any software development lifecycle, adhering to standards is crucial for reliability, maintainability, and collaboration. For instance, [MLOps Guidelines](https://mlops-coding-course.fmind.dev/) might cover aspects like: - Repository structure and essential files (`README.md`, `.gitignore`, `LICENSE`) - Dependency management (`pyproject.toml`, `requirements.txt`) - Code quality (linting, formatting, typing) - Testing (setup, coverage) - Automation (pre-commit hooks, CI/CD workflows) - MLOps specific practices (configuration management, experiment tracking, model registry usage, security scanning) Manually verifying these across multiple repositories is tedious. GitWorks aims to automate this verification, providing quick, consistent feedback directly within the development workflow. ### How GitWorks Works: An Agentic Workflow for Repository Review GitWorks operates through a clear agentic process: **Define -\> Fetch -\> Analyze -\> Report**. **Define Guidelines (Configuration):** The core of GitWorks is _your_ set of guidelines. You define these in a clear text format (e.g., the MLOps Code Repository Checklist provided in the notebook ). This tells the agent _what_ to look for. ```markdown ## MLOps Code Repository Checklist This checklist helps assess the maturity of an MLOps project based on artifacts and configurations found within its GitHub repository. --- ### Level 1: Prototype _Focus: Basic functionality, primarily for project actors._ - **Repository Initialization:** `.git` directory exists, indicating version control is used. - **Basic Code Structure:** Source code files exist (e.g., `.py` files or notebooks). - **Initial README:** A basic `README.md` file exists, perhaps with a project title and brief description. - **Environment/Dependency Listing (Basic):** A `requirements.txt` or initial `pyproject.toml` might exist, listing key dependencies. --- ``` **Setup & Authentication:** Standard setup involves obtaining API keys/tokens for GitHub and Gemini. These are securely stored using Colab’s secrets management. The script uses the `PyGithub` library for GitHub interactions and `google-genai` for Gemini. ```python # GitHub github_auth = gh.Auth.Token(GITHUB_ACCESS_TOKEN) github = gh.Github(auth=github_auth) # Gemini genai_client = genai.Client(api_key=GEMINI_API_KEY) ``` **Fetch Repository Contents (Perception):** The agent connects to the specified GitHub repository using the provided token. It recursively fetches the content of all files in the repository, handling potential decoding errors. All file contents are concatenated into a single string context. ```python repository = github.get_repo(REPOSITORY) contents = [] stack = repository.get_contents("") while stack: content = stack.pop(0) if content.type == "dir": new_contents = repository.get_contents(content.path) stack.extend(new_contents) else: contents.append(content) string = io.StringIO() for content in contents: path = content.path try: text = content.decoded_content.decode() part = f"--- file: {path} ---\n{text}\n" string.write(part) except Exception as error: print(f'[ERROR] Path: "{path}", Error: {error}') string = string.getvalue() ``` **Analyze Contents (Reasoning & Action):** This is where Gemini comes in. The concatenated repository content is sent to the Gemini model (`gemini-2.0-flash` is a suitable choice) along with a system prompt instructing it to act as a Senior Software Engineer and review the code against the provided guidelines. Crucially, the prompt asks for a structured output: a summary of the review and a list of specific guidelines needing improvement, with suggestions for fixes. The desired output format is defined using a Pydantic model (`GitHubIssue`) to ensure the response is structured JSON containing a `title` and `body`. ```python class GitHubIssue(pdt.BaseModel): """GitHub Issue.""" title: str body: str instructions = f""" You are a Senior Software Engineer. Given the following guidelines, give a detailed review the repository content. Provide a general summary, and then lists the guidelines that need improvements and how to fix it. {guidelines} """ review = genai_client.models.generate_content( model=MODEL, contents=string, config=gt.GenerateContentConfig( temperature=TEMPERATURE, max_output_tokens=MAX_OUTPUT_TOKENS, system_instruction=instructions, response_mime_type='application/json', response_schema=GitHubIssue, ), ) ``` **Report Results (Output):** The structured JSON response from Gemini is parsed back into the Pydantic object. The review (`title` and `body`) is displayed in Markdown format within Colab. Optionally, if `CREATE_ISSUE` is set to `True`, the agent uses the GitHub API to automatically create a new issue in the target repository containing the review title and body. ```python if CREATE_ISSUE: issue = repository.create_issue(title=review.parsed.title, body=review.parsed.body) print('Issue created:', issue.html_url) ``` ### The Value Proposition: Why Use GitWorks? This focused agent delivers significant advantages: - **Automated Guideline Enforcement:** Ensures consistent application of your standards across projects. - **Efficiency:** Drastically reduces the time spent on manual repository reviews. - **Actionable Feedback:** Provides specific, structured feedback on areas needing improvement, directly in the repository as an issue. - **Customizable:** Works with _your_ specific guidelines, adaptable to any project type or standard. - **Low Price**: For a [relatively large ML repository](https://github.com/fmind/mlops-python-package) of 1,072,299 characters, the review cost \$0.05 using [Gemini 2.0 Flash](https://ai.google.dev/gemini-api/docs/pricing#gemini-2.0-flash). - **Practical AI Application:** Demonstrates how GenAI can automate complex analysis and reporting tasks based on code content and defined rules. ### Conclusion: Empowering Development with AI-Driven Reviews GitWorks provides a new practical example of how agentic workflows, powered by Generative AI, can automate and enhance crucial software development processes. By combining the data access capabilities of the GitHub API with the analytical power of Gemini models guided by custom instructions and structured schemas, we can transform the tedious task of repository review into an automated, efficient feedback loop. It underscores that AI’s value isn’t limited to complex conversational systems; focused, task-specific agents like GitWorks can deliver immediate, tangible benefits by automating routine checks and enforcing standards. For developers and MLOps practitioners, tools like this make it easier to maintain high quality and consistency across projects, freeing up time for more complex challenges. How else could AI agents assist in enforcing standards or automating reviews in your development workflow? Share your thoughts! **Find the notebook on GitHub:** [https://github.com/fmind/GitWorks](https://github.com/fmind/GitWorks/tree/main) ![GitWorks: an AI Agent for Enforcing GitHub Repository Standards](/static/img/articles/gitworks-an-ai-agent-for-enforcing-github-repository-standards/02.webp) ### [GenV: An Agentic Workflow for Actionable Insights from Google Meet Recordings](https://www.fmind.dev/articles/genv-an-agentic-workflow-for-actionable-insights-from-google-meet-recordings/) Video meetings on platforms like [Google Meet](https://www.google.com/search?q=google+meet&rlz=1CAAUGU_enFR1042&oq=google+meet&gs_lcrp=EgZjaHJvbWUyCQgAEEUYORiABDIGCAEQIxgnMgcIAhAAGIAEMgcIAxAAGIAEMgcIBBAAGIAEMgYIBRBFGDwyBggGEEUYPDIGCAcQRRg80gEIMTIxMmowajeoAgCwAgA&sourceid=chrome&ie=UTF-8#:~:text=Google%20Meet%20%2D%20Online,meet.google.com) are essential for collaboration, but how often do crucial details get lost moments after the call ends? Decisions are made, action items are assigned, and valuable context is shared, only to become buried in recordings that are rarely revisited. Manually scrubbing through hours of video to find specific information is tedious and inefficient. As someone deeply involved in AI, Generative AI, and MLOps, I saw another opportunity to apply these technologies to a common challenge. How can we efficiently extract the key takeaways, action items, and decisions from our meeting recordings without the manual effort? This led to the creation of [**GenV (Generative AI for Video Analytics)**](https://github.com/fmind/GenV), a [Python notebook](https://github.com/fmind/GenV/blob/main/GenV_Generative_AI_for_Video_Analytics.ipynb) designed as an agent to distill Google Meet recordings into structured, actionable insights. Similar to my previous [BKFC project for Google Chat](https://fmind.medium.com/bkfc-an-agentic-workflow-for-gathering-knowledge-from-google-chat-b521cba535d7), the goal was to build a practical, focused agentic workflow using powerful, accessible tools like [Google Colab](https://colab.google/), [Google Cloud Storage](https://cloud.google.com/storage?authuser=1), the [Vertex AI API](https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform?authuser=1), and specifically, [Vertex AI’s Gemini models](https://cloud.google.com/vertex-ai/generative-ai/docs/overview?authuser=1). ![GenV: An Agentic Workflow for Actionable Insights from Google Meet Recordings](/static/img/articles/genv-an-agentic-workflow-for-actionable-insights-from-google-meet-recordings/cover.webp) ### The Motivation: From Video Overload to Clear Takeaways [Google Meet](https://meet.google.com/) recordings often contain a wealth of information: - Detailed discussions and brainstorming sessions. - Project updates and status reports. - Decisions made and rationale explained. - Action items assigned, sometimes with owners and deadlines. - Technical details, configurations, or troubleshooting steps discussed. - Feedback and suggestions shared. Relying on memory or manual note-taking during meetings is often insufficient. Rewatching recordings is time-consuming, and note taking apps are too generic to capture specific contents and insights. [GenV](https://github.com/fmind/GenV) aims to automate the process of extracting structured knowledge from these video assets. ### How GenV Works: An Agentic Workflow for Video GenV follows a clear agentic process: **Locate -\> Prepare -\> Analyze -\> Report**. **Setup & Authentication:** Standard Google Cloud setup is required. This involves enabling the Vertex AI API in your GCP project and potentially creating a GCS bucket. The script uses `google.colab.auth` library for secure access. Google Drive access is also needed to locate the Meet recordings. **Locate & Prepare Data (Perception & Preparation):** The agent first needs access to the video files. - It mounts your Google Drive to access the specified path for Meet recordings (e.g., `MyDrive/Meet Recordings`). - It identifies video files modified within a recent period (e.g., last 30 days, configurable via `SINCE_DAYS`). - Crucially, because the Vertex AI API often works best with cloud storage, the script uploads these video files from Drive to a designated Google Cloud Storage (GCS) bucket if they don’t already exist there. This prepares the video for analysis. **Analyze Data (Reasoning & Action):** This is where Vertex AI’s Gemini model performs the heavy lifting. - **Structured Output Definition:** Like with BKFC, defining the _desired_ output structure is key. We use [Pydantic models](https://docs.pydantic.dev/latest/concepts/models/) to create a detailed schema, `MeetingInsight`, specifying exactly what information to extract. This includes fields for: - `title`: Inferred meeting title. - `summary`: A concise meeting summary. - `questions_answers`: Pairs of questions and answers. - `unanswered_questions`: Questions lacking clear answers. - `projects_discussed`: Project updates and details. - `action_items`: Tasks with potential owners and deadlines. - `decisions_suggestions`: Key decisions or proposals. - `technical_insights`: Technical details mentioned. - `key_topics_mentioned`: Other important themes or keywords.   ```python class MeetingInsight(pdt.BaseModel): """Structured insights extracted from a Google Meet recording analysis.""" title: T.Optional[str] = pdt.Field( default=None, description="The inferred title or primary subject of the meeting based on the discussion." ) summary: T.Optional[str] = pdt.Field( default=None, description="A concise summary (3-5 sentences) of the main topics discussed and key outcomes of the meeting." ) questions_answers: T.Optional[list[QuestionAnswer]] = pdt.Field( default=None, description="A list of significant questions asked and their corresponding answers from the meeting." ) unanswered_questions: T.Optional[list[str]] = pdt.Field( default=None, description="A list of significant questions that were asked but do not appear to have been clearly answered during the meeting.." ) projects_discussed: T.Optional[list[ProjectInfo]] = pdt.Field( default=None, description="A list of specific projects discussed, including status or key points." ) action_items: T.Optional[list[ActionItem]] = pdt.Field( default=None, description="A list of specific tasks or action items assigned, including owner and deadline if specified." ) decisions_suggestions: T.Optional[list[DecisionSuggestion]] = pdt.Field( default=None, description="A list of key decisions made or significant suggestions proposed during the meeting." ) technical_insights: T.Optional[list[str]] = pdt.Field( default=None, description="Specific mentions related technical solutions, configurations, or code snippets discussed." ) key_topics_mentioned: T.Optional[list[str]] = pdt.Field( default=None, description="A list of other important topics, keywords, or themes discussed that are not captured elsewhere (e.g., specific technologies, methodologies, upcoming events, general announcements)." ) ``` - **API Call:** For each video file (referenced by its GCS URI), the script sends a request to the [Gemini model](https://cloud.google.com/vertex-ai/generative-ai/docs/models) (`gemini-2.0-flash` or similar multimodal model). The key here is providing the video URI directly to the model, along with the prompt and, importantly, specifying the `response_mime_type` as `application/json` and providing the `response_schema` (our `MeetingInsight` Pydantic model). This multimodal capability combined with structured output is incredibly powerful. - **Parsing:** The Gemini API directly returns JSON conforming to the schema, which the library automatically parses into our Pydantic `MeetingInsight` object.   ```python # Simplified Gemini API Call Example uri = f"gs://{blob.bucket.name}/{blob.name}" # GCS URI of the video contents = [ GT.Part.from_uri(file_uri=uri, mime_type="video/mp4"), prompt, # Your prompt guiding the analysis ] response = genai_client.models.generate_content( model=MODEL, # e.g., "gemini-2.0-flash" contents=contents, config={ "response_mime_type": "application/json", "response_schema": MeetingInsight, # Providing the schema "temperature": TEMPERATURE }, ) insight = response.parsed # The structured Pydantic object ``` **Report Results (Output):** The final step presents the insights clearly. - The script iterates through the `MeetingInsight` objects for each video. - It generates readable Markdown summaries, organizing the extracted information under relevant headings (Summary, Q&A, Projects, Action Items, Decisions, etc.). See example below. - It also saves the raw structured data to a `jsonlines` file for potential downstream use. _**Example Output Snippet (from**_ _**`video_insights.txt`**):_** ```markdown # MLOps Community - Kubeflow ## Meeting Title Kubeflow: The ML Toolkit for Kubernetes ## Summary The speaker introduces Kubeflow as a toolkit for doing machine learning on top of Kubernetes. He outlines the agenda for the presentation, which includes components, architecture, ML workflow, Kubeflow pipelines, and a Q&A session. He emphasizes that Kubeflow is an open-source ML Ops platform based on Kubernetes, primarily using Python and YAML. ## Projects Discussed - **Kubeflow:** Kubeflow is an open-source ML Ops platform based on Kubernetes, primarily using Python and YAML. It is a tool to power the develop platform with Kubernetes. The project has been sponsored by Google. ## Technical Insights - Kubeflow uses Python and YAML. - Kubeflow is based on Kubernetes. - Kubeflow uses Docker. ## Key Topics - MLOps - Kubernetes - Python - YAML - Docker - Pipelines - Components - Architecture ``` ### The Value Proposition: Why Use GenV? This focused agent provides tangible benefits: - **Rapid Meeting Summarization:** Get the gist and key outcomes of meetings quickly without rewatching. - **Action Item Extraction:** Reliably capture tasks, owners, and deadlines mentioned during calls. - **Decision Tracking:** Easily reference key decisions made and suggestions proposed. - **Knowledge Retrieval:** Quickly find specific technical details, project updates, or answers discussed. - **Identify Follow-ups:** Surface unanswered questions needing attention. - **Efficiency Boost:** Saves significant time compared to manual review or note-taking. - **Practical AI Demonstration:** Showcases how multimodal models like Gemini combined with structured output can automate information extraction from video content. ### Conclusion: Tapping into Video Knowledge with Focused AI The [GenV notebook](https://github.com/fmind/GenV/blob/main/GenV_Generative_AI_for_Video_Analytics.ipynb) demonstrates how agentic workflows powered by multimodal AI can unlock the valuable information trapped within video recordings. By leveraging the analytical power of [Vertex AI’s Gemini models](https://cloud.google.com/vertex-ai/generative-ai/docs/models), guided by precise structured output schemas, we can transform hours of meeting video into concise, actionable summaries. This highlights that sophisticated AI benefits don’t always require complex conversational agents. Focused, task-specific agents like [GenV](https://github.com/fmind/GenV) can automate laborious processes and make information more accessible. For those of us working in AI and MLOps, building such targeted solutions using cloud platforms like Google Cloud is increasingly feasible, allowing us to enhance productivity and knowledge sharing. What other types of content could benefit from automated, structured insight extraction using Generative AI? Let me know your thoughts! ![GenV: An Agentic Workflow for Actionable Insights from Google Meet Recordings](/static/img/articles/genv-an-agentic-workflow-for-actionable-insights-from-google-meet-recordings/02.webp) ### [BKFC: An Agentic Workflow for Gathering Knowledge from Google Chat](https://www.fmind.dev/articles/bkfc-an-agentic-workflow-for-gathering-knowledge-from-google-chat/) ### BKFC: **An Agentic Workflow for** Gathering **Knowledge from Google Chat** Team collaboration often lives and breathes within chat applications like [Google Chat](https://chat.google.com/) or [Slack](https://slack.com/). It’s where questions are asked, decisions are made, project updates are shared, and solutions are brainstormed. But let’s be honest: extracting specific information from weeks or months of chat history can feel like searching for a needle in a haystack. Important context gets buried, action items are forgotten, and valuable knowledge becomes siloed. As someone passionate about AI, Generative AI, and MLOps, I’m always looking for practical ways to leverage these technologies to solve real-world problems. The challenge of unlocking the latent knowledge within Google Chat seemed like a perfect opportunity. How could we quickly surface key insights without manual scrolling and searching? This led me to create [**BKFC (Build Knowledge From Chats)**](https://github.com/fmind/BKFC), a [Python notebook](https://github.com/fmind/BKFC/blob/main/BKFC_Build_a_Knowledge_base_From_Chats.ipynb) designed to act as a simple yet powerful insight-gathering agent for Google Chat. My goal wasn’t just to extract information, but also to demonstrate how quickly valuable, focused agentic workflows can be built using modern tools like [Google Colab](https://colab.google/), the [Google Chat API](https://developers.google.com/workspace/chat), and [Vertex AI’s Gemini models](https://cloud.google.com/vertex-ai/generative-ai/docs/overview). ![BKFC: An Agentic Workflow for Gathering Knowledge from Google Chat](/static/img/articles/bkfc-an-agentic-workflow-for-gathering-knowledge-from-google-chat/cover.webp) ### The Motivation: From Chat Chaos to Actionable Insights In many teams, Google Chat spaces become informal knowledge repositories. You might find: - Questions asked and answered. - Updates on ongoing projects. - Implicit or explicit action items. - Valuable technical discussions, code snippets, or configuration details. - Feedback and suggestions for improvement. **Manually scrolling through this history is inefficient and error-prone**. Key information can be easily missed, especially as teams grow and conversations multiply. I wanted a way to automatically distill recent conversations into structured, actionable summaries. ### How BKFC Works: An Agentic Workflow in Action The notebook follows a straightforward, agentic process: **Fetch -\> Process -\> Analyze -\> Report**. **Setup & Authentication:** The first step involves standard Google Cloud setup. This means enabling the Google Chat and Vertex AI APIs in a GCP project and creating OAuth credentials (specifically for a Desktop app) to allow the script to securely access chat data on your behalf. Colab’s secrets management handles the client ID, secret, and project ID securely. Authentication uses the `gcloud auth application-default login` flow, granting the necessary `chat.spaces.readonly` and `chat.messages.readonly` scopes. ```bash !gcloud auth application-default login --no-browser --client-id-file={SECRETS} --scopes={",".join(SCOPES)} ``` **Fetch Data (Perception):** The agent’s “perception” phase involves using the `googleapiclient` library to interact with the Google Chat API. - It lists all chat spaces the authenticated user has access to. - It filters these spaces to only include those active within a defined recent period (e.g., the last 100 days, configurable via `SINCE_DAYS`). - For each relevant space, it fetches messages created since that period, respecting API pagination (`PAGE_SIZE`).   ```python spaces = [] page_token = None while True: response = chat_service.spaces().list(pageSize=PAGE_SIZE, pageToken=page_token).execute() for space in response.get('spaces', []): last_active_time = dt.datetime.fromisoformat(space['lastActiveTime']) last_active_date = last_active_time.date() if last_active_date >= since: spaces.append(space) if not page_token: break len(spaces) ``` **Process Data (Preparation):** The raw message data needs some structuring before analysis. - Messages are sorted chronologically within their respective spaces and threads. - They are then grouped by the parent chat space. - For each space, the formatted text of the messages is concatenated into a single “page” or document representing the recent conversation history for that space.   ```python messages = [] for space in spaces: page_token = None while True: response = chat_service.spaces().messages().list( parent=space['name'], filter=f'createTime > "{since}T00:00:00+00:00"', orderBy='createTime DESC', pageToken=page_token, pageSize=PAGE_SIZE, ).execute() messages.extend(response.get('messages', [])) if not page_token: break len(messages) ... messages = sorted(messages, key=message_sorted_key, reverse=True) groups = {key: list(values) for key, values in it.groupby(messages, key=message_groupby_key)} ``` **Analyze Data (Reasoning & Action):** This is where the Gen AI magic happens, powered by Gemini via the Vertex AI API (`python-genai` library). - **Structured Output Definition:** A key aspect is defining _what_ we want to extract. I used [Pydantic](https://docs.pydantic.dev/latest/) models to define a clear schema (`ChatInsight`) for the desired output. This schema includes fields for: - `summary`: A concise overview of the discussion. - `questions_answers`: Pairs of questions and their corresponding answers found in the chat. - `unanswered_questions`: Questions asked that appear unresolved. - `projects`: Mentioned projects and their status/details. - `action_items`: Tasks identified, potentially with assignees. - `feedback_suggestions`: Ideas or critiques shared. - `technical_insights`: Specific mentions of MLOps, AI tools (like Vertex AI), code, etc.   ```python # --- Define the main structure for the overall chat insights --- class ChatInsight(pdt.BaseModel): """Structured insight extracted from a Google Chat conversation history.""" summary: T.Optional[str] = pdt.Field(...) questions_answers: T.Optional[list[QuestionAnswerPair]] = pdt.Field(...) unanswered_questions: T.Optional[list[str]] = pdt.Field(...) projects: T.Optional[list[ProjectInfo]] = pdt.Field(...) action_items: T.Optional[list[ActionItem]] = pdt.Field(...) feedback_suggestions: T.Optional[list[str]] = pdt.Field(...) technical_insights: T.Optional[list[str]] = pdt.Field(...) # (Inner classes like QuestionAnswerPair, ProjectInfo defined elsewhere) ``` - **API Call:** For each space’s conversation page, a prompt is sent to the Gemini model (`gemini-2.0-flash` is a great choice for speed and cost-effectiveness here). Crucially, the API call specifies the desired `response_mime_type` as `application/json` and provides the `response_schema` (our `ChatInsight` Pydantic model). This instructs Gemini to format its response according to our defined structure. - **Parsing:** The library automatically parses the JSON response back into a Pydantic `ChatInsight` object.   ```python insights = {} for key, page in pages.items(): page = pages[key] prompt = ANALYSIS_TEMPLATE.substitute(page=page) try: response = genai_client.models.generate_content( model=MODEL, contents=prompt, config={ "response_mime_type": "application/json", "response_schema": ChatInsight, "temperature": TEMPERATURE }, ) print(key, response.usage_metadata.total_token_count) insights[key] = response.parsed except Exception as error: print(f"An error occurred during API call for space {key}: {error}") len(insights) ``` **Report Results (Output):** The final step is presenting the extracted insights. - The structured `ChatInsight` objects are iterated through. - Readable Markdown summaries are generated for each chat space, organizing the extracted information under clear headings (Summary, Q&A, Projects, Actions, etc.). - The raw structured data is also saved to a `jsonlines` file, which is ideal for potential downstream processing or integration with other tools.   ```markdown ## Summary Discussions highlight active development of AI solutions, particularly using LLMs for document analysis and market prediction. A strong emphasis emerged on the necessity of solid **MLOps** practices for managing these initiatives effectively, including standardization and deployment strategies, even as significant budgets are approved. ## Projects - **Predictive Insights Initiative:** Focuses on leveraging LLMs for market trends, with explicit discussion around needing standardized MLOps pipelines for deployment and monitoring. ## Feedback & Suggestions - Suggestion to establish shared MLOps best practices across teams working on similar ML problems to improve efficiency and consistency. ## Technical Insights - Need identified for standardized ML model deployment and monitoring pipelines. - Importance of data standardization for reliable LLM inputs emphasized. ``` ### The Value Proposition: Why Bother? This simple notebook, acting as a focused agent, delivers this added value: - **Rapid Knowledge Retrieval:** Quickly get summaries and key points from recent chats without manual searching. - **Action Item Tracking:** Surface potential tasks or commitments that might have been missed. - **Project Awareness:** Get a quick pulse check on mentioned projects. - **Identify Unanswered Questions:** Highlight areas where follow-up might be needed. - **Technical Knowledge Sharing:** Easily find references to specific tools, techniques, or solutions discussed. - **Efficiency:** Saves considerable time compared to manual review. - **Demonstrates Practical AI:** Shows how easily accessible Gen AI models and APIs can be combined to build genuinely useful tools _today_. ### Conclusion: Simple Agents, Tangible Value The [BKFC notebook](https://github.com/fmind/BKFC/blob/main/BKFC_Build_a_Knowledge_base_From_Chats.ipynb) is a practical example of how agentic workflows, even simple ones, can provide immediate value. By combining the power of the Google Chat API to access data and the reasoning capabilities of Gemini models guided by structured schemas, we can transform conversational noise into actionable insights. It highlights that you don’t always need complex, multi-turn conversational agents to benefit from AI. Focused, task-specific agents like this can automate tedious processes and unlock information trapped in our daily communication streams. As AI and MLOps practitioners, building these kinds of targeted solutions is becoming increasingly straightforward, enabling us to enhance productivity and knowledge sharing within our teams. What other routine information retrieval tasks could be automated with a similar approach? Let me know in comments! - **Github Repository**: [https://github.com/fmind/BKFC](https://github.com/fmind/BKFC) - **Colab Notebook**: [https://github.com/fmind/BKFC](https://github.com/fmind/BKFC) [https://github.com/fmind/BKFC/blob/main/BKFC_Build_a_Knowledge_base_From_Chats.ipynb](https://github.com/fmind/BKFC/blob/main/BKFC_Build_a_Knowledge_base_From_Chats.ipynb) ![BKFC: An Agentic Workflow for Gathering Knowledge from Google Chat](/static/img/articles/bkfc-an-agentic-workflow-for-gathering-knowledge-from-google-chat/02.webp) ### [Framework, Template, or Example? 🤔 Choosing the Right AI Starter Kit for Your Team ✨](https://www.fmind.dev/articles/framework-template-or-example-choosing-the-right-ai-starter-kit-for-your-team/) Building AI applications is incredibly exciting. But let’s be honest: getting started can be challenging, especially in large organizations. Setting up environments, structuring projects, writing boilerplate code, navigating CI/CD pipelines, figuring out _which_ cloud account or service to use… all of this eats up precious time before you can ship your model in production. **This is where AI Starter Kits come to the rescue**🦸‍♀️. They provide a launchpad, helping you ship AI applications faster and promoting best practices along the way. Broadly, they fall into three categories: Code Frameworks, Code Templates, and Code Examples. Picking the right one is crucial and depends heavily on your team’s needs, how cutting-edge your AI domain is, and how much flexibility you want. ![Photo by John Salvino on Unsplash](/static/img/articles/framework-template-or-example-choosing-the-right-ai-starter-kit-for-your-team/cover.webp) Photo by [John Salvino](https://unsplash.com/@jsalvino?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) Let’s dive in and unpack each one: ### 1. Code Frameworks: The Guided Path 🧭 **Definition:** Frameworks provide a set of code abstractions (often via a library 📚) that dictate _how_ you structure and write significant parts of your application. They offer pre-built components and enforce a specific way of working, like abstractions for [ML pipelines](https://mlops-coding-course.fmind.dev/2.%20Prototyping/2.5.%20Modeling.html) or [data catalogs](https://mlops-coding-course.fmind.dev/3.%20Productionizing/3.4.%20Configurations.html). **Pros:** - **Strong Guidance:** Enforces patterns and structure, keeping everyone on the same page. - **Laser Focus:** Helps developers concentrate on the unique problem they’re solving, not reinventing the wheel for plumbing code. - **Consistency is Key:** Ensures projects built with the framework share a common, understandable architecture. **Cons:** - **Potential Rigidity:** The abstractions might feel like a straitjacket if your use case doesn’t fit perfectly. Fighting a framework is never fun! - **Engineer Opinions:** Let’s face it, developers have preferences. What one finds helpful, another might see as restrictive, potentially hurting adoption. I recall having endless debate about single-quote vs. double-quote … - **Heavy Upfront Lift:** Defining really good, useful abstractions takes serious brainpower and foresight. There is a step learning curse in using framework. **When to Use:** Best suited for domains where: - Coding patterns are mature and accepted. - The _types_ of apps being built don’t vary wildly. - Consistency across many projects is a top priority. **In the AI World:** While web development enjoys mature frameworks like Django or React, the AI/ML landscape is… well, wilder! Frameworks like [Kedro](https://kedro.org/) and [ZenML](https://www.zenml.io/) offer powerful structures for pipelines and processes, but the sheer variety of ML tasks (forecasting vs. NLP vs. vision) and tech choices (MLflow vs. W&B, different clouds️) makes a one-size-fits-all framework tricky without boxing developers in. **Examples:** Django (Web), [Kedro](https://kedro.org/) (Data / ML), [ZenML](https://www.zenml.io/) (MLOps). ![Kedro-Viz — Pipeline Visualisation](/static/img/articles/framework-template-or-example-choosing-the-right-ai-starter-kit-for-your-team/02.webp) [Kedro-Viz — Pipeline Visualisation](https://kedro.org/) ### 2. Code Templates: The Structured Starting Point 🏗️ **Definition:** Templates are like code blueprints. They give you an initial project structure, config files, basic boilerplate, and setup scripts (e.g., CI/CD). They initialize your workspace, but don’t dictate your internal code style quite as strongly as frameworks. **Pros:** - **Automated Setup:** Creates a standardized project layout in seconds, often including DevOps and MLOps goodies. - **Customizable at Will:** Often use tools like Cookiecutter or GitHub Templates, letting you personalize things (project name, author, configs) right at the start. - **Sweet Flexibility:** Provides essential structure without strangling your core application logic (e.g., functions vs. objects, design patterns, …). **Cons:** - **Maintenance Duty:** templates need regular updates to keep pace with evolving tools, best practices, and cloud environments. - **One Size Fits _Most_:** To be useful broadly, they might include things not strictly needed for _every_ single project. You either put too much, or too little. - **Less Code Hand-Holding:** Doesn’t enforce specific coding patterns _inside_ your app modules like a framework does. **When to Use:** Often hits the sweet spot when: - Your code _delivery_ process (CI/CD, packaging, deployment) is fairly standardized. - You build diverse project _types_ (forecasting, recsys, classification), making rigid code abstractions difficult. - You want consistency in project structure and DevOps across different kinds of apps. **In the AI World:** For many MLOps teams, templates are golden! They standardize how models get packaged, tested, deployed, and monitored, even if the ML models themselves are totally different. They’re great for navigating that tricky “company landscape” (which account? dev/staging/prod?️) without dictating exactly how you write your code. **Examples:** [cookiecutter-pypackage](https://github.com/audreyfeldroy/cookiecutter-pypackage) (Python), [cookiecutter-flask](https://github.com/cookiecutter-flask/cookiecutter-flask) (Web), [cookiecutter-mlops-package](https://github.com/fmind/cookiecutter-mlops-package) (MLOps) ```json { "user": "fmind", "name": "MLOps Project", "repository": "{{cookiecutter.name.lower().replace(' ', '-')}}", "package": "{{cookiecutter.repository.replace('-', '_')}}", "license": "MIT", "version": "0.1.0", "description": "TODO", "python_version": "3.13", "mlflow_version": "2.20.3", "__prompts__": { "user": "GitHub User", "name": "Project Name", "repository": "GitHub Repository", "package": "Python Package", "license": "Project License", "version": "Project Version", "description": "Project Description", "python_version": "Python Version", "mlflow_version": "MLflow Version" } } ``` ### 3. Code Examples: The Concrete Illustration ✨ **Definition:** Exactly what it sounds like: a complete, working codebase solving _one specific problem_ end-to-end. It’s a reference point, a “show, don’t just tell” approach. **Pros:** - **Super Simple:** Easiest type of starter kit to build and to maintain. - **Crystal Clear:** Provides a tangible, easy-to-understand example of how to get something _done_. - **Lightning Fast:** Quickest way to offer _some_ guidance when developers tackle something new. **Cons:** - **DIY Required️:** Users have to manually copy, paste, and heavily adapt the code for their own needs. No setup automation here. - **Limited Mileage:** Doesn’t offer much leverage beyond the specific use case shown. - **Less “Value Add”:** Doesn’t enforce standards or automate setup like the framework and template. **When to Use:** Your best bet when: - You need to provide guidance _fast_ for a brand-new or rapidly evolving area (hello, GenAI! 👋). - Neither the core logic patterns nor the delivery methods are mature or standardized yet. - The goal is inspiration and illustration (“Here’s _one way_ to do it”) rather than strict enforcement or automation. **In the AI World:** This often hits the nail on the head for cutting-edge domains like Generative AI. The field is moving at warp speed — new models (multimodal), new ways to interact (agents), tons of ways to deliver value (Slack, Teams, Web Apps, APIs…). Defining solid frameworks or even stable templates is like trying to map a river during a flood. Clear, working examples for specific tasks (e.g., building a RAG pipeline, fine-tuning model X, deploying an agent via API) offer immediate help without getting bogged down in standards that might be obsolete next month. **Examples:** [mlops-python-package](https://github.com/fmind/mlops-python-package) (specific ML forecasting pipeline), repository of examples for building agent ([agent-start-pack](https://github.com/GoogleCloudPlatform/agent-starter-pack)) ![Structure of the MLOps Python Package](/static/img/articles/framework-template-or-example-choosing-the-right-ai-starter-kit-for-your-team/03.webp) [Structure of the MLOps Python Package](https://github.com/fmind/mlops-python-package) ### Making the Choice: Context is Everything! 🌍 So, which starter kit reigns supreme? Surprise! There’s no single winner. The right choice is a smart trade-off based on _your_ specific situation: 1. **Domain Maturity:** How well-paved is the road in your field? (Web Dev: Mature -\> Frameworks/Templates \| MLOps️: Maturing -\> Templates/Frameworks \| Gen AI: Nascent -\> Examples) 2. **Project Variety:** Are you building mostly clones or special snowflakes? (Low Variety -\> Frameworks \| High Variety -\> Templates/Examples) 3. **Team Needs & Vibes:** Does the team needs detailed guidance, or just a nudge? Are they happy adapting examples, or do they expect automated setup? 4. **Org Standards:** How consistent are your deployment, monitoring, and infra practices? (Standardized -\> Templates rock \| Less Standardized -\> Examples might be easier to start) 5. **Speed vs. Scale:** Examples get you moving fast initially but can lead to different paths. Frameworks ensure consistency but need upfront buy-in. Templates often strike a nice balance. ![Decision Tree: Code Framework vs. Template vs. Example for AI/ML Projects](/static/img/articles/framework-template-or-example-choosing-the-right-ai-starter-kit-for-your-team/04.webp) Decision Tree: Code Framework vs. Template vs. Example for AI/ML Projects Ultimately, any starter kit’s mission is to empower your AI/ML engineers. They should help your teammates build up the next big innovation, not wrestling with repetitive setup tasks or getting lost trying to figure out the company’s cloud maze️. Talk to your teammates to ask what they prefer! By thoughtfully weighing the pros and cons of frameworks, templates, and examples — and aligning your choice with the specific AI domain you’re in — you can provide the _perfect_ level of support. Choose wisely, remove the friction, and watch your teams build amazing AI, faster than ever! ![Photo by Markus Spiske on Unsplash](/static/img/articles/framework-template-or-example-choosing-the-right-ai-starter-kit-for-your-team/05.webp) Photo by [Markus Spiske](https://unsplash.com/@markusspiske?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [More Automation + More Reproducibility = MLOps Python Package v4.1.0](https://www.fmind.dev/articles/more-automation-more-reproducibility-mlops-python-package-v4-1-0/) The [**MLOps Python Package**](https://github.com/fmind/mlops-python-package) is your go-to solution for building robust and reproducible machine learning workflows. With the latest v4.1.0 release, we’ve doubled down on automation and reproducibility, making your MLOps journey smoother than ever. ![Credit to ashishpatel26 for the diagram and his contribution!](/static/img/articles/more-automation-more-reproducibility-mlops-python-package-v4-1-0/cover.webp) Credit to [ashishpatel26](https://github.com/ashishpatel26) for the diagram and his contribution! ### What’s New in v4.1.0? This release introduces several key features that enhance automation and reproducibility: #### 1. Switch from PyInvoke to Just We’ve transitioned from [PyInvoke](https://www.pyinvoke.org/) to [Just](https://just.systems/) for managing our automation tasks. [Just](https://just.systems/) offers a cleaner and more concise syntax compared to [PyInvoke](https://www.pyinvoke.org/) and [GNU Make](https://www.gnu.org/software/make/manual/make.html), making it easier to read and maintain our workflows. You can now trigger actions directly from your command line or use the [Just VS Code extension](https://marketplace.visualstudio.com/items?itemName=nefrob.vscode-just-syntax) for a seamless development experience. Here’s an example of the main [justfile](https://github.com/fmind/mlops-python-package/blob/v4.1.0/justfile): ```justfile # https://just.systems/man/en/ # REQUIRES docker := require("docker") find := require("find") rm := require("rm") uv := require("uv") # SETTINGS set dotenv-load := true # VARIABLES PACKAGE := "bikes" REPOSITORY := "bikes" SOURCES := "src" TESTS := "tests" # DEFAULTS # display help information default: @just --list # IMPORTS import 'tasks/check.just' import 'tasks/clean.just' import 'tasks/commit.just' import 'tasks/doc.just' import 'tasks/docker.just' import 'tasks/format.just' import 'tasks/install.just' import 'tasks/mlflow.just' import 'tasks/package.just' import 'tasks/project.just' ``` And the one the task file ([tasks/check.just](https://github.com/fmind/mlops-python-package/blob/v4.1.0/tasks/check.just)) ```justfile # run check tasks [group('check')] check: check-code check-type check-format check-security check-coverage # check code quality [group('check')] check-code: uv run ruff check {{SOURCES}} {{TESTS}} # check code coverage [group('check')] check-coverage numprocesses="auto" cov_fail_under="80": uv run pytest --numprocesses={{numprocesses}} --cov={{SOURCES}} --cov-fail-under={{cov_fail_under}} {{TESTS}} # check code format [group('check')] check-format: uv run ruff format --check {{SOURCES}} {{TESTS}} # check code security [group('check')] check-security: uv run bandit --recursive --configfile=pyproject.toml {{SOURCES}} # check unit tests [group('check')] check-test numprocesses="auto": uv run pytest --numprocesses={{numprocesses}} {{TESTS}} # check code typing [group('check')] check-type: uv run mypy {{SOURCES}} {{TESTS}} ``` #### 2. Gemini Code Assist for GitHub PR Reviews Say hello to your new AI-powered code reviewer! We’ve integrated [Gemini Code Assist](https://developers.google.com/gemini-code-assist/docs/review-github-code) to help you streamline your pull request review process. Gemini can automatically suggest code improvements, identify potential issues, and even summarize PR changes, saving you valuable time and effort. Here is an example of [a Gemini Code Assist review on GitHub](https://github.com/fmind/mlops-python-package/pull/52#pullrequestreview-2662381245): > This pull request introduces several enhancements, including Gemini code assist support, Dependabot configuration, and default GitHub rulesets. The changes appear well-structured and contribute to improved automation and security. However, there are a few areas that could benefit from further attention. > **Summary of Findings** > Missing Error Handling in `install-rulesets` task: The `install-rulesets` task in `tasks/install.just` lacks error handling for the `gh api` command. If the API call fails, the script will not report the error, potentially leading to undetected deployment issues. > README.md Updates: The README.md file was updated to include new tools, but it would be beneficial to add more context around the usage of these tools and how they integrate into the project’s workflow. > **Merge Readiness** > The pull request is almost ready for merging. Addressing the missing error handling in the `install-rulesets` task is crucial. Additionally, enhancing the README.md with more detailed information about the new tools would be beneficial. I am unable to directly approve the pull request, and recommend that others review and approve this code before merging. At a minimum, the critical and high severity comments should be addressed before merging. > Consider adding error handling to the `gh api` command to ensure that deployment failures are detected and reported. This can be done by checking the exit code of the command and logging an error message if it is non-zero. > gh api --method POST -H "Accept: application/vnd.github+json" \\\ >  "/repos/\$owner/\$repo/rulesets" --input=".github/rulesets/main.json" \|\| echo "Failed to install rul #### 3. Automatic GitHub Rulesets Push Maintaining consistent code quality and security practices is crucial for any MLOps project. With this release, we’ve automated the process of pushing [GitHub rulesets](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets). This ensures that your codebase adheres to predefined standards, reducing the risk of errors and vulnerabilities. Here’s an example of a [GitHub ruleset applied on the project main branch](https://github.com/fmind/mlops-python-package/blob/main/.github/rulesets/main.json): ```json { "name": "main", "target": "branch", "enforcement": "active", "conditions": { "ref_name": { "exclude": [], "include": [ "~DEFAULT_BRANCH" ] } }, "rules": [ { "type": "deletion" }, { "type": "required_linear_history" }, { "type": "pull_request", "parameters": { "required_approving_review_count": 0, "dismiss_stale_reviews_on_push": true, "require_code_owner_review": false, "require_last_push_approval": false, "required_review_thread_resolution": false, "allowed_merge_methods": [ "squash", "rebase" ] } }, { "type": "required_status_checks", "parameters": { "strict_required_status_checks_policy": true, "do_not_enforce_on_create": false, "required_status_checks": [ { "context": "checks", "integration_id": 15368 } ] } }, { "type": "non_fast_forward" } ], "bypass_actors": [ { "actor_id": 5, "actor_type": "RepositoryRole", "bypass_mode": "always" } ] } ``` #### 4. Deterministic Python Wheel with constraints.txt Reproducibility is paramount in MLOps. To ensure consistent builds, we’ve introduced a `constraints.txt` file that captures the exact versions of all dependencies. This guarantees that your Python wheels are built with the same dependencies every time, eliminating potential inconsistencies. Here’s an example of a [`constraints.txt`](https://github.com/fmind/mlops-python-package/blob/v4.1.0/constraints.txt) file for the project: ```text # This file was autogenerated by uv via the following command: # uv pip compile pyproject.toml --generate-hashes --output-file=constraints.txt alembic==1.14.1 \ --hash=sha256:1acdd7a3a478e208b0503cd73614d5e4c6efafa4e73518bb60e4f2846a37b1c5 \ --hash=sha256:496e888245a53adf1498fcab31713a469c65836f8de76e01399aa1c3e90dd213 # via mlflow annotated-types==0.7.0 \ --hash=sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53 \ --hash=sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89 # via pydantic antlr4-python3-runtime==4.9.3 \ --hash=sha256:f224469b4168294902bb1efa80a8bf7855f24c99aef99cbefc1bcd3cce77881b # via omegaconf blinker==1.9.0 \ --hash=sha256:b4ce2265a7abece45e7cc896e98dbebe6cead56bcf805a3d23136d145f5445bf \ --hash=sha256:ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc # via flask ``` [And the Just task to generate and install the constraints.txt file](https://github.com/fmind/mlops-python-package/blob/v4.1.0/tasks/package.just): ```justfile # run package tasks [group('package')] package: package-build # build package constraints [group('package')] package-constraints constraints="constraints.txt": uv pip compile pyproject.toml --generate-hashes --output-file={{constraints}} # build python package [group('package')] package-build constraints="constraints.txt": clean-build package-constraints uv build --build-constraint={{constraints}} --require-hashes --wheel ``` ### Upgrade Today and Experience the Difference! Ready to take your MLOps workflows to the next level? Upgrade to [**MLOps Python Package**](https://github.com/fmind/mlops-python-package) [v4.1.0](https://github.com/fmind/mlops-python-package/releases) and unlock the power of enhanced automation and reproducibility. To make these improvements even more accessible, we’ve also updated the [**Cookiecutter MLOps Package**](https://github.com/fmind/cookiecutter-mlops-package). This template provides a pre-configured project structure and CI/CD pipeline, allowing you to quickly bootstrap new MLOps projects with all the latest features. You can generate a new project with the updated template by running: ```bash pip install cookiecutter cookiecutter gh:fmind/cookiecutter-mlops-package ``` In the coming days, we will also update the [**MLOps Coding Course**](https://github.com/MLOps-Courses/mlops-coding-course) and the [**MLOps Coding Assistant**](https://mlops-coding-assistant.fmind.dev/). Follow me on [X](https://x.com/fmind_dev), [GitHub](https://github.com/fmind), [Medium](https://fmind.medium.com/), [LinkedIn](https://www.linkedin.com/in/fmind-dev/) to stay tuned! ![Photo by Lenny Kuhne on Unsplash](/static/img/articles/more-automation-more-reproducibility-mlops-python-package-v4-1-0/02.webp) Photo by [Lenny Kuhne](https://unsplash.com/@lennykuhne?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [Meet Kate: Your AI-Powered, Live Multimodal Website Assistant 🤖](https://www.fmind.dev/articles/meet-kate-your-ai-powered-live-multimodal-website-assistant/) Imagine a world where you can interact with websites using just your voice, having a conversation with an AI assistant that understands your needs and retrieves the information you’re looking for. That’s the world Kate is building. ![Meet Kate: Your AI-Powered, Live Multimodal Website Assistant 🤖](/static/img/articles/meet-kate-your-ai-powered-live-multimodal-website-assistant/cover.webp) Kate is an [open-source](https://github.com/fmind/kate?tab=MIT-1-ov-file#readme) (MIT), cutting-edge [multimodal](https://cloud.google.com/use-cases/multimodal-ai?hl=en) [live assistant](https://ai.google.dev/gemini-api/docs/multimodal-live) that leverages the power of [Gemini 2.0](https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/) and [Vertex AI Search](https://cloud.google.com/enterprise-search?hl=en) to provide a seamless and engaging user experience. She listens to your questions, understands your intent, and responds with relevant information from the website you’re browsing. ### How Kate Works ⚙️ Kate’s magic lies in her ability to combine several powerful technologies: ![Architecture of Kate: A Live Multimodal Website Assistant](/static/img/articles/meet-kate-your-ai-powered-live-multimodal-website-assistant/02.webp) Architecture of Kate: A Live Multimodal Website Assistant - **Multimodal Interaction:** Kate uses [Gemini 2.0’s live multimodal capabilities](https://ai.google.dev/gemini-api/docs/multimodal-live) to process your voice input, generate natural language responses, and even potentially incorporate visual elements like talking animations. This creates a more natural and engaging interaction compared to traditional text-based interfaces. - **Real-Time Communication:** Built on the [pipecat framework](https://github.com/pipecat-ai/pipecat), Kate integrates with platforms like [Daily.co](https://dashboard.daily.co/), allowing you to interact with her during live meetings or calls. Imagine getting instant answers to your questions without interrupting the flow of the conversation. - **Website Search:** Kate utilizes [Vertex AI Search](https://cloud.google.com/enterprise-search?hl=en) to accurately search the website’s content and provide you with precise answers to your queries. This ensures you get the most relevant information quickly and efficiently. ### The Potential of Live Assistants 🔮 Kate is a glimpse into the future of human-computer interaction. Live assistants like her have the potential to: - **Make websites more accessible:** People with disabilities or those who prefer voice interaction can easily access and navigate websites. - **Enhance productivity:** Quickly find information without typing or clicking through multiple pages. - **Personalize the browsing experience:** Kate can learn your preferences and provide tailored recommendations. - **Revolutionize customer service:** Imagine getting instant support from a knowledgeable AI assistant while browsing a website. - **Transform education:** Students can interact with educational materials in a more engaging and interactive way. ### Kate in Action: Real-World Examples 🌍 - **E-commerce:** Kate can help you find the perfect product, answer questions about shipping and returns, and even provide personalized recommendations based on your browsing history. - **Healthcare:** Kate can assist patients in finding information about their conditions, scheduling appointments, and accessing medical records. - **Finance:** Kate can help you manage your finances, track your investments, and get answers to your financial questions. Check out this playlist for a demo of Kate on [Open Textbook Library](https://open.umn.edu/opentextbooks): # An error occurred. Unable to execute JavaScript. YouTube Playlist: [https://www.youtube.com/playlist?list=PLPCnNL6Y2PbTzUxmsFICoQj0rx_PmVnk-](https://www.youtube.com/playlist?list=PLPCnNL6Y2PbTzUxmsFICoQj0rx_PmVnk-) ### Lessons Learned from Building Kate 🎓 Building Kate was an exciting journey, but it also came with its challenges: 1. **Building Live Multimodal Assistance is Hard:** Dealing with audio/video codecs, live transmissions, and browser permissions can be tricky and requires advanced knowledge of computers and protocols. 2. **Live Interaction is Dynamic:** Conversations can be interrupted at any moment, requiring the assistant to adapt and maintain context, and the developer to work in an asynchronous paradigm. 3. **Toolkits Need to Improve:** While building Kate was possible, combining all the necessary tools required perseverance and custom development. Hopefully, frameworks will be released to improve developer experience. 4. **The Magic of Live Interaction:** Interacting with Kate feels incredibly natural and removes the friction of typing, making the experience truly mesmerizing. There is a high potential for organizations willing to invest in this new way to interact with computers. ### Appreciating the Progress of Generative AI 🚀 Just a couple of years ago, building a live assistant like Kate seemed like a distant dream. Today, thanks to the rapid advancements in generative AI, it’s a reality. Kate is a testament to the progress we’ve made and a reminder that we’re still just scratching the surface of what’s possible. [**Kate is open source and available on GitHub**](https://github.com/fmind/kate) **.** If you’re interested in exploring the future of human-computer interaction, I encourage you to check out the project and contribute to its development. Feel free to reach me on [LinkedIn](https://www.linkedin.com/in/fmind-dev/) or my website: [https://www.fmind.dev/](https://www.fmind.dev/) if you want to build a new solution. ![Photo by Andy Kelly on Unsplash](/static/img/articles/meet-kate-your-ai-powered-live-multimodal-website-assistant/03.webp) Photo by [Andy Kelly](https://unsplash.com/@askkell?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [Lessons Learned from the Gemini Long Context Kaggle Competition 🧠](https://www.fmind.dev/articles/lessons-learned-from-the-gemini-long-context-kaggle-competition/) The world of Generative AI is constantly evolving, and one of the most exciting frontiers is the expansion of model's [long context window](https://cloud.google.com/vertex-ai/generative-ai/docs/long-context). Think of a context window as an AI model’s short-term memory — how much information it can hold and process at once. For years, [this has been a major limitation](https://www.perplexity.ai/page/context-window-limitations-of-FKpx7M_ITz2rKXLFG1kNiQ). Traditional models could only juggle a few thousand “tokens” (pieces of words), making it hard for them to understand complex documents, long conversations, or intricate codebases. ![Photo by Greg Rakozy on Unsplash](/static/img/articles/lessons-learned-from-the-gemini-long-context-kaggle-competition/cover.webp) Photo by [Greg Rakozy](https://unsplash.com/@grakozy?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) Then came [Google’s Gemini 1.5](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/), boasting a game-changing context window of up to **2 million tokens**. That’s like remembering 16 average novels! 🤯 This breakthrough opens up a world of possibilities, and the recent “[Gemini Long Context](https://www.kaggle.com/competitions/gemini-long-context)” Kaggle competition was a chance to explore them. I decided to jump in, and the experience was a fascinating deep dive into the potential — and the current limitations — of this cutting-edge technology. ### The Challenge: Pushing AI’s Memory to the Limit 🏋️‍♀️ The competition’s goal was simple but ambitious: create a compelling demonstration of Gemini 1.5’s long context capabilities. We were encouraged to think outside the box, showcasing use cases that would be impossible with older, smaller-context models. The Deepmind team had already provided some [impressive demos](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/#context-window). For instance, Gemini 1.5 was able to create documentation by seeing an entire code base and was able to answer questions correctly after “watching” the movie Sherlock JR. ![https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024](/static/img/articles/lessons-learned-from-the-gemini-long-context-kaggle-competition/02.webp) [https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024) ### My Approach: Building an Interactive Open Textbook Experience 📚 I decided to focus on a problem close to my heart: education. ❤ Traditional textbooks are valuable resources, packed with expert knowledge and detailed examples. But they have inherent limitations: - **Lack of Interaction:** You can’t ask a textbook a question. - **No Personalization:** The textbook doesn’t adapt to your learning style or specific needs. - **Difficult to Combine:** Mixing and matching content from multiple textbooks is cumbersome. My idea was to use Gemini 1.5’s massive context window to create an “interactive textbook” experience. Imagine being able to chat with a textbook, ask clarifying questions, and receive personalized explanations — all grounded in the textbook’s content. The goal was to combine strengths of the textbook and [Gemini Long Context](https://www.kaggle.com/competitions/gemini-long-context). ![So many textbooks, so much knowledge scattered around …](/static/img/articles/lessons-learned-from-the-gemini-long-context-kaggle-competition/03.webp) So many textbooks, so much knowledge scattered around … ### The Process: From PDFs to Personalized Learning 📄➡️🎓 My [competition code](https://www.kaggle.com/datasets/freaxmind/open-textbook-library/) followed these key steps: 1. **Dataset Selection:** I leveraged the [Open Textbook Library](https://open.umn.edu/opentextbooks), a fantastic resource with over 1,500 freely licensed textbooks. This provided a rich and diverse knowledge base. 2. **Retrieval:** I built a retrieval system using Gemini itself. Given a user’s learning goal (e.g., “I want to learn about Mathematics and Computer Science with Python”), the model would identify the most relevant textbooks from the library. 3. **Text Extraction:** I used pypdf to extract the text content from the selected textbooks. This is where the long context window really shines — I could feed entire textbooks directly into Gemini. 4. **Assistant Creation:** I crafted a system instruction for Gemini, turning it into a “friendly and helpful learning assistant”. This instruction emphasized personalization, grounding responses in the textbook content, and maintaining a pedagogic tone. 5. **Context Caching:** A crucial optimization! Gemini allows you to [cache context](https://ai.google.dev/gemini-api/docs/caching?lang=python), meaning the model doesn’t have to re-process the entire textbook for every interaction. This significantly reduces costs and improves response time. ![Example of textbooks available in the Open Textbook Library Dataset](/static/img/articles/lessons-learned-from-the-gemini-long-context-kaggle-competition/04.webp) Example of textbooks available in the [Open Textbook Library Dataset](https://www.kaggle.com/datasets/freaxmind/open-textbook-library/) ### The Good: Impressive Results and Personalized Learning ✅ The results were, in many cases, quite impressive. Here are some key wins: - **Improved Accuracy and Relevance:** The assistant, with access to the full textbook content, consistently provided more detailed and relevant answers compared to a baseline Gemini model without that context. For example, when asked about Newton’s method for finding square roots, the assistant provided a focused Python implementation, exactly as requested, while the baseline gave a more general explanation. - **Personalization in Action:** The assistant tailored its responses to the user’s specific request. It could explain complex concepts in simpler terms, provide relevant examples, and even generate exercises based on the textbook material. - **Combining Knowledge:** The system could seamlessly integrate information from multiple textbooks, creating a cross-topic learning experience. For instance, when asked about leadership in a multicultural context, the assistant drew insights from multiple relevant textbooks. - **Cost-Effectiveness with Caching:** Context caching was a game-changer. _It reduced request costs by a factor of 4_, making the system much more practical for real-world use. **All in all, The assistant has positive gains on 10 out of 11 cases.** ![Open Textbooks with Gemini Long Context — Slide 7](/static/img/articles/lessons-learned-from-the-gemini-long-context-kaggle-competition/05.webp) [Open Textbooks with Gemini Long Context — Slide 7](https://docs.google.com/presentation/d/1H7Kq8Ur3f76mknLcugR0ZaBFqtba6foLQS_i7KG_4Sc/edit#slide=id.g315f80125e3_0_10) ![Open Textbooks with Gemini Long Context — Slide 7](/static/img/articles/lessons-learned-from-the-gemini-long-context-kaggle-competition/06.webp) [Open Textbooks with Gemini Long Context — Slide 7](https://docs.google.com/presentation/d/1H7Kq8Ur3f76mknLcugR0ZaBFqtba6foLQS_i7KG_4Sc/edit#slide=id.g315f80125e3_1_28) ### The Not-So-Good: Execution Time and API Quirks ❌ The journey wasn’t without its bumps. Here’s where I encountered the limitations: - **Execution Time:** While caching helped, interacting with the assistant was still significantly slower than using the baseline model — around 25 seconds per request versus 5 seconds. This is a major hurdle for creating a truly fluid and engaging learning experience. - **File API Challenges:** Gemini offers a File API for uploading documents directly. However, I found it unreliable with very large files (like some of the textbooks) and surprisingly slow, even with caching. Extracting text and feeding it directly to the model proved more reliable, but this highlights a need for improvements in the API’s handling of large documents. ### Lessons Learned: The Future of AI-Powered Learning 💡 This competition was a valuable learning experience. Here are my key takeaways: - **Long Context is a Game-Changer:** The ability to process vast amounts of information opens up exciting possibilities for AI applications, particularly in education. Imagine AI tutors with encyclopedic knowledge, personalized learning paths tailored to individual student needs, and seamless integration of diverse learning resources. - **Performance Matters:** While long context is powerful, execution time is critical for user experience. Optimizations like caching are essential, but further improvements in model speed are needed to make these applications truly interactive. - **Beyond RAG:** The competition highlighted that with long context windows, traditional techniques like Retrieval-Augmented Generation (RAG) might become less critical. Directly feeding relevant context to the model can be surprisingly effective. While challenges remain, the potential of long context models like Gemini 1.5 and the recently announced [Gemini 2.0](https://blog.google/technology/google-deepmind/gemini-model-updates-february-2025/) is undeniable. I’m excited to see how this technology evolves and how it will transform the way we learn. **Links**: - [Kaggle Competition: Google — Gemini Long Context](https://www.kaggle.com/competitions/gemini-long-context/overview) - [Kaggle Competition: Open Textbook Library Dataset](https://www.kaggle.com/datasets/freaxmind/open-textbook-library/) - [Kaggle Competition: Winners Announcement](https://www.kaggle.com/competitions/gemini-long-context/discussion/552419) - [**Kaggle Competition: Competition Code**](https://www.kaggle.com/code/freaxmind/open-textbooks-with-gemini-long-context) - [**Kaggle Competition: Final Slides**](https://docs.google.com/presentation/d/1H7Kq8Ur3f76mknLcugR0ZaBFqtba6foLQS_i7KG_4Sc/edit#slide=id.p) ![Photo by Urban Vintage on Unsplash](/static/img/articles/lessons-learned-from-the-gemini-long-context-kaggle-competition/07.webp) Photo by [Urban Vintage](https://unsplash.com/@urban_vintage?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [Stop Building Rigid AI/ML Pipelines: Embrace Reusable Components for Flexible MLOps](https://www.fmind.dev/articles/stop-building-rigid-ai-ml-pipelines-embrace-reusable-components-for-flexible-mlops/) Let’s be honest, we’ve all been there. You start an MLOps project with the best intentions: [crafting a beautiful pipeline that combines several steps](https://mlops-coding-course.fmind.dev/2.%20Prototyping/2.5.%20Modeling.html#what-are-pipelines). Data ingestion, preprocessing, model training, evaluation, deployment — it all flows seamlessly. At first. But then, reality hits. A new data source emerges. You need to experiment with different preprocessing techniques. Some steps must be reused across several projects. Suddenly, your elegant monolithic pipeline resembles a tangled mess of spaghetti code, [complecting what should be executed (the code) and how it should be executed (the pipeline)](https://www.youtube.com/watch?v=SxdOUGdseq4). The problem? Traditional, monolithic MLOps pipelines are inherently **rigid**. They couple your individual MLOps steps tightly, making them difficult to modify, reuse, or even understand in the long run. This inflexibility slows down iteration, hinders experimentation, and ultimately, increases the risk of your AI/ML project. ![Photo by SELİM ARDA ERYILMAZ on Unsplash](/static/img/articles/stop-building-rigid-ai-ml-pipelines-embrace-reusable-components-for-flexible-mlops/cover.webp) Photo by [SELİM ARDA ERYILMAZ](https://unsplash.com/@selimarda6006?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) **The Solution: Reusable Artifacts and DAGs** What if, instead of a rigid pipeline, we could build our MLOps projects from modular, reusable building blocks? This is where the concept of **artifact-driven MLOps** comes in. Think of it like this: Instead of baking a single, giant cake (your pipeline), you’re now in a professional kitchen, preparing individual ingredients (your reusable artifacts). You have your perfectly sifted flour (data preprocessing), your expertly tempered chocolate (model training), and your precisely whipped cream (model evaluation). Each ingredient is prepared separately, stored in its own container, and ready to be combined in various ways. **Breaking Down the Artifact-Driven Approach** ![Stop Building Rigid AI/ML Pipelines: Embrace Reusable Components for Flexible MLOps](/static/img/articles/stop-building-rigid-ai-ml-pipelines-embrace-reusable-components-for-flexible-mlops/02.webp) The core idea is to encapsulate each step of your MLOps process into independent, reusable artifacts. These artifacts typically take the form of: - [**Python Packages**](https://realpython.com/python-wheels/) **:** Containing core logic for tasks like data preprocessing, feature engineering, or model training, where a [package is divided into jobs](https://github.com/fmind/mlops-python-package/tree/main/src/bikes/jobs) and exposed through entrypoint (e.g., [project scripts](https://github.com/fmind/mlops-python-package/blob/f16d240dabf0c16012a7a2adce5d76428a25624c/pyproject.toml#L47)). - [**Docker Images**](https://docs.docker.com/reference/dockerfile/) **:** Providing isolated, reproducible environments for running your code, packaged with their dependencies in a [Dockerfile](https://github.com/fmind/mlops-python-package/blob/main/Dockerfile). - [**Config Files**](https://mlops-coding-course.fmind.dev/3.%20Productionizing/3.4.%20Configurations.html) **:** Controlling the behavior of your code without requiring code changes (e.g., model hyperparameters, input/output files, …). ![Stop Building Rigid AI/ML Pipelines: Embrace Reusable Components for Flexible MLOps](/static/img/articles/stop-building-rigid-ai-ml-pipelines-embrace-reusable-components-for-flexible-mlops/03.webp) These artifacts are stored in a **code repository** like [GitHub](https://github.com/) and built by a [**CI/CD**](https://en.wikipedia.org/wiki/CI/CD) like [GitHub Actions](https://github.com/features/actions) on [release](https://github.com/fmind/mlops-python-package/releases). Here’s how it looks in practice: 1. **Modularize Your Codebase:** Each [Python package](https://github.com/fmind/mlops-python-package) contains logically grouped code, exposes functions, classes, or even CLI commands via entry points, and has [its own dependencies](https://github.com/fmind/mlops-python-package/blob/main/pyproject.toml). 2. **Build and Version Artifacts:** Use [CI/CD pipelines](https://github.com/fmind/mlops-python-package/blob/v3.0.0/.github/workflows/publish.yml) to automatically build, test, and version your Python packages and Docker images upon code changes. **From Artifacts to Action: Orchestration with DAGs** ![https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html](/static/img/articles/stop-building-rigid-ai-ml-pipelines-embrace-reusable-components-for-flexible-mlops/04.webp) [https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html) Now that we have our reusable artifacts, how do we combine them into a meaningful workflow? This is where [**Directed Acyclic Graphs (DAGs)**](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html) come into play. A DAG, often managed by orchestration tools like [**Apache Airflow**](https://airflow.apache.org/), [Dagster](https://dagster.io/), and [Prefect](https://www.prefect.io/), allows you to define a workflow as a series of interconnected tasks, each loading and executing artifacts. **Visualize the Power of DAGs:** Let’s look how we can assemble our artifacts into distinct workflows. ![Stop Building Rigid AI/ML Pipelines: Embrace Reusable Components for Flexible MLOps](/static/img/articles/stop-building-rigid-ai-ml-pipelines-embrace-reusable-components-for-flexible-mlops/05.webp) The training DAG will **prepare the data** (yellow squares), **tune/train the model** (orange and red squares), and **promote the model** (purple squares) if the new model outperforms the previous version. The DAG can parallelize the step execution (e.g., by country or product category) to improve performance, and adapt the resources allocated to each step (e.g., big countries get more powerful compute instances). ![Stop Building Rigid AI/ML Pipelines: Embrace Reusable Components for Flexible MLOps](/static/img/articles/stop-building-rigid-ai-ml-pipelines-embrace-reusable-components-for-flexible-mlops/06.webp) The Inference DAG can use the same software artifacts as training DAG. In this case, the data processing package is reusing to **process data**, and the machine learning package can generate predictions through **batch inference**. As input, this step takes the production model stored in a model registry like [MLflow](https://mlflow.org/). Notice the advantages of this approach: - **Flexibility:** You can easily swap out components. Need to experiment with a different preprocessing technique? Simply create a new artifact and plug it into the DAG. You don’t need to repeat the data transformation and preparation in the Inference DAG, since it is the same as the Training DAG. - **Reusability:** The same artifact (e.g., a Docker image containing your model training code) can be used in multiple DAGs or even across different projects. - **Extensibility**: If you need to handle a new task (e.g., new country or product category, …), you just add a new step in the DAG and provide a configuration file to change the behavior of the code. **Applying Functional Programming Paradigm in MLOps** The artifact-driven approach we’ve been discussing shares a strong conceptual link with [**functional programming**](https://en.wikipedia.org/wiki/Functional_programming). In functional programming, we strive to build software from pure functions — self-contained units of code that take input and produce output without side effects. These functions are composable, meaning they can be combined in various ways to create more complex behavior. This is very similar to how we’re using artifacts in MLOps. Each artifact (Python package, Docker image with its entry points, config files) can be viewed as a functional unit. It encapsulates a specific task (the “what”) and is independent of the execution context (the “how”). The DAG then acts as a higher-order function, orchestrating the execution of these functional units in a specific order, defining the overall workflow. **Conclusion: Build for the Future** Embracing reusable artifacts and DAGs is a paradigm shift in MLOps. It’s about moving away from rigid, monolithic pipelines towards a more modular, flexible, and scalable approach. While it may require a slight shift in mindset and tooling, the long-term benefits are undeniable. You’ll be able to iterate faster, experiment more freely, and ultimately, deliver better AI/ML solutions. So, the next time you start an [MLOps project](https://github.com/fmind/cookiecutter-mlops-package), remember the professional kitchen analogy. Stock your pantry with high-quality artifacts, and use DAGs to orchestrate your culinary (or machine learning) masterpieces. ![Photo by Calum Lewis on Unsplash](/static/img/articles/stop-building-rigid-ai-ml-pipelines-embrace-reusable-components-for-flexible-mlops/07.webp) Photo by [Calum Lewis](https://unsplash.com/@calumlewis?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [Poetry Was Good, Uv Is Better: An MLOps Migration Story](https://www.fmind.dev/articles/poetry-was-good-uv-is-better-an-mlops-migration-story/) As an MLOps Engineer, I’m always on the lookout for tools and technologies that can streamline workflows, boost performance, and enhance the maintainability of MLOps projects. Recently, I made a significant shift in the [MLOps Python Package ](https://github.com/fmind/mlops-python-package)— a repository dedicated to sharing best practices for MLOps in Python. I transitioned from [Poetry](https://python-poetry.org/), a popular dependency management and packaging tool, to [Uv](https://docs.astral.sh/uv/), a newer, blazing-fast alternative. ![Photo by Chris Briggs on Unsplash](/static/img/articles/poetry-was-good-uv-is-better-an-mlops-migration-story/cover.webp) Photo by [Chris Briggs](https://unsplash.com/@cgbriggs19?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) This wasn’t a decision I took lightly. Poetry has served my teammates and me well on many MLOps projects, but Uv’s promise of enhanced speed and stricter adherence to Python Enhancement Proposals (PEPs) piqued our interest. **After thoroughly testing it out, we were convinced. The transition is a resounding success, and we’re excited to share our experience and why we believe Uv might be the right choice for your AI/ML projects too.** ### The Need for Speed (and PEPs) The “[MLOps Python Package](https://github.com/fmind/mlops-python-package)” is a side project, but it reflects the best practices defined in the organizations I’m working with. As our MLOps projects grew, we started noticing that dependency resolution and installation with Poetry were becoming increasingly sluggish. **This was particularly noticeable in our CI/CD pipelines with runs often exceeding 25 minutes**. Moreover, while Poetry is a powerful tool, it sometimes requires its own configurations, deviating from [standard PEP practices](https://peps.python.org/pep-0621/). Enter [Uv](https://docs.astral.sh/uv/). This new tool, developed by [Astral](https://astral.sh/), promised to be a drop-in replacement for `pip`, `pip-tools`, `pipx`, `poetry`, `pyenv`, `twine`, and `virtualenv`, boasting significant speed improvements thanks to its Rust foundation. It also claimed [strict compliance with PEPs](https://peps.python.org/pep-0621/), which meant a more standardized and potentially less complex project structure. ### The Transition: Smooth Sailing with Uv Migrating from Poetry to Uv was surprisingly straightforward. The most interesting part was adapting our [pyproject.toml](https://github.com/fmind/mlops-python-package/blob/main/pyproject.toml), [CI/CD workflows (GitHub Actions in our case)](https://github.com/fmind/mlops-python-package/tree/main/.github/workflows), [development tasks](https://github.com/fmind/mlops-python-package/tree/v3.0.0/tasks) and [Dockerfile](https://github.com/fmind/mlops-python-package/blob/main/Dockerfile). #### A Cleaner, More Compliant `pyproject.toml` The move to Uv brought welcome changes to our [`pyproject.toml`](https://github.com/fmind/mlops-python-package/blob/main/pyproject.toml) file. We were able to remove the Poetry-specific sections, like `[tool.poetry]` and its associated configurations: ```toml [tool.poetry] name = "bikes" version = "2.0.0" description = "Predict the number of bikes available." ... [tool.poetry.dependencies] python = "^3.12" loguru = "^0.7.2" ... [tool.poetry.group.checks.dependencies] bandit = "^1.7.9" ... ``` Now we could rely solely on [standards-compliant sections](https://peps.python.org/pep-0621/) like `[project]` and `[project.optional-dependencies]`: ```toml [project] name = "bikes" version = "3.0.0" description = "Predict the number of bikes available." ... dependencies = [ "loguru>=0.7.2", ... ] ... [dependency-groups] checks = [ "bandit>=1.8.0", ... ] ``` **GitHub Actions: Setup** Previously, our [setup action](https://github.com/fmind/mlops-python-package/blob/v3.0.0/.github/actions/setup/action.yml) involved installing `pipx`, `invoke`, and `poetry`: ```yaml - run: pipx install invoke poetry shell: bash - uses: actions/setup-python@v5 with: python-version: 3.12 cache: poetry ``` With Uv we simply need to adopt a new GitHub Action: ```yaml - name: Install uv uses: astral-sh/setup-uv@v4 with: enable-cache: true - name: Setup Python uses: actions/setup-python@v5 with: python-version-file: .python-version ``` The [`astral-sh/setup-uv`](https://github.com/astral-sh/setup-uv) action made installing Uv incredibly simple, and we could rely on the action to specify the python version in the `.python-version` file. **GitHub Actions: Checks and Publishing** Our [check workflow](https://github.com/fmind/mlops-python-package/blob/v3.0.0/.github/workflows/check.yml) also saw few changes. Instead of running a single checker step with Poetry: ```yaml - run: poetry install --with checks - run: poetry run invoke checks ``` We now have separate checkers with Uv to improve speed and debugging: ```yaml - run: uv sync --group=checks - run: uv run invoke checks.format - run: uv run invoke checks.type - run: uv run invoke checks.code - run: uv run invoke checks.security - run: uv run invoke checks.coverage ``` Similarly, in our [publishing workflow](https://github.com/fmind/mlops-python-package/blob/v3.0.0/.github/workflows/publish.yml), `poetry install --with docs` and `poetry run invoke docs` became `uv sync --group=docs` and `uv run invoke docs`, respectively. These changes highlight Uv's ability to seamlessly integrate into existing workflows while being faster than Poetry. **Development Tasks** Our pyinvoke [tasks](https://github.com/fmind/mlops-python-package/tree/v3.0.0/tasks), defined in the `tasks/` directory, also required only minor changes. For instance, our [installation](https://github.com/fmind/mlops-python-package/blob/v3.0.0/tasks/installs.py) and [format](https://github.com/fmind/mlops-python-package/blob/v3.0.0/tasks/formats.py) task went from: ```python @task def poetry(ctx: Context) -> None: """Install poetry packages.""" ctx.run("poetry install") @task def format(ctx: Context) -> None: """Check the formats with ruff.""" ctx.run("poetry run ruff format --check src/ tasks/ tests/") ``` With Uv it becomes: ```python @task def uv(ctx: Context) -> None: """Install uv packages.""" ctx.run("uv sync --all-groups") @task def format(ctx: Context) -> None: """Check the formats with ruff.""" ctx.run("uv run ruff format --check src/ tasks/ tests/") ``` #### Slimmer, Faster Docker Builds with Uv The benefits of Uv extends into our Docker builds as well. Our initial `Dockerfile` relied on installing the package wheel using `pip`: ```dockerfile FROM python:3.12 COPY dist/*.whl . RUN pip install *.whl CMD ["bikes", "--help"] ``` This was functional but not optimal. With Uv, we were able to switch to a more streamlined approach, leveraging Uv’s speed within the Docker build process. We also switched to the official `uv` image maintained by Astral: ```dockerfile FROM ghcr.io/astral-sh/uv:python3.12-bookworm COPY dist/*.whl . RUN uv pip install --system *.whl CMD ["bikes", "--help"] ``` The key change here is using `uv pip install --system *.whl` instead of `pip install *.whl`. By using the `--system` flag, we ensure that the package and its dependencies are installed into the system's site packages directory, making for a cleaner environment. This, combined with Uv's installation speed, resulted in significantly faster Docker builds. ### The Results: Speed, Simplicity, and Standards The impact of switching to Uv was immediately noticeable. **Our CI/CD pipelines became significantly faster, thanks to Uv’s rapid dependency resolution and installation.** Locally, development felt snappier too. But it wasn’t just about speed. **Uv’s strict adherence to PEP standards meant we could simplify our project configuration**. We no longer needed a separate `poetry.toml` file. Everything was managed within the standard `pyproject.toml`, making our [project more aligned with the broader Python ecosystem](https://peps.python.org/pep-0621/). ### Why You Might Want to Consider Uv Our experience with [Uv](https://docs.astral.sh/uv/) has been overwhelmingly positive. If you’re working on an MLOps project (or any Python project, for that matter) and value speed, simplicity, and adherence to standards, we highly recommend giving Uv a try. Here’s a quick recap of the benefits we observed: - **Blazing Fast:** Uv’s Rust-based architecture delivers significant speed improvements over Poetry. - **PEP Compliance:** Strict adherence to PEPs means a more standardized and potentially less complex project structure. - **Seamless Integration:** Uv integrates smoothly into existing workflows, often simplifying them in the process. - **Drop-in Replacement:** Uv can replace `pip`, `pip-tools`, `pipx`, `poetry`, `pyenv`, `twine`, `virtualenv`, and more, making it a versatile tool for various project needs. ### Conclusion: Embracing the Future of Python Dependency Management The transition from Poetry to Uv in our “[MLOps Python Package](https://github.com/fmind/mlops-python-package)” project has been a rewarding experience. It’s a testament to the evolving landscape of Python tooling and the benefits of embracing new technologies that align with our goals of efficiency, maintainability, and community standards. We encourage you to explore Uv and see if it’s the right fit for your projects. The future of Python dependency management is here, and it’s faster and more standardized than ever before! ![Photo by Harley-Davidson on Unsplash](/static/img/articles/poetry-was-good-uv-is-better-an-mlops-migration-story/02.webp) Photo by [Harley-Davidson](https://unsplash.com/@harleydavidson?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [Taming Python Installation with Pyenv for AI/ML and MLOps Projects](https://www.fmind.dev/articles/taming-python-installation-with-pyenv-for-ai-ml-and-mlops-projects/) [Python](https://www.python.org/) is the beloved language of data scientists, machine learning engineers, and anyone dabbling in the fascinating world of AI. But with its ever-evolving landscape of versions and dependencies, managing your Python environment can quickly turn into a chaotic nightmare. Fear not, fellow data wizards! This blog post introduces you to [**pyenv**](https://github.com/pyenv/pyenv), your secret weapon for taming the Python beast and ensuring a smooth, reproducible, and headache-free development experience. ![https://m.xkcd.com/1987/](/static/img/articles/taming-python-installation-with-pyenv-for-ai-ml-and-mlops-projects/cover.webp) [https://m.xkcd.com/1987/](https://m.xkcd.com/1987/) ### The Python Version Problem AI/ML practitioners are juggling between multiple projects. One requires [Python 3.9](https://www.python.org/downloads/release/python-390/) for compatibility with a crucial library, while another demands the shiny new features of Python [3.13](https://docs.python.org/3.13/whatsnew/3.13.html). Switching between these versions manually can be a recipe for disaster, leading to broken dependencies and frustrating debugging sessions. System-wide Python installations become a minefield, and [virtual environments](https://docs.python.org/3/library/venv.html), while helpful, can still be cumbersome to manage across multiple projects. ### Enter Pyenv: The Python Version Manager [pyenv](https://github.com/pyenv/pyenv) is a lightweight command-line tool that allows you to effortlessly install and switch between multiple Python versions on your system. It acts as a gatekeeper, intercepting Python commands and directing them to the appropriate version based on your project’s needs. This means you can have Python 3.11, 3.12, 3.13, and other versions coexisting peacefully, each neatly tucked away in its own isolated environment. ### Why Choose Pyenv? - **Simplicity:** pyenv’s command-line interface is clean and intuitive, making it easy to learn and use. - **Flexibility:** Install and manage any Python version, from legacy releases to the latest cutting-edge builds. - **Project-Specific Versions:** Specify the exact Python version for each project, ensuring consistent and reproducible results. - **No Root Privileges Required:** Install Python versions locally without needing administrator access, perfect for shared systems or environments where you don’t have root privileges. - **Lightweight and Fast:** pyenv’s minimal footprint ensures it doesn’t bog down your system, providing quick and efficient version switching. ### Choosing the Right Python Version For new AI/ML and MLOps projects, embracing the latest stable Python release is generally recommended. Newer versions often bring performance enhancements, bug fixes, and exciting new features that can boost your productivity. However, compatibility with existing libraries and frameworks is paramount. **Always check the requirements of your production environment before upgrading to ensure a smooth transition**. The official Python website provides [a list of supported versions](https://www.python.org/downloads/), and it’s wise to steer clear of unsupported releases to avoid security vulnerabilities and compatibility issues. ### Installing Pyenv The installation process is straightforward, varying slightly depending on your operating system. The comprehensive instructions on the official [pyenv](https://github.com/pyenv/pyenv) GitHub repository ([https://github.com/pyenv/pyenv#installation](https://www.google.com/url?sa=E&q=https%3A%2F%2Fgithub.com%2Fpyenv%2Fpyenv%23installation)) provide detailed guidance for different platforms. #### Getting Pyenv For macOS, UNIX and Windows+[WSL](https://learn.microsoft.com/en-us/windows/wsl/install) systems, use the automatic installer: ```bash curl https://pyenv.run | bash ``` #### Setting Up Your Shell Environment For bash and zsh shells, add these lines to both ~/.bashrc ~/.zshrc or ~/.profile: ```bash export PYENV_ROOT="$HOME/.pyenv" command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH" eval "$(pyenv init -)" ``` #### Restart Your Shell Finally, restart your shell to update your shell PATH: ```bash exec "$SHELL" ``` **Important: Be sure** [**to install Python build dependencies**](https://github.com/pyenv/pyenv/wiki#suggested-build-environment) **on your system after the installation.** ### Using Pyenv Once installed, using pyenv is a breeze: - **Install a Python Version:**   ```bash pyenv install 3.12.0 ``` - **Set a Global Version (Optional):**   ```bash pyenv global 3.12.0 ``` - **Set a Local Version (Project-Specific):** Create a .python-version file in your project’s root directory containing the desired version (e.g., 3.12.0). pyenv automatically switches to this version when you enter the project directory.   ```text # in .python-version 3.12 ``` - **Check Active Version:**   ```bash pyenv version ``` ### Example Workflow Let’s say you’re starting a new project that requires Python 3.12: ```bash mkdir my-mlops-project cd my-mlops-project pyenv install 3.12 echo "3.12" > .python-version python --version # Verify the correct version is active ``` Now, whenever you work within the my-mlops-project directory, pyenv ensures you’re using the correct Python version. ### Conclusion [pyenv](https://github.com/pyenv/pyenv) is an indispensable tool for any data scientist or MLOps engineer working with Python. It simplifies version management, promotes project isolation, and ensures reproducibility, allowing you to focus on what you do best: crafting innovative AI/ML solutions. So, ditch the Python version headaches and embrace the power of pyenv! Your future self will thank you. _To learn more about best practices for MLOps Coding, check this course:_ [_https://mlops-coding-course.fmind.dev/_](https://mlops-coding-course.fmind.dev/) ![Take care of your Python environment. Photo by Karsten Würth on Unsplash](/static/img/articles/taming-python-installation-with-pyenv-for-ai-ml-and-mlops-projects/02.webp) Take care of your Python environment. Photo by [Karsten Würth](https://unsplash.com/@karsten_wuerth?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [Bromate: Automate Your Browser with Agentic Workflows 🧭](https://www.fmind.dev/articles/bromate-automate-your-browser-with-agentic-workflows/) The AI landscape is constantly evolving, pushing the boundaries of what’s possible with technology. One of the most exciting frontiers is the rise of [**agentic workflows**](https://en.wikipedia.org/wiki/Intelligent_agent), a paradigm shift in automation that should revolutionize the way we interact with software. Imagine a world where you can simply tell your browser what you want to achieve, and it figures out the _how_ on its own, executing complex tasks with minimal human intervention. This is the power of agentic workflows, and [**Bromate**](https://github.com/fmind/bromate), an open-source Python project, is leading the charge in bringing this transformative technology to web browser automation. ![Photo by Jamie Street on Unsplash](/static/img/articles/bromate-automate-your-browser-with-agentic-workflows/cover.webp) Photo by [Jamie Street](https://unsplash.com/@jamie452?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### Decoding Agentic Workflows for Automation 💡 [Agentic workflows](https://en.wikipedia.org/wiki/Software_agent) represent a new level of sophistication in automation. Instead of relying on pre-programmed scripts or rigid rules, they leverage the power of large language models (LLMs) to understand user intent and dynamically generate the necessary steps to achieve a desired outcome. Think of it as having a highly skilled virtual assistant that’s capable of understanding natural language instructions and translating them into precise actions within your browser. This approach offers several key advantages over traditional automation methods: - **Intuitive Interaction** 🔃**:** Agentic workflows bridge the gap between human intent and machine execution, allowing users to interact with software in a more natural and intuitive way. - **Flexibility and Adaptability** ➰**:** LLMs can handle variations in user input and adapt to changes in the environment, making agentic workflows more robust and versatile than rule-based systems. - **Reduced Development Time** ⌛**:** By automating the process of creating automation workflows, agentic workflows free up developers to focus on more complex and strategic tasks. ### Bromate: Your Agentic Browser Automation Companion ⚙️ [**Bromate**](https://github.com/fmind/bromate) is an open-source experiment designed to bring the power of agentic workflows to web browser automation. It provides a framework that allows developers to run scripts capable of understanding user queries and executing actions within a browser environment. **Here’s how Bromate works:** 1. **User Query** ❗**:** You provide Bromate with a natural language query describing the task you want to automate, such as “Fill out the registration form on this website.” 2. **Workflow Generation** ✦**:** Bromate’s LLM analyzes the query and generates a workflow consisting of a series of actions that need to be performed in the browser. 3. **Action Execution** 🔥**:** Bromate’s agent executes the workflow, interacting with the browser using Selenium to perform actions like clicking buttons, filling out forms, and navigating pages. 4. **Feedback Loop** ⟳**:** After each action, Bromate captures the browser’s state (screenshot and page source) and feeds it back to the LLM, allowing it to refine the workflow and make adjustments as needed. [**Bromate Actions**](https://github.com/fmind/bromate/blob/main/src/bromate/actions.py) are defined in Python, like in the two functions below: ```python @declare( schema=agents.Schema( type=agents.Type.OBJECT, properties={ "url": agents.Schema(type=agents.Type.STRING, description="URL of the web page to open") }, required=["url"], ) ) def get(driver: drivers.Driver, config: ActionConfig, url: str) -> agents.Structure: """Open a web page in the browser window.""" driver.get(url=url) # wait loading time.sleep(config.sleep_time) return agents.Structure( name=get.__name__, response={ "title": driver.title, "url": driver.current_url, "page_source": driver.page_source, }, ) @declare( schema=agents.Schema( type=agents.Type.OBJECT, properties={ "css_selector": agents.Schema( type=agents.Type.STRING, description="CSS selector of the element to click on." ), }, required=["css_selector"], ) ) def click(driver: drivers.Driver, config: ActionConfig, css_selector: str) -> agents.Structure: """Click on an element given its CSS selector.""" element = driver.find_element(by=drivers.CSS, value=css_selector) element.click() time.sleep(config.sleep_time) return agents.Structure( name=click.__name__, response={ "title": driver.title, "url": driver.current_url, "page_source": driver.page_source, }, ) ``` ### Gemini: The Brain Behind Bromate’s Agentic Power 🧠 Bromate leverages the power of [**Google’s Gemini models**](https://ai.google.dev/gemini-api), a family of advanced LLMs, to implement its agentic workflows. Gemini’s ability to understand natural language and generate contextually relevant responses makes it an ideal choice for powering Bromate’s core functionality. Here’s the core algorithm showing how Gemini is integrated into Bromate: ```python from bromate import actions, agents, drivers, types def execute( query: str, agent: agents.Agent, driver: drivers.Driver, config: ExecutionConfig, action_config: actions.ActionConfig, agent_functions: list[agents.Function] = actions.AGENT_FUNCTIONS, ) -> Execution: """Execute a query given a config.""" # contents query_content = agents.Content(role=agents.Role.USER.value, parts=[agents.Part(text=query)]) contents = [query_content] # tools agent_tool = agents.Tool(function_declarations=agent_functions) tools = [agent_tool] # steps while True: done = False # response response = agent.generate_content(contents=contents, tools=tools) # parts structures: list[agents.Structure] = [] for i, part in enumerate(response.parts, start=1): if call := part.function_call: name, kwargs = call.name, call.args if name in config.stop_actions: done = True # stop execution if action := getattr(actions, name): try: structure = action(driver=driver, config=action_config, **kwargs) except Exception as error: kwargs_text = ", ".join(f"{key}={val}" for key, val in kwargs.items()) logger.error( f"Error while executing action '{name}' with kwargs '{kwargs_text}': {error}" ) structure = agents.Structure(name=name, response={"error": str(error)}) structures.append(structure) else: raise ValueError(f"Cannot execute action (unknown action name): {name}!") elif part.text: pass else: raise ValueError(f"Cannot handle agent response (unknown part type): {part}!") # output agent_content = agents.Content(role=agents.Role.AGENT.value, parts=response.parts) if done is True: return agent_content contents.append(agent_content) user_input = yield agent_content # input message = user_input or config.default_message returned = [agents.Part(function_response=s) for s in structures] screenshot = agents.Blob(mime_type="image/png", data=driver.get_screenshot_as_png()) user_content = agents.Content( role=agents.Role.USER.value, parts=[ agents.Part(inline_data=screenshot), agents.Part(text=message), ] + returned, # action calls ) contents.append(user_content) ``` ### A Deep Dive into Bromate’s Architecture 🧰 Bromate’s architecture is built on a foundation of powerful tools and libraries: - [**Google’s Generative AI Platform**](https://ai.google.dev/gemini-api) **:** Bromate leverages [Google’s Gemini models](https://deepmind.google/technologies/gemini/) for its LLM capabilities, enabling it to understand user queries and generate workflows. - [**Selenium**](https://selenium-python.readthedocs.io/) **:** This popular [browser automation framework](https://www.testim.io/blog/browser-test-automation/) provides the necessary tools for Bromate’s agent to interact with web browsers. - [**Pydantic**](https://docs.pydantic.dev/latest/) **:** This data validation library ensures that Bromate’s configurations and data structures are well-defined and consistent. - [**Loguru**](https://loguru.readthedocs.io/en/stable/) **:** This logging library provides detailed insights into Bromate’s execution process, facilitating debugging and monitoring. ![External Systems and Modules of the Bromate project](/static/img/articles/bromate-automate-your-browser-with-agentic-workflows/02.webp) External Systems and Modules of the Bromate project ### Getting Started with Bromate 🔋 Bromate is easy to install and use. The [project’s README](https://github.com/fmind/bromate/blob/main/README.md) provides a comprehensive guide on setting up your environment and running your first automated queries. **Here’s a simple example of how to use Bromate:** ```bash # Install Bromate using pip pip install bromate # Export you Gemini API key export GOOGLE_API_KEY=... # Example 1: Subscribe to the MLOps Community Newsletter bromate "Open the https://MLOps.Community website. Click on the 'Join' link. Write the address 'hello@mlops'" ``` # An error occurred. Unable to execute JavaScript. ```bash # Example 2: Summarize the features of the next Python release bromate --interaction.stay_open=False \ --agent.name "gemini-1.5-pro-latest" \ "Go to Python.org. Click on the downloads page. Click on the PEP link for the future Python release. Summarize the release schedule dates." ``` # An error occurred. Unable to execute JavaScript. You can explore Bromate options on the [project README.md file](https://github.com/fmind/bromate) or by typing `bromate -h` in your shell. ### Limitations and Future Directions 🚧 - **Accuracy vs. Speed:** Sending full browser context (screenshots and source code) to the LLM after every action can be slow. Bromate optimizes this by sending source code only when needed (e.g., on page change). - **Instruction Granularity:** Users may need to provide detailed instructions in some cases. Future development will explore techniques like “chain of thought” prompting to enhance the LLM’s ability to plan multi-step actions. - **Action Selection and CSS Selectors:** LLMs can sometimes choose incorrect actions or hallucinate CSS selectors. Bromate uses a retry mechanism with error feedback to improve accuracy. - **File Downloads:** File download support is not yet implemented. Contributions are welcome! We are actively working to address these limitations and expand Bromate’s capabilities. Your feedback and contributions are invaluable to improve the project! ### Embracing the Agentic Future 💪 Agentic workflows are transforming the way we think about automation. [**Bromate**](https://github.com/fmind/bromate) provides a powerful and accessible platform for exploring the potential of this exciting technology in the context of web browser automation. By embracing agentic workflows, developers and businesses can unlock new levels of efficiency, productivity, and innovation. **Join the** [**Bromate community**](https://github.com/fmind/bromate) **and be part of the agentic revolution!** ![Photo by Jacek Dylag on Unsplash](/static/img/articles/bromate-automate-your-browser-with-agentic-workflows/03.webp) Photo by [Jacek Dylag](https://unsplash.com/@dylu?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [MLOps Package Template: Turbocharge the Creation of AI/ML Projects ⚡](https://www.fmind.dev/articles/mlops-package-template-turbocharge-the-creation-of-ai-ml-projects/) ### **MLOps Package Template**: Turbocharge the Creation of AI/ML Projects ⚡ The world of AI/ML is evolving at an electrifying pace, and MLOps has emerged as a cornerstone for translating innovative ideas into production-ready solutions. Yet, setting up a robust MLOps project can feel like navigating a labyrinth of tools, configurations, and best practices. This is where the [**Cookiecutter MLOps Package**](https://github.com/fmind/cookiecutter-mlops-package) steps in, offering a **powerful code template to jumpstart your MLOps journey**, accelerating your development process and ensuring a solid foundation for success. ![Photo by Jeremy Bishop on Unsplash](/static/img/articles/mlops-package-template-turbocharge-the-creation-of-ai-ml-projects/cover.webp) Photo by [Jeremy Bishop](https://unsplash.com/@jeremybishop?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### Why Code Templates Matter 💡 Think of a code template as a blueprint for success. It provides a **standardized structure and pre-configured tools**, eliminating the need for repetitive setup tasks and allowing you to **focus on the core problem you’re trying to solve**. This streamlined approach not only saves valuable time and effort but also promotes consistency and adherence to best practices across multiple projects. ### The Cookiecutter MLOps Package: Built for Versatility ⚙️ The [**Cookiecutter MLOps Package**](https://github.com/fmind/cookiecutter-mlops-package) is designed with a **platform-agnostic philosophy**, recognizing that the fundamental principles of packaging and deployment are applicable across various MLOps environments. Whether you’re working with [Kubernetes](https://kubernetes.io/), [Vertex AI](https://cloud.google.com/vertex-ai), [Databricks](https://www.databricks.com/), [Azure ML](https://azure.microsoft.com/en-us/products/machine-learning), or [AWS SageMaker](https://aws.amazon.com/sagemaker/), the **template provides a common foundation**, empowering you to integrate your code seamlessly into your preferred platform. ### A Powerful Toolkit at Your Fingertips 🧰 The [**Cookiecutter MLOps Package**](https://github.com/fmind/cookiecutter-mlops-package) equips you with an arsenal of tools and features to enhance your MLOps development: - **Streamlined Project Structure**: Say goodbye to chaotic project setups. The template provides [a well-defined directory structure](https://github.com/fmind/cookiecutter-mlops-package/tree/main/%7B%7Bcookiecutter.repository%7D%7D) for your code, tests, documentation, and more. - **Dependency Management with** [**Poetry**](https://python-poetry.org/): Effortlessly manage your Python dependencies and build your package with [Poetry](https://python-poetry.org/), ensuring a consistent and reproducible environment. - **Automated Testing and Quality Checks**: Enjoy a robust testing framework with [Pytest](https://docs.pytest.org/), [Ruff](https://docs.astral.sh/ruff/), [Mypy](https://mypy.readthedocs.io/), [Bandit](https://bandit.readthedocs.io/), and [Coverage](https://coverage.readthedocs.io/), guaranteeing code quality, style, security, and type safety. - **Pre-commit Hooks**: Automatically format and lint your code with [pre-commit hooks](https://pre-commit.com/), enforcing coding standards and preventing regressions. - [**MLflow**](https://mlflow.org/) **Integration**: Seamlessly execute your jobs using [MLflow projects](https://mlflow.org/docs/latest/projects.html), enabling easy experimentation, tracking, and reproducibility. - **Dockerized Deployment**: Build and run your package within a [Docker](https://www.docker.com/) container, ensuring consistency and portability across different environments. - [**PyInvoke**](https://www.pyinvoke.org/) **for Task Automation**: Automate repetitive development tasks with PyInvoke, streamlining your workflow and saving time. - **Comprehensive Documentation**: Generate API documentation with [pdoc](https://pdoc.dev/) and leverage Markdown files for clear usage instructions. - [**GitHub Actions**](https://github.com/features/actions) **for CI/CD**: Set up continuous integration and deployment workflows with GitHub Actions, automating testing, checks, and publishing. ### The MLOps Ecosystem: Course, Package, and Template 🔌 The [**Cookiecutter MLOps Package**](https://github.com/fmind/cookiecutter-mlops-package) is part of a broader ecosystem designed to empower ML practitioners: - [**MLOps Coding Course**](https://mlops-coding-course.fmind.dev/): This comprehensive course dives deep into software development best practices for AI/ML, providing the foundational knowledge to structure and manage MLOps projects effectively. - [**MLOps Python Package**](https://github.com/fmind/mlops-python-package): This companion repository showcases a practical implementation of the concepts and best practices discussed in the course on a Predictive ML project. ### Getting Started with the Cookiecutter MLOps Package🔋 To get started, install [Cookiecutter](https://cookiecutter.readthedocs.io/) and generate your MLOps project: ```bash pip install cookiecutter cookiecutter gh:fmind/cookiecutter-mlops-package ``` You’ll be prompted to provide values for the following variables: ```toml user = "your-github-username" name = "Your Project Name" repository = "your-project-repository" package = "your_project_package" license = "MIT" version = "0.1.0" description = "A brief description of your project" python_version = "3.12" mlflow_version = "2.14.3" ``` Then, initialize a git repository and activate the [GitHub pages workflow](https://pages.github.com/): ```bash cd your-project-repository git init ``` ### Showcasing Automated Tasks ✨ The [**Cookiecutter MLOps Package**](https://github.com/fmind/cookiecutter-mlops-package) empowers you to automate various development tasks using [PyInvoke](https://www.pyinvoke.org/). Here are some examples: [**Install Dependencies**](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/installs.py) **:** [This task](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/installs.py) installs all project dependencies using [Poetry](https://python-poetry.org/) and sets up [pre-commit hooks](https://pre-commit.com/). ```bash invoke installs ``` [**Format Code**](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/formats.py) **:** [This task](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/formats.py) automatically formats your code using [Ruff](https://docs.astral.sh/ruff/), ensuring consistent style. ```bash invoke formats ``` [**Run Tests and Checks**](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/checks.py) **:** [This task](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/checks.py) runs unit tests with [Pytest](https://docs.pytest.org/en/stable/), lints your code with [Ruff](https://docs.astral.sh/ruff/), performs type checks with [Mypy](https://mypy.readthedocs.io/en/stable/index.html), analyzes code security with [Bandit](https://bandit.readthedocs.io/en/latest/), and generates a code coverage report with [Coverage](https://coverage.readthedocs.io/). ```bash invoke checks ``` [**Build Python Package**](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/packages.py) **:** [This task](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/packages.py) builds your Python package as a [wheel file](https://pythonwheels.com/), ready for distribution. ```bash invoke packages ``` [**Run an MLflow Project**](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/projects.py) **:** [This task](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/projects.py) executes your [MLflow project](https://mlflow.org/docs/latest/projects.html), as defined in your [MLproject](https://github.com/fmind/cookiecutter-mlops-package/blob/main/%7B%7Bcookiecutter.repository%7D%7D/MLproject) file. ```bash invoke projects ``` [**Build and Run Docker Image**](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/containers.py) **:** [This task](https://github.com/fmind/cookiecutter-mlops-package/blob/v1.0.0/%7B%7Bcookiecutter.repository%7D%7D/tasks/containers.py) builds your Docker image based on your [Dockerfile](https://github.com/fmind/cookiecutter-mlops-package/blob/main/%7B%7Bcookiecutter.repository%7D%7D/Dockerfile) and runs it in a container. ```bash invoke containers ``` ### The Power of Templates: Embrace Efficiency and Quality 💪 The [**Cookiecutter MLOps Package**](https://github.com/fmind/cookiecutter-mlops-package) is more than just a time-saver; it’s a quality enhancer, ensuring that every project you start adheres to best practices and is built on a solid foundation. By leveraging this template, you can: - **Accelerate Development**: Focus on the unique aspects of your project, not the repetitive setup tasks. - **Enhance Consistency**: Promote uniformity and best practices across all your projects. - **Boost Collaboration**: Create a shared development environment for your team, reducing setup time and confusion. - **Improve Maintainability**: Create structured and well-documented projects that are easier to maintain and update. Embark on your MLOps journey with the [**Cookiecutter MLOps Package**](https://github.com/fmind/cookiecutter-mlops-package) and experience the power of templates to streamline your development process and elevate your AI/ML projects to new heights. ![Photo by Jan Huber on Unsplash](/static/img/articles/mlops-package-template-turbocharge-the-creation-of-ai-ml-projects/02.webp) Photo by [Jan Huber](https://unsplash.com/@jan_huber?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [MLOps Coding Course: Mastering Observability for Reliable ML 💡](https://www.fmind.dev/articles/mlops-coding-course-mastering-observability-for-reliable-ml/) In [the last blog article](https://home.mlops.community/public/blogs/mlops-coding-course-bridging-the-gap-between-data-scientists-and-machine-learning-engineers), we constructed [a robust and production-ready MLOps codebase](https://github.com/fmind/mlops-python-package). But the journey doesn’t end with deployment. The real test begins when your model encounters the dynamic and often unpredictable world of production. That’s where [**Observability**](https://mlops-coding-course.fmind.dev/7.%20Observability/index.html), the focus of [Chapter 7](https://mlops-coding-course.fmind.dev/7.%20Observability/index.html) in the [MLOps Coding Course](https://mlops-coding-course.fmind.dev/), takes center stage. This article dives deep into the essential tools and practices for achieving comprehensive observability in your ML projects. We’ll unravel key concepts, showcase practical code examples from the accompanying [MLOps Python Package](https://github.com/fmind/mLOps-python-package), and explore the benefits of integrating industry-leading solutions like [MLflow](https://mlflow.org/). ![Photo by Elisa Schmidt on Unsplash](/static/img/articles/mlops-coding-course-mastering-observability-for-reliable-ml/cover.webp) Photo by [Elisa Schmidt](https://unsplash.com/@elisasch?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) _Note: The course is also available on the_ [_MLOps Community Learning Platform_](https://learn.mlops.community/courses/languages/mlops-coding-course/) ### Why Observability is Your ML’s Guardian Angel 😇 Deploying a model that initially shines with stellar performance only to witness its accuracy fade over time is a nightmare scenario for any ML engineer. Without observability, you’re left fumbling in the dark, trying to diagnose issues in a black box. Observability empowers you to: - **Preempt Disaster with Proactive Monitoring:** Continuously track crucial metrics like data drift, concept drift, or model performance degradation. Set up alerts to notify you of potential issues _before_ they impact users, allowing for timely interventions. - **Unlock the Secrets of Your Model’s Decision-Making:** Employ explainability techniques to understand feature contributions and identify potential biases. This transparency builds trust with stakeholders and ensures responsible AI practices. - **Optimize for Peak Performance and Efficiency**: Gain deep insights into infrastructure usage and resource consumption. This knowledge allows you to pinpoint bottlenecks, optimize performance, and make data-driven decisions for cost-effective scaling. - **Ensure Confidence and Reproducibility**: Track the lineage of data and models, meticulously documenting their journey from source to production. This practice fosters reproducibility, enabling you to recreate experiments, validate findings, and ensure consistent behavior across different environments. ### [MLflow](https://mlflow.org/): Your Observability Command Center 📡 [MLflow](https://mlflow.org/), the open-source platform we’ve come to rely on, rises to the occasion once again, providing a versatile and powerful set of tools for managing [the entire ML lifecycle](https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning). The [MLOps Coding Course](https://mlops-coding-course.fmind.dev/) leverages MLflow’s capabilities to the fullest, demonstrating how to: **1. Guarantee** [**Reproducibility**](https://mlops-coding-course.fmind.dev/7.%20Observability/7.0.%20Reproducibility.html) **with** [**MLflow Projects**](https://mlflow.org/docs/latest/projects.html) **:** Standardize the way you package your ML code, dependencies, and environment configurations using [MLflow Projects](https://mlflow.org/docs/latest/projects.html). This ensures consistent execution across different environments and facilitates seamless sharing and collaboration. [**MLproject file**](https://github.com/fmind/mlops-python-package/blob/main/MLproject) **:** ```yaml # Define the structure of your MLflow project name: bikes python_env: python_env.yaml entry_points: main: parameters: conf_file: path command: "PYTHONPATH=src python -m bikes {conf_file}" ``` **2. Shine a Light on** [**Model Monitoring**](https://mlops-coding-course.fmind.dev/7.%20Observability/7.1.%20Monitoring.html) **with** [**MLflow Model Evaluation**](https://mlflow.org/docs/latest/model-evaluation/index.html) **:** Employ [MLflow’s evaluate API](https://mlflow.org/docs/latest/model-evaluation/index.html) to compute and log a comprehensive suite of model performance metrics. Define thresholds to trigger alerts when metrics deviate from expected ranges. [**Evaluation Job file**](https://github.com/fmind/mlops-python-package/blob/main/src/bikes/jobs/evaluations.py) **:** ```python from bikes.core import metrics as metrics_ # Define the metrics to track metrics = [ metrics_.SklearnMetric(name="mean_squared_error", greater_is_better=False), metrics_.SklearnMetric(name="r2_score", greater_is_better=True), ] # Define thresholds for specific metrics (optional) thresholds = { "r2_score": metrics_.Threshold(threshold=0.5, greater_is_better=True) # Alert if R-squared drops below 0.5 } ``` ![Model Monitoring with MLflow Model Evaluation](/static/img/articles/mlops-coding-course-mastering-observability-for-reliable-ml/02.webp) Model Monitoring with [MLflow Model Evaluation](https://mlflow.org/docs/latest/model-evaluation/index.html) **For data and model drift detection, integrate** tools like [Evidently](https://www.evidentlyai.com/) to automate the generation of interactive reports. Visualize data drift, model performance variations, and other critical insights, enabling you to understand and address potential issues quickly. [**Evidently Example**](https://docs.evidentlyai.com/examples/introduction) **:** ```python import pandas as pd from evidently.report import Report from evidently.metric_preset import DataDriftPreset # Load reference data (data used for training) reference_data = pd.read_csv('reference.csv') # Load current data (data the model is currently predicting on) current_data = pd.read_csv('current.csv') # Generate an Evidently report for data drift detection report = Report(metrics=[DataDriftPreset()]) report.run(reference_data=reference_data, current_data=current_data) report.show() # Display the interactive report in a web browser # or # report.save_html('my_report.html') # Save the report as an HTML file ``` **3. Set up** [**Alerting**](https://mlops-coding-course.fmind.dev/7.%20Observability/7.2.%20Alerting.html) **for Timely Interventions:** During development, utilize a simple alerting service based on the [Plyer](https://plyer.readthedocs.io/) library. Send instant desktop notifications to developers about significant events in the MLOps pipeline. [**Alerting Service file**](https://github.com/fmind/mlops-python-package/blob/main/src/bikes/io/services.py) **:** ```python from bikes.io import services # Initialize the alerting service, enable notifications, and set application name and timeout alerts_service = services.AlertsService(enable=True, app_name="Bikes", timeout=5) # Within a job's run() method, send a notification when a task completes self.alerts_service.notify(title="Training Complete", message=f"Model version: {model_version.version}") ``` For production environments, integrate with powerful platforms like [Datadog](https://www.datadoghq.com/). Datadog offers comprehensive dashboards, customizable alerts, and flexible notification channels to keep you informed. **4. Trace the** [**Data/Model Lineage**](https://mlops-coding-course.fmind.dev/7.%20Observability/7.3.%20Lineage.html) **with** [**MLflow Dataset Tracking**](https://mlflow.org/docs/latest/tracking.html#tracking-datasets) **:** Employ [MLflow Data API](https://mlflow.org/docs/latest/tracking/data-api.html) to meticulously track the lineage of your data, documenting its origin, transformations, and usage within your models. This creates a transparent and auditable record, essential for debugging, reproducibility, and data governance. [**Lineage in Training Job file**](https://github.com/fmind/mlops-python-package/blob/main/src/bikes/jobs/training.py) **:** ```python import mlflow.data.pandas_dataset as lineage # Within a job's run() method inputs_lineage = lineage.from_pandas( df=data, name=name, source=self.path, targets=targets, predictions=predictions ) mlflow.log_input(dataset=inputs_lineage, context=self.run_config.name) ``` ![Data Lineage information gathered with MLflow Data API](/static/img/articles/mlops-coding-course-mastering-observability-for-reliable-ml/03.webp) Data Lineage information gathered with [MLflow Data API](https://mlflow.org/docs/latest/tracking/data-api.html) **5.** [**Manage Costs and Measure Success with KPIs**](https://mlops-coding-course.fmind.dev/7.%20Observability/7.4.%20Costs-KPIs.html) **:** The [MLOps Python Package](https://github.com/fmind/mlops-python-package) provides a practical [notebook](https://github.com/fmind/mlops-python-package/blob/main/notebooks/indicators.ipynb) demonstrating how to extract and analyze technical cost and KPI data from an [MLflow server](https://mlflow.org/docs/latest/getting-started/running-notebooks/index.html). This data empowers you to understand resource consumption patterns, identify bottlenecks, and optimize your project’s performance and budget. ![Visualize the run time of experiment runs from the MLflow Server](/static/img/articles/mlops-coding-course-mastering-observability-for-reliable-ml/04.webp) Visualize the run time of experiment runs from the MLflow Server **6. Open the Black Box with** [**Explainability**](https://mlops-coding-course.fmind.dev/7.%20Observability/7.5.%20Explainability.html) **:** Integrate [SHAP (SHapley Additive exPlanations)](https://shap.readthedocs.io/en/latest/) to unveil the decision-making process of your models. Analyze feature importance scores, both globally and for individual predictions, to gain insights into model behavior, identify potential biases, and guide model improvement efforts. [**Explain samples from Models file**](https://github.com/fmind/mlops-python-package/blob/main/src/bikes/core/models.py) **:** ```python @T.override def explain_samples(self, inputs: schemas.Inputs) -> schemas.SHAPValues: """Explain model outputs on input samples. Args: inputs (schemas.Inputs): The input data samples. Returns: schemas.SHAPValues: A dataframe containing the SHAP values for each feature. """ model = self.get_internal_model() regressor = model.named_steps["regressor"] transformer = model.named_steps["transformer"] transformed = transformer.transform(X=inputs) explainer = shap.TreeExplainer(model=regressor) shap_values = schemas.SHAPValues( data=explainer.shap_values(X=transformed), columns=transformer.get_feature_names_out(), ) return shap_values ``` ![SHAP Values for explaining feature influences on data samples](/static/img/articles/mlops-coding-course-mastering-observability-for-reliable-ml/05.webp) [SHAP Values](https://shap.readthedocs.io/en/latest/example_notebooks/overviews/An%20introduction%20to%20explainable%20AI%20with%20Shapley%20values.html#) for explaining feature influences on data samples **7. Keep a Watchful Eye on** [**Infrastructure**](https://mlops-coding-course.fmind.dev/7.%20Observability/7.6.%20Infrastructure.html) **with** [**MLflow System Metrics**](https://mlflow.org/docs/latest/system-metrics/index.html) **:** Enable [MLflow system metrics](https://mlflow.org/docs/latest/system-metrics/index.html) logging to capture valuable hardware performance indicators during the execution of your MLOps jobs. This data provides a window into resource utilization, helps you identify potential performance bottlenecks or issues, and enables you to make data-driven decisions regarding scaling and resource allocation. ![Collect and display System Metrics with MLflow](/static/img/articles/mlops-coding-course-mastering-observability-for-reliable-ml/06.webp) Collect and display System Metrics with MLflow ### Conclusions Observability is the key to unlocking the true potential of your ML solutions. The [MLOps Coding Course arms](https://mlops-coding-course.fmind.dev/) you with the knowledge and tools to build robust, insightful, and production-ready monitoring systems, ensuring your AI/ML initiatives thrive in the dynamic world of production. Embrace the principles and practices outlined in the course, integrate powerful tools like [MLflow](https://mlflow.org/docs/latest/index.html), [Evidently](https://www.evidentlyai.com/) or [Datadog](https://www.datadoghq.com/solutions/machine-learning/), and watch your MLOps projects blossom with enhanced reliability, performance, and trustworthiness. ![Photo by Luca Bravo on Unsplash](/static/img/articles/mlops-coding-course-mastering-observability-for-reliable-ml/07.webp) Photo by [Luca Bravo](https://unsplash.com/@lucabravo?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [MLOps Coding Course: Bridging the gap between Data Scientists and Machine Learning Engineers](https://www.fmind.dev/articles/mlops-coding-course-bridging-the-gap-between-data-scientists-and-machine-learning-engineers/) As a Freelance MLOps Engineer working for [Decathlon Digital](https://digital.decathlon.net/), I’ve witnessed firsthand the growing need for data scientists to transition into machine learning engineers. The increasing complexity of AI/ML projects demands more than just modeling skills; it requires a deep understanding of software development practices to ensure that models can be deployed, scaled, and maintained effectively in production environments. This observation sparked the creation of the [**MLOps Coding Course**](https://mlops-coding-course.fmind.dev/), an open-source course specifically designed to bridge the gap between data science and software engineering. It’s a comprehensive guide that offers practical knowledge and tools to build, deploy, and manage production-ready AI/ML systems. ![MLOps Coding Course: https://mlops-coding-course.fmind.dev/](/static/img/articles/mlops-coding-course-bridging-the-gap-between-data-scientists-and-machine-learning-engineers/cover.webp) MLOps Coding Course: [https://mlops-coding-course.fmind.dev/](https://mlops-coding-course.fmind.dev/) ### Why Coding Skills Are Essential for MLOps The course emphasizes **coding best practices** because they are fundamental for building robust and maintainable MLOps systems. Strong coding skills enable ML engineers to: - **Structure code effectively:** Organizing code into packages, modules, and functions promotes modularity, reusability, and easier maintenance. - **Implement robust validation:** Applying techniques like typing, linting, and testing ensures code quality, reduces errors, and facilitates collaboration. - **Automate tasks efficiently:** Scripting common tasks with tools like [PyInvoke](https://www.pyinvoke.org/) streamlines workflows, saving time and reducing manual effort. - **Manage dependencies effectively:** Utilizing tools like [Poetry](https://python-poetry.org/) simplifies the management of dependencies, ensuring consistent environments across development and production. - **Build reproducible environments:** Leveraging containers with [Docker](https://www.docker.com/) ensures consistent deployment environments, mitigating “it works on my machine” issues. ### Course Highlights The [**MLOps Coding Course**](https://mlops-coding-course.fmind.dev/) aims at establishing a solid foundation. You will learn how to [**set up your system and installing necessary tools**](https://mlops-coding-course.fmind.dev/1.%20Initializing/index.html) such as [Python](https://python.org/), [pyenv](https://github.com/pyenv/pyenv), [Poetry](https://python-poetry.org/), [Git](https://git-scm.com/), [GitHub](https://github.com/), and [VS Code](https://code.visualstudio.com/). The course then dive into [**prototyping with Jupyter Notebooks**](https://mlops-coding-course.fmind.dev/2.%20Prototyping/index.html), where we cover best practices for managing [imports](https://mlops-coding-course.fmind.dev/2.%20Prototyping/2.1.%20Imports.html), [configurations](https://mlops-coding-course.fmind.dev/2.%20Prototyping/2.2.%20Configs.html), [datasets](https://mlops-coding-course.fmind.dev/2.%20Prototyping/2.3.%20Datasets.html), [analysis](https://mlops-coding-course.fmind.dev/2.%20Prototyping/2.4.%20Analysis.html), [modeling](https://mlops-coding-course.fmind.dev/2.%20Prototyping/2.5.%20Modeling.html), and [evaluation](https://mlops-coding-course.fmind.dev/2.%20Prototyping/2.6.%20Evaluations.html). The course then moves on to [**productionization**](https://mlops-coding-course.fmind.dev/3.%20Productionizing/index.html), guiding you on how to structure code into proper Python [packages](https://mlops-coding-course.fmind.dev/3.%20Productionizing/3.0.%20Package.html). You will gain an understanding of [modules](https://mlops-coding-course.fmind.dev/3.%20Productionizing/3.1.%20Modules.html), [programming paradigms](https://mlops-coding-course.fmind.dev/3.%20Productionizing/3.2.%20Paradigms.html) like OOP and functional programming, and learn how to set up [entry points](https://mlops-coding-course.fmind.dev/3.%20Productionizing/3.3.%20Entrypoints.html). We also address externalizing [configurations](https://mlops-coding-course.fmind.dev/3.%20Productionizing/3.4.%20Configurations.html), [documenting](https://mlops-coding-course.fmind.dev/3.%20Productionizing/3.5.%20Documentations.html) code effectively, and creating [VS Code workspaces](https://mlops-coding-course.fmind.dev/3.%20Productionizing/3.6.%20VS%20Code%20Workspace.html) to facilitate collaborative development. A significant portion of the course is dedicated to [**code validation**](https://mlops-coding-course.fmind.dev/4.%20Validating/index.html), a cornerstone of robust MLOps pipelines. You will learn how to implement typing using [type hints](https://mlops-coding-course.fmind.dev/4.%20Validating/4.0.%20Typing.html) and tools like [Mypy](https://mypy.readthedocs.io/), and learn to [lint your code](https://mlops-coding-course.fmind.dev/4.%20Validating/4.1.%20Linting.html) with [Ruff](https://docs.astral.sh/ruff/) for style and quality checks. We also cover [testing your code](https://mlops-coding-course.fmind.dev/4.%20Validating/4.2.%20Testing.html) with [pytest](https://pytest.org/), including unit testing, fixture usage, and coverage analysis. Further refining your codebase involves exploring [logging](https://mlops-coding-course.fmind.dev/4.%20Validating/4.3.%20Logging.html) with [Loguru](https://loguru.readthedocs.io/) for monitoring and debugging, [securing](https://mlops-coding-course.fmind.dev/4.%20Validating/4.4.%20Security.html) your codebase with tools like [Bandit](https://bandit.readthedocs.io/) and [GitHub Dependabot](https://github.com/dependabot), and ensuring consistent [formatting](https://mlops-coding-course.fmind.dev/4.%20Validating/4.5.%20Formatting.html) with [Black](https://black.readthedocs.io/en/stable/) and [Ruff](https://docs.astral.sh/ruff/). Lastly, you will gain practical skills in [debugging](https://mlops-coding-course.fmind.dev/4.%20Validating/4.6.%20Debugging.html) effectively using [VS Code’s integrated debugger](https://code.visualstudio.com/docs/editor/debugging). The [**refining stage**](https://mlops-coding-course.fmind.dev/5.%20Refining/index.html) of the course goes even further by presenting advanced concepts such as [software design patterns](https://mlops-coding-course.fmind.dev/5.%20Refining/5.0.%20Design%20Patterns.html) like [Strategy](https://en.wikipedia.org/wiki/Strategy_pattern), [Factory](https://en.wikipedia.org/wiki/Factory_method_pattern), and [Adapter](https://en.wikipedia.org/wiki/Adapter_pattern), and explores [task automation](https://mlops-coding-course.fmind.dev/5.%20Refining/5.1.%20Task%20Automation.html) with [PyInvoke](https://www.pyinvoke.org/). You will learn to use [pre-commit hooks](https://mlops-coding-course.fmind.dev/5.%20Refining/5.2.%20Pre-Commit%20Hooks.html) for early quality checks and set up [CI/CD workflow](https://mlops-coding-course.fmind.dev/5.%20Refining/5.3.%20CI-CD%20Workflows.html)s with [GitHub Actions](https://github.com/features/actions). Additionally, we guide you on building and deploying [software containers](https://mlops-coding-course.fmind.dev/5.%20Refining/5.4.%20Software%20Containers.html) with [Docker](https://www.docker.com/), [tracking and managing ML experiments](https://mlops-coding-course.fmind.dev/5.%20Refining/5.5.%20AI-ML%20Experiments.html) with [MLflow](https://mlflow.org/), and [utilizing model registries](https://mlops-coding-course.fmind.dev/5.%20Refining/5.6.%20Model%20Registries.html) for version control and deployment. Finally, the course tackles the crucial aspect of [**sharing your MLOps projects**](https://mlops-coding-course.fmind.dev/6.%20Sharing/index.html) with others. We discuss setting up and managing [code repositories](https://mlops-coding-course.fmind.dev/6.%20Sharing/6.0.%20Repository.html), selecting an appropriate [software license](https://mlops-coding-course.fmind.dev/6.%20Sharing/6.1.%20License.html), writing a comprehensive [README.md file](https://mlops-coding-course.fmind.dev/6.%20Sharing/6.2.%20Readme.html), managing [project releases](https://mlops-coding-course.fmind.dev/6.%20Sharing/6.3.%20Releases.html), and building [code templates](https://mlops-coding-course.fmind.dev/6.%20Sharing/6.4.%20Templates.html) with [Cookiecutter](https://cookiecutter.readthedocs.io/en/stable/) and [cruft](https://cruft.github.io/cruft/). We also cover setting up [cloud workstations](https://cloud.google.com/workstations) for collaborative development and strategies for fostering [contributions](https://mlops-coding-course.fmind.dev/6.%20Sharing/6.6.%20Contributions.html) and building a thriving community around your project. ### Personalized Support: MLOps Coding Assistant and Mentoring The course goes beyond static content, offering: - [**MLOps Coding Assistant**](https://mlops-coding-assistant.fmind.dev/) **:** A [premium AI-powered chatbot](https://mlops-coding-course.fmind.dev/0.%20Overview/0.5.%20Assistants.html) specifically trained on the course material to provide tailored responses to your questions and offer code feedback from your inputs. - [**Mentoring Sessions**](https://mlops-coding-course.fmind.dev/0.%20Overview/0.4.%20Mentoring.html) **:** Personalized guidance and support from experienced MLOps professionals to help you apply the course concepts to your specific challenges. ### Companion Repository: MLOps Python Package To complement the theoretical aspects of the course, we’ve developed the [**MLOps Python Package**](https://github.com/fmind/mlops-python-package), a practical companion repository. This resource serves as a demonstration of the concepts and best practices discussed throughout the course. It offers a flexible, robust, and productive Python package structure that you can use as a foundation for your own MLOps initiatives. By examining the code and structure of the MLOps Python Package, you can gain a deeper understanding of how to apply the course’s teachings to real-world projects, accelerating your journey from theory to practice. ### Embracing MLOps for Success Whether you’re a data scientist eager to explore the world of MLOps or a seasoned ML engineer seeking to refine your skills, the [MLOps Coding Course](https://mlops-coding-course.fmind.dev/) provides a valuable resource to enhance your knowledge and elevate your projects. We encourage you to explore the course materials and embark on this journey of mastering MLOps. [This course is a community-driven effort](https://github.com/MLOPS-Teaching/mlops-coding-course), released under the [Creative Commons Attribution 4.0 International license](https://github.com/MLOps-Courses/mlops-coding-course/blob/main/LICENSE.txt). We believe in the power of open-source collaboration and welcome contributions from anyone passionate about MLOps. If you have insights, examples, or resources to share, please join us in making this course even more comprehensive and valuable for the entire [MLOps community](https://mlops.community/). _Thanks to the course’s co-author_ [_Matthieu Jimenez_](https://website.jimenez.lu/) _for its support and contributions._ ![Photo by Aditya Chinchure on Unsplash](/static/img/articles/mlops-coding-course-bridging-the-gap-between-data-scientists-and-machine-learning-engineers/02.webp) Photo by [Aditya Chinchure](https://unsplash.com/@adityachinchure?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [Make your MLOps code base SOLID with Pydantic and Python’s ABC](https://www.fmind.dev/articles/make-your-mlops-code-base-solid-with-pydantic-and-pythons-abc/) MLOps projects are straightforward to initiate, but challenging to perfect. While AI/ML projects often start with a notebook for prototyping, deploying them directly in production is often considered [poor practice by the MLOps community](https://www.youtube.com/watch?v=7jiPeIFXb6U). Transitioning to a dedicated Python code base is essential for industrializing the project, yet this move presents several challenges: 1) How can we maintain a code base that is robust yet flexible for agile development? 2) Is it feasible to implement proven design patterns while keeping the code base accessible to all developers? 3) How can we leverage Python’s dynamic nature while adopting strong typing practices akin to static languages? Throughout my career, I have thoroughly explored various strategies to make my code base both simple and powerful. In 2009, I had the opportunity to collaborate with seasoned developers and enthusiasts of design patterns in object-oriented languages such as [C++](https://isocpp.org/) and [Java](https://www.java.com/en/). By 2015, I had devoted hundreds of hours to mastering functional programming paradigms with languages like [Clojure](https://clojure.org/) (LISP) and [Haskell](https://www.haskell.org/). This journey led me to discover both modern and time-tested practices, which I have applied to my AI/ML projects. I am eager to share these practices and reveal the most effective solutions I’ve encountered. ![The Three Little Pig illustrates the needs for building robust structures](/static/img/articles/make-your-mlops-code-base-solid-with-pydantic-and-pythons-abc/cover.webp) The Three Little Pig illustrates the needs for building robust structures **In this article, I propose a method to develop high-quality MLOps projects using Python's ABC and Pydantic**. I begin by emphasizing the importance of implementing SOLID software practices in AI/ML codebases. Next, I offer some background on design patterns and the SOLID principles. Then, I recount my experiences with various code architectures and their limitations. Finally, I explain how Python's ABC and Pydantic can enhance the quality of your Python code and facilitate the adoption of sound coding practices. ### Motivations Let’s review the motivations for using advanced patterns in MLOps projects. #### Freedom to Choose AI/ML code bases are tasked with performing several critical operations: - Reading data from various sources - Processing data to generate diverse sets of features - Training models using available machine-learning libraries - Generating and evaluating predictions or content across numerous scenarios - Adapting the code base to fit the project infrastructure, such as model registries While supporting a single option at each of these steps is relatively straightforward, AI/ML solutions should remain flexible and adaptable. [The AI/ML field is dynamic, with better algorithms and models continuously being developed](https://paperswithcode.com/). [Data competitions highlight the importance of creativity and experimentation in achieving peak performance](https://mlcontests.com/state-of-competitive-machine-learning-2023/). Therefore, an AI/ML project must be designed with this flexibility in mind, offering the capacity to integrate new solutions over time within the same code base. #### Robustness of the code The dynamic nature of Python serves as both an advantage and a drawback. On one side, it offers developers the flexibility to mold the code base into any desired form, enabling the creation of innovative abstractions. However, this same flexibility can lead to disorganized code structures, often referred to humorously as [code lasagna](https://wiki.c2.com/?LasagnaCode), [code spaghetti](https://en.wikipedia.org/wiki/Spaghetti_code), and other varieties of ‘code pasta.’ Echoing the well-known adage from Spider-Man, “With great power comes great responsibility,” Python developers, more than those in other static languages, need to exercise discipline to ensure their code remains robust. This can be achieved through practices such as gradual typing, thorough validation of inputs and outputs, and the use of interfaces that clearly define the expectations for each software component. #### Efficiency of developers Developer time is a valuable asset, one that shouldn’t be squandered on constant refactoring or the elimination of technical debt. For example, web frameworks like Django or Flask offer a suite of abstractions that enable developers to quickly start building websites using industry-proven patterns. These frameworks afford the flexibility to switch between database systems, integrate internal objects with database tables through [Object-Relational Mapping](https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping) (ORM), or incorporate [middleware](https://en.wikipedia.org/wiki/Middleware) and callbacks to seamlessly integrate new systems. In a similar vein, an MLOps code base should offer the same level of flexibility and efficiency. By adopting the right design patterns, developers are empowered to investigate more solutions, achieve greater productivity, and deliver enhanced value to their projects. This approach can mean the difference between struggling against your code base and leveraging it to enhance the team’s ability to deliver effectively. ### Definitions This section outlines key concepts used in this article. You can skip it if you’re already knowledgeable about these topics. #### Gradual Typing [Gradual typing](https://en.wikipedia.org/wiki/Gradual_typing) is a feature in programming languages that allows for the incremental introduction of [type annotations](https://en.wikipedia.org/wiki/Type_signature) into a code base, enhancing its robustness. This feature is supported by several dynamic languages, including JavaScript (via [TypeScript](https://www.typescriptlang.org/)), PHP (via [Hack](https://hacklang.org/)), [Dart](https://dart.dev/), and Python (via [mypy](https://mypy.readthedocs.io/en/stable/config_file.html)). Consider the following example: ```python import numpy as np # without type annotations def split_data(data, test_ratio): shuffled_indices = np.random.permutation(len(data)) test_set_size = int(len(data) * test_ratio) test_indices = shuffled_indices[:test_set_size] train_indices = shuffled_indices[test_set_size:] return data[train_indices], data[test_indices] # with type annotations def split_data(data: np.ndarray, test_ratio: float) -> tuple[np.ndarray, np.ndarray]: shuffled_indices = np.random.permutation(len(data)) test_set_size = int(len(data) * test_ratio) test_indices = shuffled_indices[:test_set_size] train_indices = shuffled_indices[test_set_size:] return data[train_indices], data[test_indices] ``` While dedicating time to assigning types to expressions might seem laborious, the benefits frequently surpass the costs. This is particularly relevant for MLOps code bases, which often manage large, untyped structures like DataFrames or AI/ML models. Utilizing dataframe schemas with tools like [Pandera](https://pandera.readthedocs.io/en/stable/) and defining model signatures with MLflow can significantly aid in object validation and the clear communication of structures. These practices not only improve code quality and maintainability but also facilitate better collaboration among developers by making the code more self-documenting and easier to understand. #### Design Patterns [Design patterns](https://en.wikipedia.org/wiki/Design_Patterns) are standardized solutions devised to address recurring problems in software development. For example, the [Memento pattern](https://en.wikipedia.org/wiki/Memento_pattern) enables saving the state of an entire program, while the Singleton pattern ensures a class has only one instance throughout the application, providing a single point of access to it. The example below showcases the [Decorator pattern](https://en.wikipedia.org/wiki/Decorator_pattern) for extending the capabilities of a Python function: ```python import time from functools import wraps def timer(func): """Decorator for timing functions.""" @wraps(func) def wrapper(*args, **kwargs): start_time = time.time() result = func(*args, **kwargs) end_time = time.time() print(f"{func.__name__} completed in {end_time - start_time} seconds") return result return wrapper @timer def train_model(data): """Simulate the training of a model.""" time.sleep(2) # Placeholder for the actual training logic return "Model trained" # Calling the decorated train model function train_model("sample_data") ``` I wrote [an article that delves into the design patterns I find most pertinent to AI/ML code bases](https://mlops.community/become-the-maestro-of-your-mlops-abstractions/), such as the Factory, Strategy, and Adapter patterns. Although it’s impractical to explore every pattern, a wealth of literature exists on the subject. [Numerous books](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html) and [articles](https://wiki.c2.com/) offer detailed guidance on understanding and choosing the most suitable design patterns for specific scenarios. #### SOLID Principles The [SOLID principles](https://en.wikipedia.org/wiki/SOLID) are fundamental to developing flexible and maintainable object-oriented code bases. These principles encourage the design of software in such a way that it facilitates easy maintenance and extension. Instead of confining your program to a single workflow, applying SOLID principles allows for the creation of modular code components. These components can be easily interchanged and reused throughout your project, enhancing both the scalability and robustness of the code base. SOLID is an acronym that represents five key design principles: - ***S*ingle Responsibility Principle **— A class should have one, and only one, reason to change. - ***O*pen/Closed Principle** — Software entities should be open for extension, but closed for modification. - ***L*iskov Substitution Principle** — Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program. - ***I*nterface Segregation Principle** — No client should be forced to depend on methods it does not use. - ***D*ependency Inversion Principle **— High-level modules should not depend on low-level modules. Both should depend on abstractions. ### Solutions Let’s discuss key methods for organizing MLOps code bases and their drawbacks. #### Just Write a Script The prevalent approach for structuring AI/ML code bases, as often found in online examples, is straightforward: consolidate everything into a single Python script. This method is appealing for its simplicity, ensuring the code base remains concise and focused. However, this simplicity can soon prove to be inadequate for addressing the complexities of real-world applications. The typical characteristics of this approach are illustrated in the example below: ```python # Simplistic AI/ML Python Script Example import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score # Load and preprocess data data = pd.read_csv('dataset.csv') data.fillna(0, inplace=True) # Split data X, y = data.drop('target', axis=1), data['target'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Train model model = LogisticRegression() model.fit(X_train, y_train) # Evaluate model predictions = model.predict(X_test) accuracy = accuracy_score(y_test, predictions) print(f"Model Accuracy: {accuracy}") ``` This script, while concise, combines data loading, preprocessing, model training, and evaluation in a single block. This approach makes it challenging to add new features, adjust preprocessing steps, or swap out the model without significant changes to the code base. It also violates SOLID principles by not being particularly well-organized for scalability, maintainability, or flexibility. #### Functional Programming (FP) Adopting [Functional Programming](https://en.wikipedia.org/wiki/Functional_programming) (FP) represents a significant enhancement compared to merely scripting. Rather than delineating a singular workflow for your program, FP allows for the encapsulation of code components into functions. These functions can then be orchestrated in a declarative workflow, enhancing code clarity and structure. This approach embraces core functional programming concepts, such as [high-order functions](https://en.wikipedia.org/wiki/Higher-order_function), [immutability](https://en.wikipedia.org/wiki/Immutable_object) and [pure functions](https://en.wikipedia.org/wiki/Pure_function). For instance, the example below showcases how a model can be created from a string using the two high-order level function _get_model()_ and _train_model():_ ```python from typing import Callable, Tuple import pandas as pd from sklearn.model_selection import train_test_split from sklearn.base import BaseEstimator from sklearn.linear_model import LogisticRegression from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import accuracy_score def load_and_preprocess_data(filepath: str, fill_na_value: float, target_name: str) -> Tuple[pd.DataFrame, pd.Series]: """Load and preprocess data.""" data = pd.read_csv(filepath) data = data.fillna(fill_na_value) X = data.drop(target_name, axis=1) y = data[target_name] return X, y def split_data(X: pd.DataFrame, y: pd.Series, test_size: float, random_state: int) -> Tuple[pd.DataFrame, pd.DataFrame, pd.Series, pd.Series]: """Split the data into a train and testing sets.""" return train_test_split(X, y, test_size=test_size, random_state=random_state) def train_model(X_train: pd.DataFrame, y_train: pd.Series, model_func: Callable[[], BaseEstimator], **kwargs) -> BaseEstimator: """Train the model with inputs and target data.""" model = model_func(**kwargs) model.fit(X_train, y_train) return model def evaluate_model(model: BaseEstimator, X_test: pd.DataFrame, y_test: pd.Series) -> float: """Evaluate the model with a single metric.""" predictions = model.predict(X_test) accuracy = accuracy_score(y_test, predictions) return accuracy def get_model(model_name: str) -> Callable[[], BaseEstimator]: """High-order function to select the model to train.""" if model_name == "logistic_regression": return LogisticRegression elif model_name == "random_forest": return RandomForestClassifier else: raise ValueError(f"Model {model_name} is not supported.") def run_workflow(model_name: str, model_kwargs: dict, filepath: str, fill_na_value: float, target_name: str, test_size: float, random_state: int) -> None: """Orchestrate the training workflow.""" X, y = load_and_preprocess_data(filepath, fill_na_value, target_name) X_train, X_test, y_train, y_test = split_data(X, y, test_size, random_state) model_func = get_model(model_name) model = train_model(X_train, y_train, model_func, **model_kwargs) evaluate_model(model, X_test, y_test) # Example usage run_workflow( filepath='dataset.csv', fill_na_value=0.0, target_name='target', test_size=0.2, random_state=42, model_name='random_forest', # Or 'logistic_regression' model_kwargs={'n_estimators': 30}, ) ``` Functional programming strikes a great balance between simplicity and power, but it faces a significant challenge in Python. Python doesn’t support advanced functional programming concepts as well as languages like Haskell or Clojure do. Although you can write functions and use libraries like [Toolz](https://github.com/pytoolz/toolz) or [Fn.py](https://github.com/kachayev/fn.py), it’s not straightforward to use advanced techniques such as [monads](https://en.wikipedia.org/wiki/Monad_%28functional_programming%29), [currying](https://en.wikipedia.org/wiki/Currying), or [persistent data structures](https://en.wikipedia.org/wiki/Persistent_data_structure). Additionally, Python primarily relies on [subtyping](https://en.wikipedia.org/wiki/Subtyping) for [polymorphism](https://en.wikipedia.org/wiki/Polymorphism_%28computer_science%29), which isn’t as compatible with functional programming as [ad-hoc](https://en.wikipedia.org/wiki/Ad_hoc_polymorphism) or [parametric polymorphism](https://en.wikipedia.org/wiki/Parametric_polymorphism). Despite my wish for Python to lean more towards functional programming, trying to fully adopt this paradigm in Python might be a frustrating experience. #### Object-Oriented Programming (OOP) [Object-Oriented Programming](https://en.wikipedia.org/wiki/Object-oriented_programming) (OOP) is an excellent way to use SOLID principles in Python. Python fully supports OOP concepts, making it easy to work with. Additionally, many online tools and frameworks like [scikit-learn](https://scikit-learn.org/) and [pandas](https://pandas.pydata.org/) use OOP in their APIs. An example of this is defining a Model base class with the ABC module, which is then extended by two subclasses: _RandomForestModel_ and _KerasBinaryClassifier_. A _ModelFactory_ can select and configure the appropriate model based on external inputs. ```python from abc import ABC, abstractmethod from typing import Tuple, Type import pandas as pd from sklearn.model_selection import train_test_split from sklearn.base import BaseEstimator from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import accuracy_score from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense class Model(ABC): """Abstract base class for models.""" @abstractmethod def train(self, X_train: pd.DataFrame, y_train: pd.Series) -> None: pass @abstractmethod def predict(self, X: pd.DataFrame) -> pd.Series: pass class RandomForestModel(Model): """Random Forest Classifier model.""" def __init__(self, n_estimators: int = 20, max_depth: int = 5) -> None: self.model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth) def train(self, X_train: pd.DataFrame, y_train: pd.Series) -> None: self.model.fit(X_train, y_train) def predict(self, X: pd.DataFrame) -> pd.Series: return self.model.predict(X) class KerasBinaryClassifier(Model): """Simple binary classification model using Keras.""" def __init__(self, input_dim: int, epochs: int = 100, batch_size: int = 32) -> None: self.epochs = epochs self.batch_size = batch_size self.model = Sequential([ Dense(64, activation='relu', input_shape=(input_dim,)), Dense(1, activation='sigmoid') ]) self.model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy']) def train(self, X_train: pd.DataFrame, y_train: pd.Series) -> None: self.model.fit(X_train, y_train, epochs=self.epochs, batch_size=self.batch_size) def predict(self, X: pd.DataFrame) -> pd.Series: predictions = self.model.predict(X) return (predictions > 0.5).flatten() class ModelFactory: """Factory to create model instances.""" @staticmethod def get_model(model_name: str, **kwargs) -> Model: # Assume all model classes are defined in the global scope. model_class = globals()[model_name] return model_class(**kwargs) class Workflow: """Main workflow class for model training and evaluation.""" def run_workflow(self, model_name: str, model_kwargs: dict, filepath: str, fill_na_value: float, target_name: str, test_size: float, random_state: int) -> None: X, y = self.load_and_preprocess_data(filepath, fill_na_value, target_name) X_train, X_test, y_train, y_test = self.split_data(X, y, test_size, random_state) model = ModelFactory.get_model(model_name, **model_kwargs) model.train(X_train, y_train) accuracy = self.evaluate_model(model, X_test, y_test) print(f"Model Accuracy: {accuracy}") def load_and_preprocess_data(self, filepath: str, fill_na_value: float, target_name: str) -> Tuple[pd.DataFrame, pd.Series]: """Load and preprocess data.""" data = pd.read_csv(filepath) data = data.fillna(fill_na_value) X = data.drop(target_name, axis=1) y = data[target_name] return X, y def split_data(self, X: pd.DataFrame, y: pd.Series, test_size: float, random_state: int) -> Tuple[pd.DataFrame, pd.DataFrame, pd.Series, pd.Series]: """Split the data into a train and testing sets.""" return train_test_split(X, y, test_size=test_size, random_state=random_state) def evaluate_model(self, model: Model, X_test: pd.DataFrame, y_test: pd.Series) -> float: """Evaluate the model with a single metric.""" predictions = model.predict(X_test) accuracy = accuracy_score(y_test, predictions) return accuracy # Example usage workflow = Workflow() workflow.run_workflow( filepath='dataset.csv', fill_na_value=0.0, target_name='target', test_size=0.2, random_state="42", model_name='RandomForestModel', # Or 'KerasBinaryClassifier' model_kwargs={'n_estimators': 30}, ) ``` This code structure is often seen in well-developed MLOps projects, but it has its issues. Firstly, it lacks input and output validation at startup, leading to potential errors like mistaking a string for an integer in the _random_state_ input. Secondly, implementing a design pattern like the ModelFactory class can be complex and hard for beginners to use. To overcome these problems, I suggest using Pydantic to streamline the design and enhance the code’s reliability. ### Pydantic and ABC [Pydantic](https://docs.pydantic.dev/latest/) is a powerful tool for creating and validating objects, ensuring that data fits your specifications. On the other hand, [Python’s Abstract Base](https://docs.python.org/3/library/abc.html) Classes (ABC) allow you to define reusable code interfaces, setting a blueprint for others to follow. Let’s dive into how these solutions can be used effectively. #### Object Validation Pydantic primarily focuses on [object validation](https://docs.pydantic.dev/latest/why/#type-hints). **By adding type annotations to your class attributes, you can ensure your inputs are checked right when your program starts**. This is especially useful in MLOps, where incorrect inputs can disrupt lengthy training sessions and waste valuable resources. ```python from typing import Optional from pydantic import BaseModel, Field class RandomForestClassifierModel(BaseModel): n_estimators: int = Field(default=100, gt=0) max_depth: Optional[int] = Field(default=None, gt=0, allow_none=True) random_state: Optional[int] = Field(default=None, gt=0, allow_none=True) model = RandomForestClassifierModel(n_estimators=120, max_depth=5, random_state=42) ``` #### Discriminated Union The [Discriminated Union](https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions) feature in Pydantic is a standout tool. **This feature lets you choose the class within a Union type by using a specific attribute (like KIND)** and validates that class with its particular attributes. With [Pydantic’s serialization abilities](https://docs.pydantic.dev/latest/why/#serialization), you can use this pattern as a streamlined alternative to the traditional Factory pattern, avoiding a lot of repetitive code. ```python from typing import Literal, Union from pydantic import BaseModel, Field class Model(BaseModel): KIND: str class RandomForestModel(Model): KIND: Literal["RandomForest"] n_estimators: int = 100 max_depth: int = 5 random_state: int = 42 class SVMModel(Model): KIND: Literal["SVM"] C: float = 1.0 kernel: str = "rbf" degree: int = 3 # Union of all model configurations ModelKind = Union[RandomForestModel, SVMModel] class Job(BaseModel): model: ModelKind = Field(..., discriminator="KIND") # initialize a job from a configs config = { "model": { "KIND": "RandomForest", "n_estimators": 100, "max_depth": 5, "random_state": 42, } } job = Job.model_validate(config) ``` #### Abstract Base Classes [Python’s Abstract Base Classes](https://docs.python.org/3/library/abc.html) (ABC) enhance MLOps code by supporting the SOLID principles. **Think of it as ensuring that different pieces fit together perfectly, like matching puzzle pieces**. In the example below, we create a Model class and two subclasses: RandomForestModel and SVMModel. These subclasses align with the base class shape, allowing them to be used interchangeably without issues. ```python from typing import Literal, Union from abc import ABC, abstractmethod import pandas as pd from pydantic import BaseModel, Field class Model(BaseModel, ABC): KIND: str @abstractmethod def fit(self, X: pd.DataFrame, y: pd.DataFrame) -> None: pass @abstractmethod def predict(self, X: pd.DataFrame) -> pd.DataFrame: pass class RandomForestModel(Model): KIND: Literal["RandomForest"] n_estimators: int = 100 max_depth: int = 5 random_state: int = 42 def fit(self, X: pd.DataFrame, y: pd.DataFrame) -> None: print("Fitting RandomForestModel...") def predict(self, X: pd.DataFrame) -> pd.DataFrame: print("Predicting with RandomForestModel...") return pd.DataFrame() class SVMModel(Model): KIND: Literal["SVM"] C: float = 1.0 kernel: str = "rbf" degree: int = 3 def fit(self, X: pd.DataFrame, y: pd.DataFrame) -> None: print("Fitting SVMModel...") def predict(self, X: pd.DataFrame) -> pd.DataFrame: print("Predicting with SVMModel...") return pd.DataFrame() # Union of all model configurations ModelKind = Union[RandomForestModel, SVMModel] class Job(BaseModel): model: ModelKind = Field(..., discriminator="KIND") def run(self) -> pd.DataFrame: X_train, X_test, y_train = ..., ..., ... self.model.fit(X=X_train, y=y_train) predictions = self.model.predict(X=X_test) return predictions # initialize a job from a configs config = { "model": { "KIND": "RandomForest", "n_estimators": 100, "max_depth": 5, "random_state": 42, } } job = Job.model_validate(config) job.run() ``` ### Limitations While Pydantic and Python’s ABC greatly simplify the creation of SOLID MLOps code bases, they also pose several limitations. Let’s explore those in this section. #### Open/Closed Principle SOLID consists of five main principles, with Pydantic and Python’s ABC aiding in three: the Liskov Substitution Principle, Interface Segregation Principle, and Dependency Inversion Principle. They serve as practical tools in these areas, while the Single Responsibility Principle acts as a design guideline for developers. However, Pydantic and Python’s ABC don’t directly support the [Open/Closed Principle](https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle). Specifically, Pydantic’s use of Tagged Unions encourages concrete class unions (like Union\[RandomForestModel, SVMModel\]) over abstract class usage (such as Model), diverging from traditional object-oriented practices. Although this approach might seem limiting, the practical benefits often outweigh the constraints, especially for applications not intended as reusable libraries. Nonetheless, you can adjust by redefining the type union in your application to fit your needs. #### ABC vs Protocol Python offers two ways to define code interfaces: [Abstract Base Classes](https://docs.python.org/3/library/abc.html) (ABC), which use [Nominal Typing](https://en.wikipedia.org/wiki/Nominal_type_system), and [Protocol](https://peps.python.org/pep-0544/), which uses [Structural Typing](https://en.wikipedia.org/wiki/Structural_type_system). Nominal Typing relies on class hierarchies to define relationships clearly, such as a RandomForestModel being a type of Model, making connections between classes explicit. On the other hand, Protocol is at the heart of Python’s [duck typing](https://realpython.com/lessons/duck-typing/) philosophy. It allows any class that implements certain methods to be compatible, even without an explicit declaration, meaning a RandomForestModel just needs to act like a Model to be considered one. While Protocols can replace ABCs in some cases, their use is less straightforward with Pydantic. Pydantic’s requirement for concrete class unions makes the less explicit nature of Protocols less beneficial. Moreover, incorporating default methods into an abstract class is simpler than setting up similar functionalities with Protocols. ### Conclusions This article explored how Pydantic and Python’s ABC can streamline implementing SOLID principles in your projects, providing a simpler, more elegant approach. **With Pydantic, MLOps developers can enjoy automatic class initialization and object validation, while Python’s ABC offers robust abstraction capabilities**. This combination allows developers to concentrate on the core logic of their programs instead of getting bogged down with custom factories and validators. For those interested in seeing Pydantic and Python’s ABC applied in real-world MLOps, check out this GitHub repository: [https://github.com/fmind/mlops-python-package](https://github.com/fmind/mlops-python-package). It not only demonstrates these concepts but also delves into [code linting](https://en.wikipedia.org/wiki/Lint_%28software%29) and [unit testing](https://en.wikipedia.org/wiki/Unit_testing). While I’ve found Pydantic and Python’s ABC to be highly effective for MLOps development, the quest for even simpler and more powerful solutions continues. I envision a [declarative paradigm](https://en.wikipedia.org/wiki/Declarative_programming) for MLOps, akin to what [Kubernetes](https://kubernetes.io/) offers for cloud infrastructure or [Ludwig](https://twitter.com/ludwig_ai) for deep learning. Such an approach would allow data scientists to focus on configurations, while ML engineers could concentrate on rolling out new features. But let’s save that discussion for another time. For now, march forward, and aim to be as SOLID as a rock! ![Make your Python code base rock SOLID!](/static/img/articles/make-your-mlops-code-base-solid-with-pydantic-and-pythons-abc/02.webp) Make your Python code base rock SOLID! ### [Become the maestro of your MLOps abstractions](https://www.fmind.dev/articles/become-the-maestro-of-your-mlops-abstractions/) The MLOps ecosystem is evolving into a sophisticated symphony, composed of diverse tools, methodologies, and cultures. This diversity, while beneficial, also introduces a complexity reminiscent of the challenges encountered in [Big Data systems](https://en.wikipedia.org/wiki/Big_data). Data experts had to navigate through immense data characterized by its [Volume, Variety, and Velocity](https://www.techtarget.com/whatis/definition/3Vs). Such intricacies can lead to a state of [analysis paralysis](https://en.wikipedia.org/wiki/Analysis_paralysis), where decision-makers are inundated with options, hesitant to commit for fear of poor design choices. To manage this complexity, it became essential to develop and own [**abstractions**](https://en.wikipedia.org/wiki/Abstraction) that encapsulate the underlying complexity, offering a seamless and adaptable architecture for integrating new components**.** For instance, [Apache Spark](https://spark.apache.org/) emerged as an exemplary abstraction for managing Big Data applications at scale, courtesy of its immutable and lazy programming approach. Complementary systems like [Hive Metastore](https://hive.apache.org/) further enriched the ecosystem with its capabilities. This raises a pivotal question: Can similar solutions be adopted for MLOps? ![Photo by Julio Rionaldo on Unsplash](/static/img/articles/become-the-maestro-of-your-mlops-abstractions/cover.webp) Photo by [Julio Rionaldo](https://unsplash.com/@juliorionaldo?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) In this article, I aim to delineate a roadmap for constructing robust MLOps platforms and projects**.** Initially, I will underscore the importance of devising and mastering your own MLOps abstractions. Following this, I will outline key design patterns essential for forging simple yet potent abstractions for your projects. Lastly, I will delve into real-world case studies, illustrating the critical role of abstractions in the success of various projects. ### Why are abstractions essential for MLOps? [An MLOps platform encompasses various components](https://docs.zenml.io/concepts/stack_components) for training, tracking, deploying, and monitoring AI/ML models. It also integrates with data solutions like data warehouses, lakes, and catalogs. Each component has numerous proprietary and open-source solutions. Over time, these components may undergo changes, be it through upgrades or replacements with more efficient alternatives. ![THE 2023 MAD (MACHINE LEARNING, ARTIFICIAL INTELLIGENCE & DATA) LANDSCAPE](/static/img/articles/become-the-maestro-of-your-mlops-abstractions/02.webp) [THE 2023 MAD (MACHINE LEARNING, ARTIFICIAL INTELLIGENCE & DATA) LANDSCAPE](https://mattturck.com/landscape/mad2023.pdf) Direct integration of each component in a project poses several risks. On one hand, it can decelerate project development as each modification or addition demands extensive refactoring. On the other hand, changes might be necessitated due to shifts in organizational infrastructure, vendor negotiations, or the need to accommodate new technology stacks. **Abstractions** offer a solution to these challenges. By utilizing flexible abstractions, you can facilitate the addition of new components over time, avoiding direct implementation dependencies. This architectural approach is so prevalent in our industry that it’s encapsulated in a gold principle known as [**SOLID**](https://en.wikipedia.org/wiki/SOLID): - ***S*ingle Responsibility Principle **— A class should have one, and only one, reason to change. - ***O*pen/Closed Principle** — Software entities should be open for extension, but closed for modification. - ***L*iskov Substitution Principle** — Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program. - ***I*nterface Segregation Principle** — No client should be forced to depend on methods it does not use. - ***D*ependency Inversion Principle **— High-level modules should not depend on low-level modules. Both should depend on abstractions. ### Why should I own my MLOps abstractions? Creating your own abstractions in MLOps is essential due to [the absence of a universal standard](https://mlops.community/we-need-posix-for-mlops/) like [POSIX for UNIX systems](https://en.wikipedia.org/wiki/POSIX) in our industry. Diverse model libraries such as Scikit-learn, XGBoost, PyTorch, Lightning, TensorFlow, and JAX present different interfaces for managing AI/ML models. Similarly, platforms like GCP, Azure, and Databricks offer varied toolkits for training and serving models. Relying solely on a single toolkit or platform can stifle innovation and confine you to solutions with limited control. ![Photo by Martijn Baudoin on Unsplash](/static/img/articles/become-the-maestro-of-your-mlops-abstractions/03.webp) Photo by [Martijn Baudoin](https://unsplash.com/@martijnbaudoin?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) By developing your own abstractions, you gain the flexibility to adopt solutions at your pace, conceal unnecessary complexities, and achieve interoperability by integrating multiple systems as needed. The investment in creating your own abstractions, if executed effectively, is manageable. Consider the universality of USB-C for computer I/O or Docker for application portability as analogous examples. However, it’s important to be cautious of ‘[leaky abstractions](https://en.wikipedia.org/wiki/Leaky_abstraction)’, which can adversely impact your project. Crafting effective abstractions requires time and expertise. When uncertain, it’s advisable to start with minimal abstractions and expand them gradually, tailoring them to your project’s core components. Avoid over-anticipating your abstraction needs. Instead, address issues as they emerge and devise solutions that are most appropriate for the immediate challenges at hand. ### Design Patterns 101 [**Design patterns**](https://en.wikipedia.org/wiki/Design_Patterns) serve as invaluable tools for creating abstractions, offering standardized solutions to recurrent problems. These patterns simplify the design process by providing proven techniques adaptable to various contexts, including MLOps. Let’s delve into the three primary types of design patterns and their application in MLOps environments. I also encourage you to check [this article from Laszlo SRAGNER](https://laszlo.substack.com/p/you-only-need-2-design-patterns-to) and [his slides from PyData London 2022](https://laszlo.substack.com/p/slides-for-my-talk-at-pydata-london) if you want to learn more about design patterns for MLOps. #### Behavioral pattern: Strategy (Abstract) The [**Strategy pattern**](https://en.wikipedia.org/wiki/Strategy_pattern) is instrumental in MLOps for its ability to **abstract** the “what” from the “how” in algorithmic operations. In MLOps, this translates to separating the goal (e.g., model training) from the method (e.g., using TensorFlow, XGBoost, or PyTorch). By employing the Strategy pattern, different algorithms or frameworks can be interchanged without altering the code structure, adhering to the Open/Closed Principle. This flexibility is crucial for MLOps, allowing dynamic adaptations like switching models or data sources based on runtime conditions. ![Strategy pattern : define a Model interface to switch between AI/ML frameworks at runtime](/static/img/articles/become-the-maestro-of-your-mlops-abstractions/04.webp) **Strategy pattern**: define a Model interface to switch between AI/ML frameworks at runtime #### Creational pattern: Factory (Adapt) Following the establishment of common interfaces, the [**Factory pattern**](https://en.wikipedia.org/wiki/Factory_method_pattern) becomes essential for dynamically **adapting** program behavior at runtime. This pattern facilitates object creation, enabling control through external configurations. In MLOps, this means allowing users to modify the scope and settings of AI/ML pipelines without changing the codebase. Python’s dynamic nature, coupled with tools like [Pydantic](https://docs.pydantic.dev/latest/) and its [Discriminated Union](https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions-with-str-discriminators) feature, simplifies implementing this pattern, enhancing user input validation and program object instantiation. ![Factory pattern : instantiate Model objects from model name using external configurations](/static/img/articles/become-the-maestro-of-your-mlops-abstractions/05.webp) **Factory pattern**: instantiate Model objects from model name using external configurations #### Structural pattern: Adapter (Overcome) Given the lack of a unified standard in MLOps, the [**Adapter pattern**](https://en.wikipedia.org/wiki/Adapter_pattern) plays a critical role. It acts as a bridge between disparate systems, **overcoming** incompatible interfaces. This pattern is particularly valuable in MLOps for integrating various external components, such as training and inference systems, across different platforms (like Databricks and Kubernetes for instance). By employing an Adapter, calls from one system can be translated and made compatible with another, ensuring seamless integration and generalization of external components. ![Adapter pattern : translate torch and sklearn interfaces to your own project abstractions](/static/img/articles/become-the-maestro-of-your-mlops-abstractions/06.webp) **Adapter pattern**: translate torch and sklearn interfaces to your own project abstractions ### Examples To illustrate the design patterns highlighted in the previous section, let’s take 3 examples to motivate their uses and their benefits for MLOps projects. #### Case 1: ZenML (Full-Stack) [**ZenML**](https://www.zenml.io/) emerges as [a comprehensive solution for integrating MLOps components](https://www.zenml.io/blog/zenml-why-we-built-it). Rather than developing bespoke abstractions, ZenML’s built-in offerings facilitate a swift initiation of your projects, allowing for custom abstractions as needed. This represents a significant undertaking by the ZenML team, as it involves [integrating and adapting various libraries to meet standard specifications](https://www.zenml.io/integrations). ZenML proves especially valuable in ecosystems that utilize diverse software stacks (e.g., Databricks, SageMaker, Kubernetes, etc.), or when there’s a need to rapidly switch or incorporate other solutions. It exemplifies how design patterns can simplify user experiences and harmonize development environments. ![Become the maestro of your MLOps abstractions](/static/img/articles/become-the-maestro-of-your-mlops-abstractions/07.webp) #### Case 2: Evidently (Monitoring) My recent adoption of [**Evidently**](https://www.evidentlyai.com/) to power a monitoring stack was prompted by a review of vendor solutions that [lacked essential features](https://fmind.medium.com/is-ai-ml-monitoring-just-data-engineering-10a2525a9c73). Evidently offers notable capabilities, [such as generating metrics and dashboards](https://docs.evidentlyai.com/examples/introduction) that can be ingested in our MLOps platform: ```python report = Report(metrics=[ DataDriftPreset(), ]) report.run(reference_data=reference, current_data=current) report ``` ![Data Drift report overview](/static/img/articles/become-the-maestro-of-your-mlops-abstractions/08.webp) [Data Drift report overview](https://docs.evidentlyai.com/examples/introduction) However, Evidently [does not support tabular exports yet](https://docs.evidentlyai.com/docs/library/output_formats). While contributing to Evidently’s codebase is appealing, employing design patterns strategically is imperative for the short-term adoption of the solution and safeguarding our codebase [against external uncertainties](https://web.archive.org/web/20240203173056/https://www.seldon.io/resources/licensing-faqs). #### Case 3: MLOps Python Package (Development) I recently introduced a GitHub repository showcasing tips for crafting an [**MLOps Python Package**](https://www.fmind.dev/articles/a-great-mlops-project-should-start-with-a-good-python-package/). This [repository](https://github.com/fmind/mlops-python-package) exemplifies some patterns discussed in this article. For example, the package offers [an interface for unifying AI/ML models](https://github.com/fmind/mlops-python-package/blob/618b83a0f1d8f218d131a8f415e95a44929654af/src/wines/models.py) from various frameworks (e.g., sklearn, pytorch), demonstrating the [strategy pattern](https://en.wikipedia.org/wiki/Strategy_pattern). ```python import abc import typing as T import pydantic as pdt from sklearn import ensemble, pipeline ParamKey = str ParamValue = T.Any Params = T.Dict[ParamKey, ParamValue] class Model(abc.ABC, pdt.BaseModel): """Base class for a model.""" KIND: str def get_params(self, deep: bool = True) -> Params: """Get the model params.""" params: Params = {} for key, value in self.dict().items(): if not key.startswith("_") and not key.isupper(): params[key] = value return params def set_params(self, **params: ParamValue) -> "Model": """Set the model params in place.""" for key, value in params.items(): setattr(self, key, value) return self @abc.abstractmethod def fit(self, inputs: schemas.Inputs, target: schemas.Target) -> "Model": """Fit the model on the given inputs and target.""" @abc.abstractmethod def predict(self, inputs: schemas.Inputs) -> schemas.Output: """Generate an output with the model for the given inputs.""" ``` It also features a [high-level job API](https://github.com/fmind/mlops-python-package/blob/618b83a0f1d8f218d131a8f415e95a44929654af/src/wines/jobs.py), enabling users to alter the task type via configuration files. The package readily supports Tuning, Training, and Inference jobs, showcasing the [factory pattern](https://en.wikipedia.org/wiki/Factory_method_pattern)’s role in initiating programs based on external settings. ```python import abc import typing as T import pydantic as pdt Locals = T.Dict[str, T.Any] class Job(abc.ABC, pdt.BaseModel): """Base class for a job.""" KIND: str @abc.abstractmethod def run(self) -> Locals: """Run the job in context.""" class TrainingJob(Job): """Train and register a single AI/ML model""" KIND: T.Literal["TrainingJob"] = "TrainingJob" inputs: datasets.DatasetKind target: datasets.DatasetKind model: models.ModelKind = models.BaselineSklearnModel() metric: metrics.MetricKind = metrics.SklearnMetric() def run(self) -> Locals: """Run the training job in context.""" # lots of code here ... return locals() ``` Lastly, the package incorporates an [adapter pattern](https://en.wikipedia.org/wiki/Adapter_pattern) for [configuration file loading](https://github.com/fmind/mlops-python-package/blob/618b83a0f1d8f218d131a8f415e95a44929654af/src/wines/configs.py). Although [OmegaConf](https://omegaconf.readthedocs.io/) can parse and merge YAML files, it lacks native support for cloud storage like S3 or GCP. Integrating with the [cloudpathlib](https://cloudpathlib.drivendata.org/stable/) package allows for configuration file access from any location, seamlessly hiding internal complexities. ```python import typing as T from cloudpathlib import AnyPath from omegaconf import DictConfig, ListConfig, OmegaConf Config = T.Union[ListConfig, DictConfig] def load_config(path: str) -> Config: """Load a configuration file.""" any_path = AnyPath(path) text = any_path.read_text() config = OmegaConf.create(text) return config ``` ### Conclusions: Abstract, Adapt, Overcome This article exemplifies how design patterns can be effectively utilized to evolve MLOps codebases and facilitate the integration of external components. This approach aligns with [David J. Wheeler’s foundational theorem of software development](https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering), which posits, _‘We can solve any problem by introducing an extra level of indirection’._ Crafting robust abstractions requires time and [thoughtful consideration](https://www.youtube.com/watch?v=f84n5oFoZBc); often, it’s wiser to start without them and then implement them to encapsulate emerging patterns. ![Become the maestro of your MLOps abstractions](/static/img/articles/become-the-maestro-of-your-mlops-abstractions/09.webp) While abstractions offer significant benefits, they cannot fully address the ‘curse of components’ inherent in the [MLOps landscape](https://mattturck.com/landscape/mad2023.pdf). The sheer complexity of integrating every conceivable solution in the market with others presents a daunting, perhaps intractable, challenge. This complexity could potentially stifle innovation and hinder long-term integration efforts. The concept of a **‘** [**Language of the System**](https://www.youtube.com/watch?v=ROor6_NGIWU) **’**, as articulated by Rich Hickey in his insightful talk, could be a game-changer for MLOps in this regard. Until such a paradigm emerges, the best practice is to maintain strong abstractions and strive for simplicity in your codebase. # An error occurred. Unable to execute JavaScript. The Language of the System — Rich Hickey ### [How to configure VS Code for AI, ML and MLOps development in Python 🛠️️](https://www.fmind.dev/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/) [Visual Studio Code](https://code.visualstudio.com/) is a remarkable application. In the past, developers faced a choice between simple, lightweight [text editors](https://en.wikipedia.org/wiki/List_of_text_editors) like Vim, Atom, and Emacs or powerful yet complex [IDEs](https://en.wikipedia.org/wiki/Comparison_of_integrated_development_environments) such as PyCharm and Eclipse. I can recall a time when running my IDE (Netbeans), my browser (Firefox), and my application (Java) simultaneously on a laptop with just 512MB of RAM was a challenge. Today, with VS Code, programmers have access to a versatile, modern, powerful, open-source tool well-supported by the community. But possessing a good tool is only the beginning. **Just as a skilled craftsman must master their tools, it’s essential for programmers to become proficient with their editor for enhanced productivity**. In this article, I will outline the steps for configuring VS Code for data scientists and machine learning engineers. I’ll start by listing extensions that augmente your programming environment. Then, I will share some settings and keybindings to enhance your development experience. Finally, I’ll provide tips and tricks to boost your coding efficiency with VS Code. ![Photo by Andrew Ruiz on Unsplash](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/cover.webp) Photo by [Andrew Ruiz](https://unsplash.com/@andrewruiz?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### Extensions 🔌 This section lists the extensions you can install from [VS Code Marketplace](https://marketplace.visualstudio.com/VSCode). #### A Tier: The Must - [**donjayamanne.python-extension-pack**](https://marketplace.visualstudio.com/items?itemName=donjayamanne.python-extension-pack) **: ultimate pack for Python development: autoDocString, Python, Jinja, IntelliCode, Python Indent, Python Environment Manager, Django.** - [**donjayamanne.python-environment-manager**](https://marketplace.visualstudio.com/items?itemName=donjayamanne.python-environment-manager): access and manage your Python environments (venv). - [**KevinRose.vsc-python-indent**](https://marketplace.visualstudio.com/items?itemName=KevinRose.vsc-python-indent): improve the default indentation for Python files. - [**ms-python.black-formatter**](https://marketplace.visualstudio.com/items?itemName=ms-python.black-formatter): format your code automatically. - [**ms-python.isort**](https://marketplace.visualstudio.com/items?itemName=ms-python.isort) **:** format your import statements automatically. - [**ms-python.mypy-type-checker**](https://marketplace.visualstudio.com/items?itemName=ms-python.mypy-type-checker): source typing for Python. Great to validate your code and communicate it should be used. - [**ms-python.pylint**](https://marketplace.visualstudio.com/items?itemName=ms-python.pylint): linting support for Python. Improve your code quality and avoid code smells. - [**ms-python.python**](https://marketplace.visualstudio.com/items?itemName=ms-python.python): language support for Python (linting, debugging, code formatting, and more). - [**ms-python.vscode-pylance**](https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance): enhanced language server for Python (static checker, intellicode, …). - [**ms-toolsai.jupyter**](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter): extension pack for Jupyter Notebooks: Keymap, Slideshow, Notebook Renderer, and Cell Tags. - [**ms-toolsai.jupyter-keymap**](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter-keymap): keybindings from Jupyter Notebooks. - [**ms-toolsai.jupyter-renderers**](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter-renderers): render notebook outputs (e.g., plots). - [**ms-toolsai.vscode-jupyter-cell-tags**](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.vscode-jupyter-cell-tags): edit cell tags in notebooks (i.e., metadata used by some plugins). #### B Tier: The Great - [**alefragnani.project-manager**](https://marketplace.visualstudio.com/items?itemName=alefragnani.project-manager): organize, manage, and access VS Code workspaces and Git repositories. - [**ms-azuretools.vscode-docker**](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker): manage and connect to Docker through VS Code. - [**ms-kubernetes-tools.vscode-kubernetes-tools**](https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools): everything you need to develop Kubernetes applications from VS Code. - [**ms-vscode-remote.remote-containers**](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers): develop your code from a Docker container instead of your local system. - [**ms-vscode-remote.remote-ssh**](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh): connect to a remote instance for editing files and developing your project. - [**ms-vscode-remote.remote-ssh-edit**](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh-edit): language support for SSH configuration files. - [**ms-vscode-remote.vscode-remote-extensionpack**](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack): extension pack for remote developments: Dev Containers, Remote SSH, Remote Tunnels, WSL. - [**ms-vscode.remote-explorer**](https://marketplace.visualstudio.com/items?itemName=ms-vscode.remote-explorer): view remote machines (SSH) and tunnels. - [**ms-vscode.remote-repositories**](https://marketplace.visualstudio.com/items?itemName=ms-vscode.remote-repositories): remotely browse and edit git repositories. - [**ms-vsliveshare.vsliveshare**](https://marketplace.visualstudio.com/items?itemName=MS-vsliveshare.vsliveshare): edit code from your colleague computer (and vice versa). Useful for collaboration and troubleshooting. - [**mutantdino.resourcemonitor**](https://marketplace.visualstudio.com/items?itemName=mutantdino.resourcemonitor): display the resource usage of your system in the status bar (CPU, RAM, …). Always useful for data projects. - [**njpwerner.autodocstring**](https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring): automatically generate Python docstrings from object definitions. - [**redhat.vscode-yaml**](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml): YAML language support. Great file format for configuring your application. - [**streetsidesoftware.code-spell-checker**](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker): highlight and fix spelling mistakes in your code. - [**tamasfe.even-better-toml**](https://marketplace.visualstudio.com/items?itemName=tamasfe.even-better-toml): TOML language support. Improve the edition of your`pyproject.toml` file. - [**usernamehw.errorlens**](https://marketplace.visualstudio.com/items?itemName=usernamehw.errorlens): display warnings and errors next to your code (instead of a dedicated window). - [**VisualStudioExptTeam.vscodeintellicode**](https://marketplace.visualstudio.com/items?itemName=VisualStudioExptTeam.vscodeintellicode): AI-assisted development features for VS Code. - [**yzhang.markdown-all-in-one**](https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one): tons of feature for editing Markdown files (shortcuts, table of content, language support, …). #### C Tier: The Good - [**aaron-bond.better-comments**](https://marketplace.visualstudio.com/items?itemName=aaron-bond.better-comments): highlight comments based on a prefix (e.g., \*, !, ?, TODO, …). - [**bierner.markdown-mermaid**](https://marketplace.visualstudio.com/items?itemName=bierner.markdown-mermaid): display [Mermaid](https://mermaid.js.org/) diagrams in Markdown. Great to share and visualize complex concepts and design decisions. - [**christian-kohler.path-intellisense**](https://marketplace.visualstudio.com/items?itemName=christian-kohler.path-intellisense): autocomplete file paths on your system. - [**dchanco.vsc-invoke**](https://marketplace.visualstudio.com/items?itemName=dchanco.vsc-invoke): execute [Invoke](https://www.pyinvoke.org/) tasks from VS Code (alternative to [GNU Make](https://www.gnu.org/software/make/manual/make.html)). - [**donjayamanne.githistory**](https://marketplace.visualstudio.com/items?itemName=donjayamanne.githistory): visualize your Git history (files, branches, commits, …). - [**eamodio.gitlens**](https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens): enhanced your git experience (e.g., git blame, code lens, …). - [**GitHub.remotehub**](https://marketplace.visualstudio.com/items?itemName=GitHub.remotehub): remotely browse and edit GitHub repositories. - [**github.vscode-github-actions**](https://marketplace.visualstudio.com/items?itemName=GitHub.vscode-github-actions): manage GitHub Actions workflows from VS Code. - [**GitHub.vscode-pull-request-github**](https://marketplace.visualstudio.com/items?itemName=GitHub.vscode-pull-request-github): manage GitHub Pull Requests from VS Code. - [**Gruntfuggly.todo-tree**](https://marketplace.visualstudio.com/items?itemName=Gruntfuggly.todo-tree): view all TODO, FIXME, and other annotations from a Tab. - [**IBM.output-colorizer**](https://marketplace.visualstudio.com/items?itemName=IBM.output-colorizer): syntax highlighting for log files. - [**jebbs.plantuml**](https://marketplace.visualstudio.com/items?itemName=jebbs.plantuml): create UML diagrams to document your project. - [**mechatroner.rainbow-csv**](https://marketplace.visualstudio.com/items?itemName=mechatroner.rainbow-csv): colorize the columns of CSV files to improve their readability. - [**mhutchie.git-graph**](https://marketplace.visualstudio.com/items?itemName=mhutchie.git-graph): view and edit your git graph (tags, branches, stashes, …). - [**mikestead.dotenv**](https://marketplace.visualstudio.com/items?itemName=mikestead.dotenv): syntax highlighting for dotenv files (i.e., contain the environment variables of your program). - [**ms-vscode.live-server**](https://marketplace.visualstudio.com/items?itemName=ms-vscode.live-server): host a local server for web development. - [**oderwat.indent-rainbow**](https://marketplace.visualstudio.com/items?itemName=oderwat.indent-rainbow): make code indentation more readable. - [**pomdtr.excalidraw-editor**](https://marketplace.visualstudio.com/items?itemName=pomdtr.excalidraw-editor): edit Excalidraw diagrams. Great to design architecture diagrams during brainstorm sessions. - [**sleistner.vscode-fileutils**](https://marketplace.visualstudio.com/items?itemName=sleistner.vscode-fileutils): create, copy, move, rename, and delete files from VS Code commands. - [**vsls-contrib.gistfs**](https://marketplace.visualstudio.com/items?itemName=vsls-contrib.gistfs): manage and share code snippets on GitHub Gist. - [**wayou.vscode-todo-highlight**](https://marketplace.visualstudio.com/items?itemName=wayou.vscode-todo-highlight): highlight TODO, FIXME and other keywords in your comments. - [**wholroyd.jinja**](https://marketplace.visualstudio.com/items?itemName=wholroyd.jinja): language support for Jinja. Popular template language for substituting variables in text files. #### S Tier: The Super (Optional) Three features have remarkably enhanced my programming productivity throughout my career: [VIM](https://marketplace.visualstudio.com/items?itemName=vscodevim.vim), [Terminal](https://code.visualstudio.com/docs/terminal/basics), and [GitHub Copilot](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot). **Each of these features contributes to a minimum of a 30% increase in my daily output**. Yet, they may be one of the most difficult VS Code features to master. #### [Vim](https://marketplace.visualstudio.com/items?itemName=vscodevim.vim) 📝 [If we were living in a simulation, I’m convinced that Vim (and LISP) would be the tools used to program it](https://xkcd.com/224/). Programmers spend a significant portion of their day working with text, and Vim allows you to manipulate text at an astonishing pace. Instead of moving the cursor with a mouse, you can perform high-level operations with just a few keystrokes. For example, “Move the cursor to the next word” is ‘w,’ “delete the current line” is ‘dd,’ or “copy the following line seven times” is ‘yy7p.’ These operations can be combined to create complex commands, such as “Replace all the content between the current parentheses” using ‘ci(‘. The possibilities are virtually limitless, to the extent that there are even [games designed to teach Vim usage](https://vim-adventures.com/)! ![Vim graphical cheat sheet](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/02.webp) Vim Cheatsheet: [http://www.viemu.com/a_vi_vim_graphical_cheat_sheet_tutorial.html](https://web.archive.org/web/20200108045441/http://www.viemu.com/a_vi_vim_graphical_cheat_sheet_tutorial.html) Allow me to emphasise the value of Vim with an anecdote. During my master’s degree, I encountered a configuration issue with my Apache HTTP Server and sought help from my teacher, a sysadmin at Peugeot (now Stellantis). He used Vim, and his actions were so rapid that I couldn’t keep up. In a matter of minutes, he had troubleshooted my problem. That’s when I made the decision to master Vim and the Linux ecosystem. The best part is that you can use Vim within VS Code as your primary or secondary keybinding. All it takes is installing the [VSCodeVim](https://marketplace.visualstudio.com/items?itemName=vscodevim.vim) extension and configuring some settings. Keep in mind that mastering Vim requires time, effort, and practice, but if you plan to edit lots of code files in the near future, it’s a valuable investment! **Bonus: You can use Vim in your Browser with the** [**Vimum**](https://github.com/philc/vimium) **extension.** - [**vscodevim.vim**](https://marketplace.visualstudio.com/items?itemName=vscodevim.vim): Vim emulation for VS Code, highly recommended! - [**VSpaceCode.vspacecode**](https://marketplace.visualstudio.com/items?itemName=VSpaceCode.vspacecode): Spacemacs like keybindings (advanced shortcuts relying on Vim leader key). #### [Terminal](https://code.visualstudio.com/docs/terminal/basics) 👨‍💻 Before switching to VS Code, my tools of choice were [NeoVim](https://neovim.io/) (or [Spacemacs](https://www.spacemacs.org/)), and the terminal served as my IDE. A terminal provides access to an extensive array of tools and commands (cp, tee, awk, grep, find, and more) that can be combined in countless ways. Need to sort and deduplicate all your files? Just use `find . -type f | sort | uniq`. Want to resize a batch of images? You can do it with `find . -type f -name "*.jpg" -exec mogrify -resize 100x100 {} \;`. **Although the terminal isn’t an extension, it’s a vital feature of VS Code, and its power should not be underestimated**. How many times have I needed to troubleshoot issues or perform advanced tasks that would have required writing complex programs. The terminal’s flexibility is a true asset! ![Command Line Fu: https://xkcd.com/196/](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/03.webp) Command Line Fu: [https://xkcd.com/196/](https://xkcd.com/196/) If you’re interested in configuring your terminal, numerous individuals share their “dotfiles” repositories on GitHub. These repositories contain the tools and configurations they use to enhance their command-line interface. You can find mine at this address: [https://github.com/fmind/dotfiles](https://github.com/fmind/dotfiles). Please note that to install them, you’ll need both [Python](https://www.python.org/downloads/) and [Ansible](https://www.ansible.com/). #### [GitHub Copilot](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot) ✈️ [**GitHub Copilot**](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot) **is a game changer for programmers**. Once you figure out the program in your head, you have to type it and the time spent between thinking to typing maybe 1:10. What if you could just “Tab” you way out? Instead of writing the text yourself, express your intent with clear names or comments, and let GitHub Copilot write the rest for you by pressing “Tab”. This is an autocompletion engine cranked up to 9999, and it is even better now with its new [GitHub Copilot Chat](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat) feature! ![Example of GitHub Copilot in action: https://github.com/features/copilot](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/04.mp4) Example of GitHub Copilot in action: [https://github.com/features/copilot](https://github.com/features/copilot) GitHub Copilot proves invaluable in various scenarios: it assists in generating unit tests from code definitions, completes unfamiliar API calls, and creates relevant examples based on your context. It’s intelligent enough to examine your project’s files and adapt to your coding style. This experience is both gratifying and highly productive, and I strongly recommend trying Copilot as soon as possible! - [**GitHub.copilot**](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot): The smartest autocompletion engine ever! - [**GitHub.copilot-chat**](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat): don’t ask you colleague, ask Copilot Chat instead :) ### Settings and Keybindings 🤖 Within VS Code, there are two sets of configurations: [Settings](https://code.visualstudio.com/docs/getstarted/settings) that control the application’s behavior and [Keybindings](https://code.visualstudio.com/docs/getstarted/keybindings) that modify your shortcuts. Both can be edited from the “Gear” icon located at the bottom left of VS Code. #### [Settings](https://gist.github.com/fmind/67c1ce34d0f21005b4aa7aa79b9162aa#file-settings-json) 🔧 Below, you’ll find [the link to the settings I utilize with VS Code](https://gist.github.com/fmind/67c1ce34d0f21005b4aa7aa79b9162aa#file-settings-json). Each setting is annotated to help you understand what is does. Keep in mind that you can hover your cursor over a setting key in VS Code to access more information about that setting. You can also switch between the textual and visual interface using the “Go to document” icon at the top left of the editor settings. **Remember that it’s essential to tailor the settings to your preferences since every user is unique**. Additionally, strive to understand what each setting does to optimize your satisfaction and productivity. ![https://gist.github.com/fmind/67c1ce34d0f21005b4aa7aa79b9162aa#file-settings-json (163 lines)](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/05.webp) [https://gist.github.com/fmind/67c1ce34d0f21005b4aa7aa79b9162aa#file-settings-json](https://gist.github.com/fmind/67c1ce34d0f21005b4aa7aa79b9162aa#file-settings-json) (163 lines) #### [Keybindings](https://gist.github.com/fmind/67c1ce34d0f21005b4aa7aa79b9162aa#file-keybindings-json) 🎹 **Keybindings are primarily a matter of personal preference**. Some individuals may opt for minor modifications, while others may choose to completely rebind commands to their liking. In my case, I prefer to assign quick shortcuts to the most frequently used commands and utilize the command selection (CTRL+SHIFT+P) for less common commands. To execute these quick shortcuts, I employ a technique I call the “Alt-key trick.” You’re welcome to adopt this approach if you find it beneficial or explore alternatives like [VSpaceCode](https://vspacecode.github.io/) (Spacemacs keybindings for VS Code). #### The Alt-key Trick 🗡 By default, most desktop applications use the Alt key to navigate through the application menu, allowing actions like “Alt+i i” to access the “Insert” menu and insert an image into a Word document. In VS Code, you can disable this behavior by setting `"window.enableMenuBarMnemonics": false` and create your own custom keybindings. Another advantage is that this behavior works across all major operating systems: Linux, Mac, and Windows. [The code snippet provided below replaces Alt-key shortcuts with custom keybindings](https://gist.github.com/fmind/67c1ce34d0f21005b4aa7aa79b9162aa#file-keybindings-json). For example, pressing “Alt+j” will move to the next editor, and “Alt+,” will open a new terminal. The code snippet is divided into several sections: 1. **ACTIONS**: Primary shortcuts associated with each key on your keyboard. 2. **DISABLED**: Resolving conflicts found in the extensions I’m using. 3. **JUPYTER**: Additional shortcuts for Jupyter. 4. **POPUPS**: Navigating popup menus similar to Vim. 5. **TERMINALS**: Shortcuts related to terminal manipulation. ![https://gist.github.com/fmind/67c1ce34d0f21005b4aa7aa79b9162aa#file-keybindings-json (300 lines)](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/06.webp) [https://gist.github.com/fmind/67c1ce34d0f21005b4aa7aa79b9162aa#file-keybindings-json](https://gist.github.com/fmind/67c1ce34d0f21005b4aa7aa79b9162aa#file-keybindings-json) (300 lines) ### Tips and Tricks 👉 #### 1. Synchronization ↔️ Leverage VS Code’s ‘[Settings Sync](https://code.visualstudio.com/docs/editor/settings-sync)’ feature for a seamless development experience across multiple machines. Activate this by clicking the ‘Gear’ icon at the bottom-left of the application to synchronize your configurations, facilitating effortless transitions between different [work profiles](https://code.visualstudio.com/docs/editor/profiles) like Java/Python or personal/professional. ![https://code.visualstudio.com/docs/editor/settings-sync](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/07.webp) [https://code.visualstudio.com/docs/editor/settings-sync](https://code.visualstudio.com/docs/editor/settings-sync) #### 2. Command Palette 🎨 Quickly access VS Code’s shortcuts through the [Command Palette](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette) by using ‘CTRL+SHIFT+P’. Ideal for infrequent commands or when the exact name eludes you. This feature is an essential tool in the VS Code arsenal! ![https://code.visualstudio.com/docs/getstarted/userinterface#\_command-palette](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/08.webp) [https://code.visualstudio.com/docs/getstarted/userinterface#\_command-palette](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette) #### 3. Put the mouse away 🐭 Optimize your coding workflow by mastering keyboard shortcuts; [they surpass the mouse in boosting typing efficiency](https://blog.superhuman.com/keyboard-vs-mouse/). For instance, Vim enthusiasts can adopt the recommended [hand positioning below to enhance performance](https://stackoverflow.com/questions/14021283/position-of-fingers-for-better-productivity-in-vim). Consider [remapping the Caps Lock key to Ctrl (Windows/Linux) or Cmd (Mac)](https://www.endpointdev.com/blog/2023/04/use-caps-lock-as-escape/) to prevent finger strain. ![https://stackoverflow.com/questions/14021283/position-of-fingers-for-better-productivity-in-vim](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/09.gif) [https://stackoverflow.com/questions/14021283/position-of-fingers-for-better-productivity-in-vim](https://stackoverflow.com/questions/14021283/position-of-fingers-for-better-productivity-in-vim) #### 4. Code Workspace File 🗄️ Customize your VS Code environment globally, by folder, or using a workspace. [Workspaces](https://code.visualstudio.com/docs/editor/workspaces) bundle folders for project management, ensuring consistent settings among developers. Create one via ‘File’ \> ‘Save Workspace As…’ to generate a ‘code-workspace’ file, typically placed at your project’s root.” ![VS Code Workspace for the MLOps Python Package project on GitHub](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/10.webp) VS Code Workspace for the MLOps Python Package project on GitHub #### 5. Python Interactive Window 🕹️ Transform your Python script into an interactive notebook within VS Code using the [Python Interactive Window](https://code.visualstudio.com/docs/python/jupyter-support-py). Insert ‘# %%’ comments to delineate cells. Edit on the left, execute with ‘Ctrl+Enter’, and view results on the right. You can greatly streamline your coding with this simple integration! ![Python Interactive Window on a simple script](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/11.webp) Python Interactive Window on a simple script #### 6. Automation with Make or Invoke 🤖 Automate repetitive commands with a task automation system, directly through VS Code or your shell, to enhance productivity. Consider these three systems: - [Make](https://www.gnu.org/software/make/): A time-tested tool compatible across all platforms. - [Invoke](https://www.pyinvoke.org/): Python specific. Great if you want to mix Python code and shell commands. You can improve VS Code integration with [this extension](https://marketplace.visualstudio.com/items?itemName=dchanco.vsc-invoke). - [VS Code Tasks](https://code.visualstudio.com/docs/editor/tasks): while native to VS Code, I favor Make or Invoke for better collaboration with other developers. ![Example of Invoke tasks in https://github.com/fmind/mlops-python-package](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/12.webp) Example of Invoke tasks in [https://github.com/fmind/mlops-python-package](https://github.com/fmind/mlops-python-package) #### 7. Xonsh, Fish, Zsh, PtPython, and others 🐚 Move beyond basic bash with these more advanced shells: - [Xonsh](https://xon.sh/): merges the familiarity of Python with shell capabilities — a harmonious blend for those preferring Python’s syntax over bash’s complexity. - [Fish](https://fishshell.com/): affers a superb out-of-the-box experience, though it may not always align with bash standards — a choice for the bash-averse :) - [Zsh](https://www.zsh.org/): outshines bash in power but requires fine-tuning. Supercharge it with [https://ohmyz.sh/](https://ohmyz.sh/) - [IPython](https://ipython.org/)/[PtPython](https://github.com/prompt-toolkit/ptpython): not for system shell use, but excellent for an enriched Python interpreter interface. ![Xonsh shell used for executing Python code and Make tasks https://xon.sh/](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/13.webp) Xonsh shell used for executing Python code and Make tasks [https://xon.sh/](https://xon.sh/) There are also tons of resources only to further extend your configuration: - [https://github.com/alebcay/awesome-shell](https://github.com/alebcay/awesome-shell) - [https://github.com/jorgebucaran/awsm.fish](https://github.com/jorgebucaran/awsm.fish) - [https://github.com/xonsh/awesome-xontribs](https://github.com/xonsh/awesome-xontribs) - [https://github.com/agarrharr/awesome-cli-apps](https://github.com/agarrharr/awesome-cli-apps) - [https://github.com/awesome-lists/awesome-bash](https://github.com/awesome-lists/awesome-bash) - [https://github.com/unixorn/awesome-zsh-plugins](https://github.com/unixorn/awesome-zsh-plugins) - [https://github.com/janikvonrotz/awesome-powershell](https://github.com/janikvonrotz/awesome-powershell) #### 8. Text file can be used for so many things 📄 With proficiency in VS Code, you’ll discover that text-based operations often outperform GUIs in productivity. Activities suited for a text-centric approach include: - Writing Slides with [Marp for VS Code](https://marketplace.visualstudio.com/items?itemName=marp-team.marp-vscode) - Creating UML diagrams with [PlantUML](https://marketplace.visualstudio.com/items?itemName=jebbs.plantuml) - Creating diagrams and charts with [Mermaid](https://marketplace.visualstudio.com/items?itemName=bierner.markdown-mermaid) - Managing and tracking your time with [Org Mode](https://marketplace.visualstudio.com/items?itemName=tootone.org-mode) - Generate graphs and concept maps with [GraphViz](https://graphviz.org/) ![Integration of Mermaid with VS Code https://marketplace.visualstudio.com/items?itemName=bierner.markdown-mermaid](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/14.webp) Integration of Mermaid with VS Code [https://marketplace.visualstudio.com/items?itemName=bierner.markdown-mermaid](https://marketplace.visualstudio.com/items?itemName=bierner.markdown-mermaid) #### 9. Look at other configurations on GitHub 😺 Developers love to publish their settings and keybindings on GitHub, and VS Code is no exception. You can check this [topic](https://github.com/topics/vscode-settings) to tracks VS Code configurations on GitHub. I also recommend you subscribe to these sources of tips and tricks: - [https://vscode.email/](https://vscode.email/) (newsletter) - [https://twitter.com/code](https://twitter.com/code) (official X account) - [https://code.visualstudio.com/updates](https://code.visualstudio.com/updates) (official blog) - [https://www.youtube.com/channel/UCs5Y5_7XK8HLDX0SLNwkd3w](https://www.youtube.com/channel/UCs5Y5_7XK8HLDX0SLNwkd3w) (official YouTube account) ![https://vscode.email/about](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/15.webp) [https://vscode.email/about](https://vscode.email/about) #### 10. Use GhostText to use VS Code on the web 👻 When VS Code’s reach seems limited, [GhostText](https://ghosttext.fregante.com/) emerges as an ally. This [Browser](https://chromewebstore.google.com/detail/ghosttext/godiecgffnchndlihlpaajjcplehddca)/[VS Code](https://marketplace.visualstudio.com/items?itemName=fregante.ghost-text) extension enables you to edit browser text areas directly in VS Code. It offers bi-directional sync, allowing edits anywhere, anytime. Disengage sync when your editing is complete. ![https://ghosttext.fregante.com/](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/16.webp) [https://ghosttext.fregante.com/](https://ghosttext.fregante.com/) ### Conclusions 🌅 **Congratulations on reaching the end of this guide!** Your dedication to refining your development environment is commendable. Remember, the crux of an optimal setup is personalization. Embrace and experiment with your tools’ potential to suit your unique preferences. Farewell, skilled developer, may your code be efficient and your professional path successful! ![Photo by Taylor Brandon on Unsplash](/static/img/articles/how-to-configure-vs-code-for-ai-ml-and-mlops-development-in-python/17.webp) Photo by [Taylor Brandon](https://unsplash.com/@house_42?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### [Forging a Personal Chatbot with OpenAI API, Chroma DB, HuggingFace Spaces, and Gradio 🔥](https://www.fmind.dev/articles/forging-a-personal-chatbot-with-openai-api-chroma-db-huggingface-spaces-and-gradio/) [If you have checked the Internet in 2023](https://www.zdnet.com/article/generative-ai-is-everything-everywhere-all-at-once/), you’re likely familiar with [Generative AI](https://en.wikipedia.org/wiki/Generative_artificial_intelligence). The [launch of ChatGPT](https://openai.com/blog/chatgpt) has sparked a surge in [interest](https://github.com/f/awesome-chatgpt-prompts), [investment](https://fortune.com/2023/08/30/chatgpt-creator-openai-earnings-80-million-a-month-1-billion-annual-revenue-540-million-loss-sam-altman/), and [innovative projects](https://autogpt.net/) focused on [Large Language Models](https://en.wikipedia.org/wiki/Large_language_model) (LLMs), [Artificial General Intelligence](https://en.wikipedia.org/wiki/Artificial_general_intelligence) (AGI), and [Retrieval Augmented Generation](https://www.promptingguide.ai/techniques/rag) (RAG). Yet, [navigating this burgeoning field can be challenging](https://web.archive.org/web/20230616022719/https://mlops.community/surveys/llm/): What tangible benefits does Generative AI offer? How complex is it to develop a Generative AI application? What kind of performance can one expect from such applications in real-world projects? **In this article, I outline a project I developed to field questions about my resume and deepen my understanding of Generative AI**. The first section introduces the use case, objectives, and tools chosen for the project. The second section details the steps involved in converting unstructured documents into Chroma DB, a vector database. The third section elaborates on how to construct and integrate a chatbot assistant utilizing the OpenAI API, HuggingFace Spaces, and Gradio. Finally, I delve into the merits of learning through practical experience, along with critical design choices — such as the deliberate decision not to employ LangChain for this project. ![Forging a Personal Chatbot with OpenAI API, Chroma DB, HuggingFace Spaces, and Gradio 🔥](/static/img/articles/forging-a-personal-chatbot-with-openai-api-chroma-db-huggingface-spaces-and-gradio/cover.webp) ### A Case for a Chatbot 💼 #### What? 💬 - **Develop a chatbot capable of answering questions related to my** [**LinkedIn profile**](https://www.linkedin.com/in/fmind-dev/) **.** - Process unstructured documents and rank them based on user queries. - Deploy the chatbot online and integrate it into [my website](https://www.fmind.dev/). #### Why? 📡 - **Acquire hands-on experience with new technology via a tangible project.** - Engage recruiters through more naturalized interactions. - Evaluate the readiness of Generative AI tooling. #### How? 🛠️ - **Large Language Model (LLM)**: [gpt-3.5-turbo-16k](https://platform.openai.com/docs/models) ([OpenAI](https://openai.com/)) - **Embedding Model**: [text-embedding-ada-002](https://platform.openai.com/docs/guides/embeddings) ([OpenAI](https://openai.com/)) - **Encoding Model**: cl100k_base ([tiktoken](https://github.com/openai/tiktoken)) - **Vector Database**: [Chroma DB](https://www.trychroma.com/) **Note**: [LangChain](https://www.langchain.com/) was deliberately not utilized (discussed later). ### From Data to Vector Database ↗️ #### Downloading my LinkedIn Page 🌐 Websites are keen to protect their assets, even when those assets are your personal information. To furnish my chatbot with sufficient data, I manually downloaded [my public LinkedIn profile](https://www.linkedin.com/in/fmind-dev/) and saved it as an HTML file. While automation could have been employed, I chose not to, in order to sidestep CAPTCHA challenges and other potential restrictions. This decision aligns well with the 80–20 rule, whereby 80% of the desired outcomes are achieved with just 20% of the effort. ![My public profile on LinkedIn: https://www.linkedin.com/in/fmind-dev/](/static/img/articles/forging-a-personal-chatbot-with-openai-api-chroma-db-huggingface-spaces-and-gradio/02.webp) My public profile on LinkedIn: [https://www.linkedin.com/in/fmind-dev/](https://www.linkedin.com/in/fmind-dev/) #### Converting HTML to Markdown 📑 HTML is a well-structured yet verbose and cluttered format, rife with attributes and styles. To streamline the content, I first converted the HTML file into a plain text file using [Pandoc](https://pandoc.org/). Subsequently, I manually transformed this text file into Markdown to enhance both its format and content. ```bash pandoc --to=plain --from=html --output=files/linkedin.txt files/linkedin.html ``` **Through my experimentation, I’ve found that data quality is pivotal in optimizing the performance of a language model**. Given the limited control one has over the flow of an LLM, ensuring that the data is correctly formatted and relevant significantly aids in validating the model’s behavior. ```markdown # Profile ## Overview - First name: Médéric - Last name: HURIER - Pseudo: Fmind - Followers: 4K - Location: Luxembourg, Luxembourg - Education: University of Luxembourg - Current position: Decathlon Technology - Public URL: www.linkedin.com/in/fmind-dev - Industry: Technology, Information and Internet - Address: 28 Avenue François Clément, 5612 Mondorf-les-Bains, Luxembourg - Headline: Freelancer | AI/ML/MLOps Engineer | Data Scientist | MLOps Community Organizer | OpenClassrooms Mentor | Hacker | PhD ``` #### Chunking and Importing Data to Chroma DB 🛢 **One of the primary constraints of Large Language Models is their context size**. As of October 2023, OpenAI’s [GPT-3.5 models](https://developers.openai.com/api/docs/models) support context sizes ranging from 4k to 16k tokens, while [GPT-4 models](https://platform.openai.com/docs/models/gpt-4) extend this range from 8k to 32k tokens. **In my evaluation, the cost-to-benefit ratio was more favorable for GPT-3.5 than for GPT-4.** ```markdown | Technical name | Model family | Price per 1000 tokens | Max tokens | | ----------------- | ------------ | ----------------------- | ---------- | | gpt-4-32k | GPT-4 | USD 0.0600 (prompt) | 32768 | | | | USD 0.1200 (completion) | | | gpt-4 | GPT-4 | USD 0.0300 (prompt) | 8192 | | | | USD 0.0600 (completion) | | | gpt-3.5-turbo-16k | GPT-3.5 | USD 0.0030 (prompt) | 16384 | | | | USD 0.0040 (completion) | | | gpt-3.5-turbo | GPT-3.5 | USD 0.0015 (prompt) | 4096 | | | | USD 0.0020 (completion) | | ``` To divide my Markdown document into manageable chunks, I developed a Python function that segments content based on header levels, easily achieved through regular expressions: ```python def segment_text(text: str, pattern: str) -> T.Iterator[tuple[str, str]]: """Segment the text in title and content pair by pattern.""" splits = re.split(pattern, text, flags=re.MULTILINE) pairs = zip(splits[1::2], splits[2::2]) return pairs segments_h1 = segment_text(text=text, pattern=r"^# (.+)") segments_h2 = segment_text(text=h1_text, pattern=r"^## (.+)") ``` Subsequently, I crafted another function to import these chunks into Chroma DB. [**Chroma DB**](https://www.trychroma.com/) **is particularly useful for straightforward applications, given its in-memory operation and simplicity**. The following code snippet outlines how to prepare the document, metadata, and identifiers for ingestion into a collection: ```python def import_file( file: T.TextIO, collection: lib.Collection, encoding_function: T.Callable, max_output_tokens: int = lib.ENCODING_OUTPUT_LIMIT, ): """Import a markdown file to a database collection.""" text = file.read() filename = file.name segments_h1 = segment_text(text=text, pattern=r"^# (.+)") for h1, h1_text in segments_h1: segments_h2 = segment_text(text=h1_text, pattern=r"^## (.+)") for h2, content in segments_h2: id_ = f"{filename} # {h1} ## {h2}" # unique doc id document = f"# {h1}\n\n## {h2}\n\n{content.strip()}" metadata = {"filename": filename, "h1": h1, "h2": h2} collection.add(ids=id_, documents=document, metadatas=metadata) ``` ### Building, Hosting, and Integrating a Chatbot 👨‍💻 #### Chatbot Function 🤖 The chatbot function serves as the system’s core. It is responsible for crafting responses based on the conversation history, the user’s input, and the context prompt. Additionally, the function can fetch pertinent information from the Vector Database to enrich the content (via RAG). **It’s crucial at this stage to not exceed the token limitations imposed by the LLM and to refrain from including irrelevant information for the end user (i.e., by filtering documents by their distance).** Once the responses are formulated, they are sent to the Large Language Model for processing. **In my case, I utilized** [**OpenAI’s GPT-3.5 16k model**](https://developers.openai.com/api/docs/models) **, as it offered the most favorable balance among model performance, context size, and cost per token**. The model’s output can then be displayed directly to the user: ```python PROMPT_CONTEXT = """ You are Fmind Chatbot, specialized in providing information regarding Médéric Hurier's (known as Fmind) professional background. Médéric is an MLOps engineer based in Luxembourg. He is currently working at Decathlon. His calendar is booked until the conclusion of 2024. Your responses should be succinct and maintain a professional tone. If inquiries deviate from Médéric's professional sphere, courteously decline to engage. You may find more information about Médéric below (markdown format): """ def answer(message: str, history: list[str]) -> str: """Answer questions about my resume.""" # counters n_tokens = 0 # messages messages = [] # - context n_tokens += len(ENCODING(PROMPT_CONTEXT)) messages += [{"role": "system", "content": PROMPT_CONTEXT}] # - history for user_content, assistant_content in history: n_tokens += len(ENCODING(user_content)) n_tokens += len(ENCODING(assistant_content)) messages += [{"role": "user", "content": user_content}] messages += [{"role": "assistant", "content": assistant_content}] # - message n_tokens += len(ENCODING(message)) messages += [{"role": "user", "content": message}] # database results = COLLECTION.query(query_texts=message, n_results=QUERY_N_RESULTS) distances, documents = results["distances"][0], results["documents"][0] for distance, document in zip(distances, documents): # - distance if distance > QUERY_MAX_DISTANCE: break # - document n_document_tokens = len(ENCODING(document)) if (n_tokens + n_document_tokens) >= PROMPT_MAX_TOKENS: break n_tokens += n_document_tokens messages[0]["content"] += document # response api_response = MODEL(messages=messages) content = api_response["choices"][0]["message"]["content"] # return return content ``` #### Chatbot Interface: Gradio 🖼️ Multiple avenues are available for constructing a chat interface, ranging from developing a full-fledged web application (e.g., using [Flask](https://flask.palletsprojects.com/en/3.0.x/)) to rapidly prototyping via [Streamlit](https://streamlit.io/) or [Dash](https://plotly.com/dash/). **In my scenario, I opted for** [**Gradio**](https://www.gradio.app/) **because it provides a prebuilt chat interface.** Despite its simplicity, [Gradio](https://www.gradio.app/) offers ample room for customization. I was able to fine-tune the interface by eliminating extraneous buttons, selecting an appropriate theme, and showcasing example interactions to the end user. ```python interface = gr.ChatInterface( fn=answer, theme=THEME, title="glass", examples=EXAMPLES, clear_btn=None, retry_btn=None, undo_btn=None, ) ``` #### Chatbot Hosting: HuggingFace Spaces 🗄️ **Choosing the right hosting option proved to be the most challenging decision**. While cloud providers like [AWS](https://aws.amazon.com/), [GCP](https://cloud.google.com/), and [Azure](https://azure.microsoft.com/en-us) offer scalable solutions, they come with a complex setup process involving permissions and public access. Conversely, I found [HuggingFace Spaces](https://huggingface.co/spaces) to be user-friendly, cost-effective, and natively supportive of Gradio. **Given that personal websites typically don’t demand high scalability, I opted for HuggingFace Spaces.** ![Hosting on HuggingFace Spaces: https://huggingface.co/spaces/fmind/resume](/static/img/articles/forging-a-personal-chatbot-with-openai-api-chroma-db-huggingface-spaces-and-gradio/03.webp) Hosting on HuggingFace Spaces: [https://huggingface.co/spaces/fmind/resume](https://huggingface.co/spaces/fmind/resume) #### Chatbot Integration: Embed with a Web Component 📲 The final stage involved integrating the chatbot into my website. This task was exceptionally straightforward, as [HuggingFace Spaces](https://huggingface.co/spaces) allows for the creation of a web component — an HTML snippet that can be seamlessly added to any site. ```html <script type="module" src="https://gradio.s3-us-west-2.amazonaws.com/3.46.0/gradio.js" ></script> <gradio-app src="https://fmind-resume.hf.space"></gradio-app> ``` And there you have it! [A personal chatbot is now integrated into my website, ready to field your questions](https://www.fmind.dev/). ![AI Chatbot integrated into my website: https://www.fmind.dev/](/static/img/articles/forging-a-personal-chatbot-with-openai-api-chroma-db-huggingface-spaces-and-gradio/04.webp) AI Chatbot integrated into my website: [https://www.fmind.dev/](https://www.fmind.dev/) ### Conclusions 🏁 #### Learning by Doing ✍ **Creating a personal AI assistant based on my LinkedIn profile was a deeply enriching experience**. It provided me an opportunity to explore new tools and APIs like Chroma DB, while revisiting familiar ones like OpenAI and Gradio. Moreover, I confronted challenges specific to RAG applications, including proper content chunking, data quality improvement, and the seamless integration of multiple tools into a comprehensive solution. #### To Use LangChain or Not? ⛓ [The utility of LangChain is a subject of debate within the community](https://minimaxir.com/2023/07/langchain-problem/). Although the framework offers several conveniences for managing LLMs, I found its practical application cumbersome. I spent more time navigating its documentation and grappling with its components than actually building my own. In my view, the abstractions it provides can be unduly complex relative to what can be accomplished more directly. #### To Infinity and Beyond with Gen AI! 💫 **The process of crafting a personal AI assistant was engaging, and educational**. My next steps involve fine-tuning a model and comparing its performance against a standard RAG application, [a topic also discussed within the MLOps Community](https://mlops-community.slack.com/archives/C04T55KFV8S/p1697887950188599). Now, more than ever, I recognize the transformative potential of Generative AI in both existing and emerging solutions. May we realize the full potential of Generative AI in our lifetime! ![DALL-E 3: Draw a picture of a bright future with Artificial Intelligence. I want something inspiring for a blog post on Medium.](/static/img/articles/forging-a-personal-chatbot-with-openai-api-chroma-db-huggingface-spaces-and-gradio/05.webp) DALL-E 3: Draw a picture of a bright future with Artificial Intelligence. I want something inspiring for a blog post on Medium. ### [Finding Harmony in MLOps: Balancing Functional and Object-Oriented Approaches ☯](https://www.fmind.dev/articles/finding-harmony-in-mlops-balancing-functional-and-object-oriented-approaches/) Programmers have always been passionate about their preferences, whether they discuss [spaces vs. tabs](https://alexkondov.com/indentation-warfare-tabs-vs-spaces/#:~:text=The%20Tab%20Character&text=It%20is%20interpreted%20by%20the,tab%20character%20represents%20multiple%20whitespaces.), [Vim vs. Emacs](https://en.wikipedia.org/wiki/Editor_war), or [light mode vs. dark mode](https://www.nngroup.com/articles/dark-mode/). These debates have withstood the test of time, indicating that there is a place for each solution, and no definitive argument can declare one superior over the other. However, when it comes to programming paradigms, the arguments tend to be more fervent. [Object-oriented languages](https://en.wikipedia.org/wiki/Object-oriented_programming) have [long dominated the programming landscape](https://www.tiobe.com/tiobe-index/), championing code reusability across various projects. In contrast, [functional programming](https://en.wikipedia.org/wiki/Functional_programming) has emerged as an alternative style in recent years, [promising code that is easier to reason with](https://www.youtube.com/watch?v=SxdOUGdseq4). When delving into machine learning projects, the question arises: **which paradigm is best suited for building an MLOps application?** ![Photo by Alex Padurariu on Unsplash](/static/img/articles/finding-harmony-in-mlops-balancing-functional-and-object-oriented-approaches/cover.webp) Photo by [Alex Padurariu](https://unsplash.com/@alexpadurariu?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=medium&utm_medium=referral) This article aims to shed light on the benefits of both programming styles and help you determine the most suitable one for your MLOps project. We will start by introducing the two main programming styles in our industry. Subsequently, we will explore the specific requirements of MLOps projects to guide our decision-making process. Finally, I will offer my opinion on the best overall style and present a compelling trade-off known as the “hybrid style.” ### A brief intro to programming paradigms Throughout my career, I’ve had the chance of working with various programming languages, starting with object-oriented ones like C++, Java, PHP, Python, Ruby, and Groovy. Each language offers its own set of advantages and disadvantages, depending on the depth of its features: ### Pros: 1. [**Real-World Modeling**](https://www.baeldung.com/cs/oop-modeling-real-world): The object-oriented paradigm closely mirrors real-world entities, making it intuitive for developers to model and design applications based on the problem domain. 2. [**Modularity and Reusability**](https://www.tutorialspoint.com/understanding-code-reuse-and-modularity-in-python-3#:~:text=Modularity%20refers%20to%20the%20act,thrives%20to%20minimize%20the%20duplication.): Object-oriented programming’s encapsulation allows for modular code design, promoting reusability and maintainability. This helps manage large codebases and fosters collaboration among team members. 3. [**Rich Ecosystem**](https://bookauthority.org/books/best-object-oriented-design-books): Object-oriented programming languages like Java and C# have extensive frameworks and design patterns, providing developers with powerful tools for building complex applications efficiently. ### Cons: 1. [**Shared Mutable State**](https://softwareengineering.stackexchange.com/questions/235558/what-is-state-mutable-state-and-immutable-state): Object-oriented programming often relies on a shared mutable state, leading to potential bugs and issues related to mutable objects being accessed from multiple locations. 2. [**Brittle Inheritance Hierarchies**](https://softwareengineering.stackexchange.com/questions/134097/why-should-i-prefer-composition-over-inheritance): Overuse of inheritance can lead to fragile class hierarchies, making it difficult to modify or extend functionality without introducing unintended side effects. 3. [**Complexity and Overhead**](https://en.wikipedia.org/wiki/Anti-pattern): Object-oriented codebases can become complex, especially in large projects, leading to increased development and debugging time. Below is a diagram illustrating an MLOps application implemented with an object-oriented style. The programmer must carefully handle the object attributes, and provide getter/setter methods to control their access. While the program representation is intuitive, it is also verbose and sometimes rigid with its adherence to object-oriented principles. ![Simplified MLOps application implemented with the Object-Oriented Style](/static/img/articles/finding-harmony-in-mlops-balancing-functional-and-object-oriented-approaches/02.webp) Simplified MLOps application implemented with the Object-Oriented Style As my career progressed, I ventured into functional-oriented languages such as Clojure, Haskell, and Elixir. These languages piqued my interest with their unique approach to state management and other concepts that [seemed tailored for data applications](https://towardsdatascience.com/the-ultimate-guide-to-functional-programming-for-big-data-1e57b0d225a3). ### Pros: 1. [**Enhanced reasoning capabilities**](https://en.wikipedia.org/wiki/Pure_function): Functional programming’s focus on pure functions ensures that executions solely depend on inputs, making testing and debugging significantly more straightforward. 2. [**Predictable Concurrency**](https://kyleshevlin.com/just-enough-fp-immutability): Immutability and statelessness inherently reduce the chances of race conditions and concurrent data access issues, making it more suitable for parallel and concurrent programming. 3. [**Simplicity in design**](https://www.youtube.com/watch?v=SxdOUGdseq4): Functional programming requires fewer complex design patterns, relying instead on other forms of [polymorphism](https://en.wikipedia.org/wiki/Polymorphism_%28computer_science%29) (e.g., ad-hoc or parametric), [high-order functions](https://en.wikipedia.org/wiki/Higher-order_function), and even [monads](https://en.wikipedia.org/wiki/Monad_%28functional_programming%29) to extend and fortify programs. ### Cons: 1. [**Learning Curve**](https://www.infoworld.com/article/2843393/functional-programming-tradeoffs-efficiency-learning-curve.html): Functional programming can be challenging for developers who are more accustomed to imperative and object-oriented paradigms. The shift in mindset and understanding of concepts like higher-order functions and recursion may take time. 2. [**Performance Overhead**](https://stackoverflow.com/questions/8659345/why-is-this-simple-haskell-algorithm-so-slow): Some functional programming constructs, such as creating many intermediate data structures during computation, may introduce performance overhead compared to optimized imperative implementations. The diagram below exemplifies an MLOps application adhering to a functional programming style. The program layout is more straightforward due to a clear separation between data structures and operations. However, this style requires functional programming constructs such as [ad-hoc polymorphism](https://www.haskell.org/tutorial/classes.html) to support the addition of both new types and functions in a robust manner. ![Simplified MLOps application implemented with the Functional Programming Style](/static/img/articles/finding-harmony-in-mlops-balancing-functional-and-object-oriented-approaches/03.webp) Simplified MLOps application implemented with the Functional Programming Style Mastering both paradigms proves to be a valuable investment, equipping developers with a diverse toolkit to design optimal solutions. Let’s now explore the specific requirements unique to MLOps projects, aiding us in selecting the best-suited programming style for this type of application. ### Requirements for MLOps projects MLOps applications present a unique blend of simplicity and complexity. On one hand, they share common concepts like datasets, models, and jobs, which can be reused across projects with slight variations. However, dealing with challenges like randomness, large data structures, and complex internal objects, such as neural networks, adds complexity to these projects. To avoid potential struggles and costly refactoring, starting MLOps applications with a well-designed foundation is crucial. Below, we list key requirements ranked by importance for an MLOps application (in my opinion): 1. [**Reproducibility**](https://en.wikipedia.org/wiki/Reproducibility): Ensure that your MLOps application produces consistent and reproducible results. 2. [**Modularity**](https://en.wikipedia.org/wiki/Modularity): Embrace modularity by breaking down your MLOps application into smaller, reusable components. 3. [**Configurability**](https://en.wikipedia.org/wiki/Software_configuration_management): Allowing program behavior changes through external configurations rather than direct code modifications. 4. [**Extensibility**](https://en.wikipedia.org/wiki/Extensibility): Facilitating the addition of new models and data sources. 5. [**Keep It Simple (KISS)**](https://en.wikipedia.org/wiki/KISS_principle): Keep the application simple, as not all MLOps contributors have advanced programming backgrounds. With these requirements in mind, let’s delve into the discussion of which programming style might be best suited for developing MLOps applications. ### So, which style is best? As we explored the pros and cons of both object-oriented and functional programming in the previous section, we see that **there is no single criterion that strongly favors one over the other for MLOps applications**. Both styles can meet the identified requirements, which is fortunate, considering most programming languages are [Turing complete and offer equivalent expressivity](https://en.wikipedia.org/wiki/Turing_completeness). However, I do have a compelling argument. **While all programming styles can be applied to develop MLOps applications, not all programming languages can effectively support both paradigms**. For instance, [Python, one of the most popular languages for data science projects](https://www.datacamp.com/blog/top-programming-languages-for-data-scientists-in-2022), is best suited for object-oriented programming when building large applications. Though it can handle functions and even high-order functions, these features represent the bare minimum to support the functional paradigm. Python lacks support for key elements of functional programming, such as 1) [ad-hoc or parametric polymorphism](https://en.wikipedia.org/wiki/Polymorphism_%28computer_science%29), 2) [tail-call optimization](https://wiki.c2.com/?TailCallOptimization=), and 3) efficient immutable data structures (e.g., [persistent data structures](https://en.wikipedia.org/wiki/Persistent_data_structure)). In contrast, it excels at using [subtyping polymorphism](https://en.wikipedia.org/wiki/Subtyping) and [mutability](https://en.wikipedia.org/wiki/Immutable_object) for various Python operations. As a result, **I tend to favor the object-oriented paradigm when building MLOps applications with Python**, **even if I prefer the functional paradigm for other application types**. Building an MLOps project is not trivial, as it requires advanced and idiomatic language features to fulfill specific requirements. Nevertheless, there is a trick that can be applied to incorporate elements of both paradigms, striking a balance that leverages the strengths of each approach. ### The Hybrid Style The hybrid style aims to combine the best aspects of functional programming with the object-oriented paradigm, creating a favorable trade-off for programming languages like Python that support both styles. By embracing this approach, your code can become more idiomatic, extensible, and easier to reason with. To implement this style effectively, consider adhering to the following principles: 1. [**Immutable attributes**](https://en.wikipedia.org/wiki/Immutable_object): Objects should not update their attributes after initialization, treating them as read-only to avoid modifying the object state directly. 2. **Output-oriented methods**: Each method should return its output rather than updating attributes, enabling other objects to handle modifications to the program state. 3. [**Idempotent method calls**](https://en.wikipedia.org/wiki/Idempotence): methods should consistently return the same output for the given inputs, akin to functional programming principles. 4. [**Centralized imperative statements**](https://www.haskell.org/tutorial/io.html): High-level classes, like a Job class, should handle imperative statements, a concept reminiscent of [the IO monad in Haskell](https://en.wikibooks.org/wiki/Haskell/Understanding_monads/IO). This ensures clear demarcation of actions that interact with the real world, such as logging or database updates. 5. [**Leverage object-oriented other benefits**](https://en.wikipedia.org/wiki/Object-oriented_programming): Embrace the advantages of object-oriented programming, such as subtyping polymorphism and intuitive representations when needed. This final diagram presents an MLOps application following the hybrid style’s guidelines. On one hand, we fall back to [subtyping polymorphism](https://en.wikipedia.org/wiki/Subtyping) and class representations to support the program's extensibility. On the other hand, we reduce the overhead of managing the program state by using and sharing read-only attributes while separating the classes that might have a side effect (i.e., jobs) from the rest of the application. ![Simplified MLOps application implemented with the Hybrid Style](/static/img/articles/finding-harmony-in-mlops-balancing-functional-and-object-oriented-approaches/04.webp) Simplified MLOps application implemented with the Hybrid Style **Remember that these are guiding principles rather than strict rules**. By applying them thoughtfully, you can design a robust application. The [MLOps Python Package](https://fmind.medium.com/a-great-mlops-project-should-start-with-a-good-python-package-7662bdf79563) was developed using these principles and can demonstrate how the hybrid style can be applied effectively to MLOps applications. ### Conclusions This article explored the strengths and weaknesses of two popular programming paradigms: [Functional](https://en.wikipedia.org/wiki/Functional_programming) and [Object-oriented](https://en.wikipedia.org/wiki/Object-oriented_programming). Both styles can be successfully applied to MLOps projects, considering their unique requirements. **The primary selection criterion should align with your chosen programming language’s characteristics, allowing you to implement the most idiomatic solutions**. For instance, opt for a functional style with Haskell or Clojure and an object-oriented style with Java or Python. Alternatively, you can leverage the hybrid style to blend the benefits of functional and object-oriented approaches in a predominantly object-oriented language. This choice respects the language’s capabilities while catering to data applications where idempotence and parallelism play crucial roles in taking your application to the next level. On a personal note, I find the object-oriented style of Python somewhat lacking in elegance. However, I hold Python in high regard for its adaptability to new concepts over time, such as [gradual typing](https://docs.python.org/3/library/typing.html) or [asynchronous programming](https://docs.python.org/3/library/asyncio.html). To improve Python’s object-oriented style, I recommend using a toolkit like [Pydantic](https://docs.pydantic.dev/latest/), a remarkable library that I’ve extensively employed in designing the [MLOps Python Package](https://fmind.medium.com/a-great-mlops-project-should-start-with-a-good-python-package-7662bdf79563). It overcomes the aforementioned limitation and significantly enhances the development process. ![Photo by Nathan Dumlao on Unsplash](/static/img/articles/finding-harmony-in-mlops-balancing-functional-and-object-oriented-approaches/05.webp) Photo by [Nathan Dumlao](https://unsplash.com/@nate_dumlao?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=medium&utm_medium=referral) ### [Is AI/ML Monitoring just Data Engineering? 🤔](https://www.fmind.dev/articles/is-ai-ml-monitoring-just-data-engineering/) [While the future of machine learning and MLOps is being debated](https://www.youtube.com/watch?v=uejAJSynLJo&t), practitioners still need to attend to their machine learning models in production. This is no easy task, as ML engineers must constantly assess the quality of the data that enters and exits their pipelines, and ensure that their models generate the correct predictions. To assist ML engineers with this challenge, several AI/ML monitoring solutions have been developed. In the past few weeks, I reviewed several of these AI/ML monitoring solutions for a client. We considered vendor solutions ([Arize](https://arize.ai/), [Superwise](https://superwise.ai/), [Aporia](https://www.aporia.com/)), open-source solutions ([Evidently](https://www.evidentlyai.com/), [Deepchecks](https://deepchecks.com/)), and even building our own solution. These reviews gave me a lot of food for thought about the essence of AI/ML monitoring and how it should fit into an [MLOps lifecycle](https://ml-ops.org/content/mlops-principles#:~:text=As%20machine%20learning%20and%20AI,debt%E2%80%9D%20in%20machine%20learning%20applications.). ![Is AI/ML Monitoring just Data Engineering? 🤔](/static/img/articles/is-ai-ml-monitoring-just-data-engineering/cover.webp) In this article, I will discuss the nature of AI/ML monitoring and how it relates to data engineering. First, I will present the similarities between AI/ML monitoring and data engineering. Second, I will enumerate additional features that AI/ML monitoring solutions can provide. Third, I will briefly touch on the topic of AI/ML observability and its relation to AI/ML monitoring. Finally, I will provide my conclusion about the field of AI/ML monitoring and how it should be considered to ensure the success of your AI/ML project. ### AI/ML Monitoring is Data Engineering … There are many similarities between AI/ML monitoring and data engineering. Let’s first look at a simplified AI/ML pipeline in production: ![Example of an AI/ML pipeline in production](/static/img/articles/is-ai-ml-monitoring-just-data-engineering/02.webp) Example of an AI/ML pipeline in production We can spot several common points with a data engineering pipeline: - Each pipeline step ingests and produces data. - Steps can be chained together (e.g., like a [UNIX pipeline](https://en.wikipedia.org/wiki/Pipeline_%28Unix%29)). - At the end of the process, it produces alerts, metrics, and dashboards. We can also spot some differences specific to AI/ML pipelines: - There is an AI/ML model used at some point. - … and that’s it! [**Does the use of an AI/ML model make a big difference in a data pipeline?**](https://mlops.community/mlops-is-mostly-data-engineering/) On the one hand, it is just another step that takes data in and generates data out. On the other hand, AI/ML models require extra attention to properly handle the methodology (e.g., avoiding data leakage), hardware (e.g., using GPUs), and new components (e.g., model registries). As this additional complexity requires a specific set of skills and expertise, I tend to think this difference matters. The best proof is that we need specific engineers to manage these challenges (i.e., ML engineers). Let’s now explore how this question affects AI/ML monitoring. ### … and it is also more than Data Engineering The added value of AI/ML monitoring can be summarized in one word: **semantics**. People are much more efficient at dealing with specific concepts than generic ones. To quote this great article from [François Chollet](https://medium.com/u/7462d2319de7) ([User experience design for APIs](https://blog.keras.io/user-experience-design-for-apis.html "Permalink to User experience design for APIs")): > Like most things, API design is not complicated, it just involves following a few basic rules. They all derive from a founding principle: **you should care about your users.** All of them. Not just the smart ones, not just the experts. Keep the user in focus at all times. Yes, including those befuddled first-time users with limited context and little patience. **Every design decision should be made with the user in mind.** For example, in neural networks, we can use user-friendly concepts such as “layers”, “dropout”, and “pooling” instead of more general terms like “operations”, “filters”, and “aggregations”. Similarly, for AI/ML monitoring, we can adapt the UI and API to deal with concepts like “segments”, “baselines”, and “environments”. The underlying techniques can be found in every data engineering pipeline, but the user experience has been tailored to focus users on their use cases and help them become more productive. ![Semantic related to Deep Neural Networks](/static/img/articles/is-ai-ml-monitoring-just-data-engineering/03.webp) Semantic related to Deep Neural Networks **This raises the question of whether this additional semantic value is valuable for data scientists and ML engineers**. I believe that it is. [Naming things](https://www.karlton.org/2017/12/naming-things-hard/) (i.e., coming up with the semantics) is hard, and [humans tend to be lazy](https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow) (i.e., systems 1 and 2). [Our main struggle is always to structure the solution and find the best abstractions to empower developers without adding too much complexity](https://www.youtube.com/watch?v=SxdOUGdseq4). Therefore, it is best to let experts in the field think of the best solutions, just as most web developers do when they use [a framework written by specialists in this domain](https://www.djangoproject.com/). Let’s now review how the authors of AI/ML monitoring solutions can help. ### My opinion on AI/ML monitoring solutions All the AI/ML monitoring solutions I tested have nailed the core workflow: data ingestion, metric computation, alert notification, and error visualization. While they all have their strengths and weaknesses, I can see how each of these solutions can bring value to end users and help them get started with better tools and practices. However, I found a major flaw with most vendor-based solutions: **they do not allow metrics to be exported to other systems**. This is problematic for two reasons. First, users cannot leverage vendor solutions to support custom use cases (e.g., to expose metrics to the business or optimize the training of their models). This means that ML engineers either have to adopt the vendor’s solution entirely and stick with it, or recreate custom pipelines to meet their other needs. Second, most vendors reimplement existing components instead of leveraging the ones developed by other vendors. For instance, I would rather use Tableau for visualization and Datadog for alerting than the tools provided by AI/ML monitoring vendors. AI/ML monitoring vendors cannot catch up to the years of development and dedication that other data vendors have put into their products. I do not blame AI/ML monitoring vendors for this. [It is challenging to create all of these integrations as there is no common protocol for MLOps systems](https://mlops.community/we-need-posix-for-mlops/). We have HTTP, SMTP, and TCP/IP as a universal bridge for the internet, but we do not have anything similar for MLOps. As a result, ML engineers are left with only two options: (1) hope that the vendors will fulfill all of their use cases now and in the future, or (2) build their own solution and focus on the interoperability of their platform. Based on your profile (i.e., end-user vs. engineer), you might choose one over the other. ![Photo by Mike Erskine on Unsplash](/static/img/articles/is-ai-ml-monitoring-just-data-engineering/04.webp) Photo by [Mike Erskine](https://unsplash.com/@mikejerskine?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) ### A note on AI/ML Observability Recently, [we had an interesting discussion about AI/ML monitoring and observability in the MLOps community](https://mlops-community.slack.com/archives/C015J2Y9RLM/p1689069088728459). [Raphaël Hoogvliets](https://medium.com/u/2ec3b8d15687) even wrote [a great article that summarizes these concepts](https://medium.com/@hoogvliets/fff574a8974f). In short, AI/ML monitoring refers to the ability to monitor individual components (e.g., which error occurred, where, and when), while AI/ML observability provides a holistic and high-level overview of the entire system (e.g., why the error occurred and what caused it). Many AI/ML monitoring vendors advertise themselves as “AI/ML observability solutions.” However, I believe this is overstated, as most of their solutions only look at individual models and consider only their inputs and outputs. They do not monitor the entire data pipeline (e.g., the first dataset used), nor are they able to relate the events that occur during its operation (e.g., a new column was added by another team). As a result, it is up to the ML engineer to provide these AI/ML observability capabilities across the entire pipeline. ML engineers can use a lineage system (e.g., [OpenLineage](https://openlineage.io/)) or implement an [Event-Driven Architecture](https://en.wikipedia.org/wiki/Event-driven_architecture) (EDA) to trace the high-level signals that are triggered throughout the pipeline’s lifetime. [Data contracts](https://dataproducts.substack.com/p/the-rise-of-data-contracts) can also be used to define what is “normal” and what is not. I believe this is a promising area of research that has the potential to improve the maturity of MLOps platforms. ![The Pyramid of Monitoring for AI/ML Solutions](/static/img/articles/is-ai-ml-monitoring-just-data-engineering/05.webp) The Pyramid of Monitoring for AI/ML Solutions ### Conclusions **AI/ML monitoring can be seen as a superset of data engineering, but it should not be treated as a subset**. In this way, AI/ML monitoring solutions can help bridge the gap between data toolkits and MLOps use cases, as long as they do not remove the ability to integrate their metrics with other systems. While the temptation and constraints to adopt the best solutions on the market can be high, I encourage you to consider whether the value proposition meets both your needs AND your software principles. To conclude this article, I would like to refer to one of my favorite books: [Gödel, Escher, Bach](https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach) by [Douglas R. Hofstadter](https://en.wikipedia.org/wiki/Douglas_Hofstadter). I love how the author describes the never-ending loop that arises when systems remain as open as possible, even to themselves. For example, DNA creates proteins that can change or manage DNA, and a program can take instructions to create another program (i.e., a compiler). I find similarities in data and AI/ML pipelines, and I would be fascinated by an MLOps process that could create a model capable of managing MLOps processes. We should strive to focus on the [composability](https://en.wikipedia.org/wiki/Composability) and [interoperability](https://en.wikipedia.org/wiki/Interoperability) of our systems, as we never know what may come next. ![Cover of Gödel, Escher, Bach by Douglas R. Hofstadter](/static/img/articles/is-ai-ml-monitoring-just-data-engineering/06.webp) Cover of Gödel, Escher, Bach by Douglas R. Hofstadter ### [A great MLOps project should start with a good Python Package 🐍](https://www.fmind.dev/articles/a-great-mlops-project-should-start-with-a-good-python-package/) MLOps practitioners (rightfully) point out [that running notebooks in production is a bad software practice](https://www.youtube.com/watch?v=7jiPeIFXb6U), but what are the alternatives? A simple script is not enough to capture the complexity of AI/ML projects, and rewriting a whole project in another programming language is both costly and time-consuming. To solve this problem, **the most efficient approach is to create a** [**Python package**](https://packaging.python.org/en/latest/overview/) that compiles the project sources and assets in a [code archive](https://pythonwheels.com/). However, building such a package can be a complex endeavor for newcomers. The Python ecosystem is [vibrant](https://github.com/topics/python), but also [fragmented](https://xkcd.com/1987/). Moreover, [machine learning projects are more complex to develop than most other software applications](https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html). ![Python Environment: https://xkcd.com/1987/](/static/img/articles/a-great-mlops-project-should-start-with-a-good-python-package/cover.webp) Python Environment: [https://xkcd.com/1987/](https://xkcd.com/1987/) **In this article, I present the implementation of a** [**Python package on GitHub**](https://github.com/fmind/mlops-python-package) **designed to support MLOps initiatives**. The goal of this package is to make the coding workflow of data scientists and ML engineers as flexible, robust, and productive as possible. First, I start by motivating the use of Python packages. Then, I provide some tools and tips you can include in your MLOps project. Finally, I explain the follow-up steps required to take this package to the next level and make it work in your environment. **Link to the repository -** [**https://github.com/fmind/mlops-python-package**](https://github.com/fmind/mlops-python-package) ### Motivations [Building Python packages is a common practice in our industry](https://github.com/topics/python). A Python package allows developers to collaborate with others, version the source code, and share code archives on a package index such as [Pypi.org](https://pypi.org/). Another benefit is that Python package can be used both as a library (i.e., imported from another code base) and an application (i.e., be executed from the command line). Python developers are also used to leverage packages developed by others, such as [Flask](https://flask.palletsprojects.com/en/latest/), [Pandas](https://pandas.pydata.org/), or [TensorFlow](https://www.tensorflow.org/) just to name a few. But despite all these benefits, Python packages are complex to build and structure properly. On one hand, [**there are a lot of tools to combine**](https://xkcd.com/1987/) and it can be difficult for developers to choose the best components without hands-on experience. On the other hand, [**machine learning is one of the most complex types of projects**](https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html), as data dimensions, randomness, and entangled workflows make everything more difficult. In my career, I spend hours researching the best set of tools and tricks to make this experience as optimal as possible for data scientists, ML engineers, and myself. I hope this initiative will help you in your MLOps journey and this will let you build great AI/ML solutions for your use cases. ![The General Problem: https://xkcd.com/974/](/static/img/articles/a-great-mlops-project-should-start-with-a-good-python-package/02.webp) The General Problem: [https://xkcd.com/974/](https://xkcd.com/974/) ### Tools & Tips The [GitHub repository](https://github.com/fmind/mlops-python-package) provides both the implementation and the design decisions for developing with the MLOps Python package. You can get all the main information in the [README.md](https://github.com/fmind/mlops-python-package/blob/main/README.md) file. Before jumping to the tools and tips, I’d like to highlight the “methodology” for selecting the elements in this package: - [**Keep It Simple Stupid (KISS)**](https://en.wikipedia.org/wiki/KISS_principle): data scientists and ML engineers are dealing with complex tasks (e.g., maths, software development, business requirements, …). The package should get out of their way as much as possible and be simple to read and follow. - **Leverage good software practices**: our nascent MLOps industry can benefit from years of software experience. We can leverage [design patterns](https://en.wikipedia.org/wiki/Software_design_pattern) and the [Python ecosystem](https://github.com/ml-tooling/best-of-python-dev) to make our development environment as powerful as possible. - **The constant trade-off of simplicity vs power**: creating an empty shell or a technical show-off is easy. The real struggle in creating such a package is to bring the best practice possible while making it accessible to the majority of end users. The [MLOps Python Package](https://github.com/fmind/mlops-python-package#tools) includes more than 30 tools. My favorite ones are: - [**Mypy**](https://github.com/fmind/mlops-python-package#typing-mypy): check that your [code types](https://docs.python.org/3/library/typing.html) are valid during development. - [**OmegaConf**](https://github.com/fmind/mlops-python-package#parser-omegaconf): parse and merge YAML files to load configurations. - [**Pydantic**](https://github.com/fmind/mlops-python-package#validator-pydantic): better definition and validation of Python classes (check out [**Tagged Union**](https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions), this is a great way to [initialize your program](https://en.wikipedia.org/wiki/Creational_pattern)!). - [**Invoke**](https://github.com/fmind/mlops-python-package#tasks-pyinvoke): define development tasks in a saner syntax than Makefile. - [**Poetry**](https://github.com/fmind/mlops-python-package#manager-poetry): manage your Python package (metadata, dependencies, …). The [MLOps Python Package](https://github.com/fmind/mlops-python-package) also includes more than 20 tips and tricks. The most important ones are: - [**SOLID Principles**](https://github.com/fmind/mlops-python-package#solid-principles): define software interface to make your code more modular and reusable. - [**Soft Coding**](https://github.com/fmind/mlops-python-package#soft-coding): change your program behavior through config files instead of code changes. - [**Data Catalog**](https://github.com/fmind/mlops-python-package#data-catalog): [separate the data you want to access from how you access it](https://www.youtube.com/watch?v=D6nYfttnVco) - [**Text Fixture**](https://github.com/fmind/mlops-python-package#test-fixtures): create contextual objects to support [Test-Driven Development](https://en.wikipedia.org/wiki/Test-driven_development) (TDD) with [Pytest](https://docs.pytest.org/en/latest/). - [**DataFrame Typing**](https://github.com/fmind/mlops-python-package#dataframe-typing): define dataframe schemas to communicate their fields and validate them with [Pandera](https://pandera.readthedocs.io/). PS: I know several people who complain that [Python is a bad programming language](https://medium.com/nerd-for-tech/python-is-a-bad-programming-language-2ab73b0bda5). On the contrary, I think Python can be a great programming language with a bit of discipline and the right tooling! ![Python: https://xkcd.com/353/](/static/img/articles/a-great-mlops-project-should-start-with-a-good-python-package/03.webp) Python: [https://xkcd.com/353/](https://xkcd.com/353/) ### Integrations **Having an MLOps Python Package is** [**just a small part of your MLOps journey**](https://medium.com/marvelous-mlops/the-minimum-set-of-must-haves-for-mlops-5dbbcf29401c). While most MLOps project starts with a Python package, this artifact should be integrated with the rest of your infrastructure: Compute Engine (e.g., [Kubernetes](https://kubernetes.io/), [Databricks](https://www.databricks.com/)), Experiment Tracking (e.g., [MLflow](https://mlflow.org/), [Neptune](https://neptune.ai/)), and Task Orchestration (e.g., [Airflow](https://airflow.apache.org/), [Kubeflow](http://kubeflow.org)). After all my research, I haven't found a one-size-fits-all infrastructure that can address everybody's use cases. On one hand, [cloud providers](https://en.wikipedia.org/wiki/Cloud_computing) provide [end-to-end solutions which are specific to their platform](https://aws.amazon.com/sagemaker/). On the other hand, [Kubernetes-based solutions have a huge learning curve](http://kubeflow.org) and are too heavyweight for data scientists. Another big issue is the lack of common protocols to easily integrate all these MLOps systems, as explained in my other article: [We need POSIX for MLOps](https://medium.com/@fmind/we-need-posix-for-mlops-e7bea8d8ec29), and more generally in this talk from Rich Hickey: [The Language of the System](https://www.youtube.com/watch?v=ROor6_NGIWU). Thus, I created this package as a common denominator for MLOps initiatives devoided of any infrastructure dependencies. It is your task, dear reader, to extend its capabilities based on your requirements and environments to suit your end-user needs. ![Standards: https://xkcd.com/927/](/static/img/articles/a-great-mlops-project-should-start-with-a-good-python-package/04.webp) Standards: [https://xkcd.com/927/](https://xkcd.com/927/) ### Conclusions Using Python packages for MLOps is a good practice, but creating the best package requires a lot of [hammock-driven development](https://www.youtube.com/watch?v=f84n5oFoZBc). I hope you will find value in [this package](https://github.com/fmind/mlops-python-package) and have the best success with your MLOps project. To reflect on this task, I like to compare programming to martial arts. The goal is not to beat your opponent with brute force tactics but to inflict the most deadly strikes in the swiftest manner. It is a balance of power, moderation, and respect for your opponent. Similarly, I think creating good software is a form of art, and a never-ending quest to surpass yourself. As I final note, I always love to discuss development practices with my peers. Feel free to drop me a message on the [MLOps Community Slack](https://go.mlops.community/slack), [create an issue on GitHub](https://github.com/fmind/mlops-python-package/issues), or [contribute directly to the repository](https://github.com/fmind/mlops-python-package/pulls). [This is the power of open source after all](https://www.youtube.com/watch?v=9sJUDx7iEJw) :) ![A great MLOps project should start with a good Python Package 🐍](/static/img/articles/a-great-mlops-project-should-start-with-a-good-python-package/05.webp) ### [Fixing the MLOps Survey on LLMs with ChatGPT API: Lessons Learned](https://www.fmind.dev/articles/fixing-the-mlops-survey-on-llms-with-chatgpt-api-lessons-learned/) [Large Language Model](https://en.wikipedia.org/wiki/Large_language_model) (LLM) is such an exciting topic. Since [the release of ChatGPT](https://openai.com/blog/chatgpt), we saw [a surge of innovation](https://www.entrepreneur.com/leadership/how-to-use-chatgpt-to-unlock-new-levels-of-innovation/446151) ranging from [education mentorship](https://techcrunch.com/2023/05/05/openai-chatgpt-chegg-edtech/) to [finance advisory](https://www.bloomberg.com/company/press/bloomberggpt-50-billion-parameter-llm-tuned-finance/). Each week is a new opportunity for [addressing new kinds of problems](https://blog.character.ai/introducing-character/), [increasing human productivity](https://www.jasper.ai/), or [improving existing solutions](https://gamerant.com/skyrim-mod-ai-npc-memories/). Yet, we may wonder if this is just a new [hype cycle](https://www.gartner.com/en/research/methodologies/gartner-hype-cycle) or if [organizations are truly adopting LLMs at scale](https://www.forbes.com/sites/forbestechcouncil/2023/04/25/the-generative-ai-frontier-mastering-llm-adoption-for-ceos-and-ctos-to-drive-business-success/) … ![Photo by Brett Jordan on Unsplash](/static/img/articles/fixing-the-mlops-survey-on-llms-with-chatgpt-api-lessons-learned/cover.webp) Photo by [Brett Jordan](https://unsplash.com/@brett_jordan?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) On March 2023, the [MLOps Community](https://mlops.community/) issued [a survey about LLMs in production](https://docs.google.com/forms/d/e/1FAIpQLSerEryK4xHEZTq0hSu-sVmBHilOzaT71BfCQgXe_uIRgIah-g/viewform) to picture the state of adoption. [The survey is full of interesting insights](https://podcasters.spotify.com/pod/show/mlops/episodes/EXCLUSIVE-EPISODE--LLM-Key-Results-e229lbi), but there is a catch: **80% of the questions are** [**open-ended**](https://en.wikipedia.org/wiki/Open-ended_question) **, which means respondents answered the survey freely from a few keywords to full sentences**. I volunteered to clean up the answers with the help of ChatGPT and let the community get a grasp of the survey experiences. In this article, I present the steps and lessons learned from my journey to shed some light on the MLOps survey on LLMs. I’m first going to present the goal and questions of the survey. Then, I will explain how I used ChatGPT to review the data and standardize the content. Finally, I’m going to evaluate the performance of ChatGPT compared to a manual review. ### Getting to know the MLOps Survey on LLMs [The MLOps Community’s survey](https://forms.gle/vecVUmAeAyd1m75J6) is composed of 17 questions about the use cases, tools, and concerns for adopting LLMs in production. 110 people replied anonymously to the survey, and the responses can [be accessed at this address](https://docs.google.com/spreadsheets/d/13wdBwkX8vZrYKuvF4h2egPh0LYSn2GQSwUaLV4GUNaU/edit#gid=501618501). The questions asked of the participants are listed below: > 0\. What is your position/title at your company? > 1\. How big is your organization? (number of employees) > 2\. Are you using LLM at in your organization? > 3\. What is your use case/use cases? > 4\. Have you integrated or built any internal tools to support LLMs in your org? If so what? > 5\. What are some of the main challenges you have encountered thus far when building with LLMs? > 6\. What are your main concerns with using LLMs in production? > 7\. How are you using LLMs? > 8\. What tools are you using with LLMs? > 9\. What areas are you most interested in learning more about? > 10\. How do you deal with reliability of output? > 11\. Any stories you have that are worth sharing about working with LLMs? > 12\. Any questions you have for the community about LLM in production? > 13\. What is the main reason for not using LLMs in your org? > 14\. What are some key questions you face when it comes to using LLM in prod? > 15\. Have you tried LLMs for different use cases in your org? > 16\. If yes, why did it not work out? **Except for questions 1, 2, and 7, participants were free to provide any text they wanted for the rest of the questions**. Thus, we can find answers such as “Entity matching, customer service responses (souped up/targetted FAQ)” to Q3 (cell H71) or “Thta it will hallucinate something that we won;t pick up in the report editing phase” to Q6 (I67). These answers are rich in information, but they are also tricky to analyze: How can we extract the relevant keywords? Can we summarize the content without losing too much information? Is it possible to automate this process and avoid time-consuming human reviews? I know it would be difficult to apply classical NLP techniques (e.g., TF-IDF, Spell Checkers, Named Entity Recognition, …) with this variety of answers, and this is where [ChatGPT API](https://openai.com/blog/introducing-chatgpt-and-whisper-apis) comes to the rescue! #### Lessons Learned: - **Open-ended questions are much more expressive than close-ended ones**. Since LLM is a new topic, the survey gave a lot of freedom to its participants to share feedback without imposing pre-defined answers. - **Open-ended questions are hard to systematically analyze and visualize.** While this type of answer is difficult to summarize, the survey gave me a good excuse to play with ChatGPT and assist the MLOps Community. - **Besides ChatGPT, I don’t think there is a solution to handle the MLOps survey at scale**. Plan B was to combine several NLP techniques, but I was pretty sure it would take as much time to develop as it would take me to review the answers manually. To quote the XKCD chart of automation: ![Automation: https://xkcd.com/1319/](/static/img/articles/fixing-the-mlops-survey-on-llms-with-chatgpt-api-lessons-learned/02.webp) Automation: [https://xkcd.com/1319/](https://xkcd.com/1319/) ### Cleaning the survey with ChatGPT API The survey analysis was my first time developing with [ChatGPT API](https://platform.openai.com/docs/api-reference). I thought it would be a good match since the task was open-handed and required a lot of background knowledge to find and extract the information. On the other hand, I didn’t want to use or fine-tune smaller LLMs to avoid the pitfalls of creating a new model for a one-shot task. To get more background on prompt engineering, I read the [Prompting Guide](https://www.promptingguide.ai/) and the [Open AI Cookbook](https://github.com/openai/openai-cookbook) before jumping on this task. I found out that the API was easy to use, especially compared to more complex libraries such as deep learning frameworks. Moreover, it seems intuitive to express requirements in natural language. My main struggle was to understand how to convert ChatGPT outputs to programming data structures with Python. I used [Google Colab](https://colab.research.google.com/) to clean up the survey. [The notebook can be accessed at this address](https://colab.research.google.com/drive/1UNWv3ehNZUUcLRBTAX5xBkCYxWnYsBRi#scrollTo=cbI7Qshr9Bqp). In the rest of this section, I’m going to highlight the main structures that helped me work on this use case. [**Google Colaboratory**\ *MLOps Survey: Preparation Notebook*colab.research.google.com](https://colab.research.google.com/drive/1UNWv3ehNZUUcLRBTAX5xBkCYxWnYsBRi#scrollTo=cbI7Qshr9Bqp "https://colab.research.google.com/drive/1UNWv3ehNZUUcLRBTAX5xBkCYxWnYsBRi#scrollTo=cbI7Qshr9Bqp") [](https://colab.research.google.com/drive/1UNWv3ehNZUUcLRBTAX5xBkCYxWnYsBRi#scrollTo=cbI7Qshr9Bqp) The code snippet below shows the [Open AI model](https://platform.openai.com/docs/api-reference/models) used for this experiment. The [“gpt-3.5-turbo” corresponds to the same model used by the ChatGPT application](https://zapier.com/blog/chatgpt-vs-gpt/). The [ChatGPT API](https://platform.openai.com/docs/api-reference/chat) exposed [a single endpoint to create chat completions from user messages](https://platform.openai.com/docs/api-reference/chat) (i.e., POST [https://api.openai.com/v1/chat/completions](https://api.openai.com/v1/chat/completions)). ```json { "created": 1677610602, "id": "gpt-3.5-turbo", "object": "model", "owned_by": "openai", "parent": null, "permission": [ { "allow_create_engine": false, "allow_fine_tuning": false, "allow_logprobs": true, "allow_sampling": true, "allow_search_indices": false, "allow_view": true, "created": 1683391732, "group": null, "id": "modelperm-Kch774kyIWxK1SMaTV7JKoho", "is_blocking": false, "object": "model_permission", "organization": "*" } ], "root": "gpt-3.5-turbo" } ``` I used the following function to associate the user inputs with the model outputs. The function takes as arguments 1) the ChatGPT model, 2) the instructions to perform the task in natural language, 3) the user inputs from a single column, and 4) the size of the batch (i.e., how many user inputs are processed at a time). The function then converts the instructions and input batch to [ChatGPT messages and sends them to the API endpoint](https://platform.openai.com/docs/api-reference/chat). Finally, the model output is parsed to Python data structures and combined into a dataframe. ```python def associate(model, instructions: str, inputs: pd.Series, batch_size: int = 20, limit: int | None = None, **kwargs) -> pd.DataFrame: """Associate the user inputs to the model outputs with the given instructions.""" # avoid wasting requests requests = inputs.dropna() # apply the limit (optional) if limit is not None: requests = requests[:limit] print('- Inputs:', len(inputs)) # iterate on the requests by batch dataframes = [] # store partial results for i, subset in batch(requests, batch_size): print('- batch:', i, '->', len(subset)) # query the model and extract the records messages = chat(instructions, subset.tolist()) content = query(model, messages=messages) records = from_jsonlines(content) # handle the case where lengths are different if len(records) != len(subset): print(f'Warning! Got: {len(records)}, expected: {len(subset)}') records = records + [{} for i in range(len(subset) - len(records))] # convert the records to an indexed dataframe df = pd.DataFrame(records, index=subset.index) dataframes.append(df) # combine the results with the inputs outputs = pd.concat(dataframes, axis='index') reviews = pd.concat([inputs, outputs], axis='columns') return reviews ``` The text snippet below shows the prompt associated with Question 3: “What is your use case/use cases?”. The first four sentences described the task to be done by ChatGPT. The next two instructions explain the output format for the model. For simplicity's sake, I choose to use [JSON lines](https://jsonlines.org/) (i.e., JSON records separated by newlines) to easily parse the output with Python. Ultimately, the last sentences give an example of the expected input and output of the model. Here we can see that the task is quite complex, as the model needs to understand both the fields associated with the answer (e.g., analyze logs -\> Computer Security) and find common NLP tasks from the inputs (e.g., question and classify text -\> Questions Answering, Text Classification). ```text I received answers from an MLOps survey about Large Language Models (LLMs). Your task is to extract all the Natural Language Processing (NLP) tasks and industry fields from each answer. You should use common NLP tasks and industry fields whenever possible to avoid synonyms and acronyms. If no tasks or fields are mentioned, you should use an empty string as a placeholder. You should output these information in the JSON lines format. You should generate one JSON line per answer. Here's an example of an answer: - analyze logs to answer questions and classify text Here's an example of the expected output: {"fields": "Computer Security", "tasks": "Question Answering, Text Classification"} Answers: ``` #### Lessons Learned: - **ChatGPT API is easy to use but hard to integrate**. You must be creative to connect the task that you are trying to automate with the rest of the pipeline. In this case, I use JSON lines to integrate the model output. - **Prompt templates are useful to avoid repetition**. I had to copy-paste a lot of prompts for each question. I think a tool like [LangChain](https://python.langchain.com/en/latest/index.html) could help avoid such unnecessary repetitions in the future. - **ChatGPT outputs are difficult to evaluate without manual reviews**. I now understand why practitioners struggle to properly evaluate LLMs. I was able to use my human capacities in this case, but what about more complex tasks or with more user inputs? - [**Reproducibility can be controlled easily by setting temperature = 0**](https://gptforwork.com/guides/openai-gpt3-temperature). Initially, I was concerned that the model output would vary too much. Fortunately, you can change the temperature to avoid randomness in the model responses. - [**The ChatGPT API lacks a good batch option**](https://community.openai.com/t/batching-with-chatcompletion-not-possible-like-it-was-in-completion/81647) **to improve performance**. Concretely, you can’t use the array of messages supported by the API to provide one user input per message (or at least, I was not able to use it in my case). **Nearly all my problems with the ChatGPT API came from this point alone.** ### Manual Reviews and Visualizations During my development, I found out that ChatGPT gave me good results 80% of the time. Still, this was not sufficient to ship the model output as-is. Thus, I performed a manual review of all the user answers, which was clearly the most tedious part of the whole experience … Out of this manual review, the most common errors I found were: - The number of model outputs doesn’t match the number of user inputs - ChatGPT hasn’t extracted all the information that I wanted - The model outputs were shifted by one or more value On the other hand, I found some pretty impressive benefits: - Nearly all model results added some value over the raw user answer - The model errors were quick to fix during the manual review - Expressing intents from natural language is accessible Following my manual review, I created [another notebook](https://colab.research.google.com/drive/1vQHEvKqedyJW4re9nzp7GlgjKundzVjb#scrollTo=Gd0pr38hoT7S) to visualize the results of the survey. You can also find [the spreadsheet generated by ChatGPT at this address](https://docs.google.com/spreadsheets/d/1c7PHCQ2mgOJnYbsP2PAr4MRh2VBrqJDy/edit#gid=362297934), and [the spreadsheet reviewed manually at this address](https://docs.google.com/spreadsheets/d/1_0xLS96xyZmu1EE1Xcrf9zy7Tte8OFyp/edit#gid=362297934). [**Google Colaboratory**\ *MLOps Survey: Visualization Notebook*colab.research.google.com](https://colab.research.google.com/drive/1vQHEvKqedyJW4re9nzp7GlgjKundzVjb#scrollTo=Gd0pr38hoT7S "https://colab.research.google.com/drive/1vQHEvKqedyJW4re9nzp7GlgjKundzVjb#scrollTo=Gd0pr38hoT7S") [](https://colab.research.google.com/drive/1vQHEvKqedyJW4re9nzp7GlgjKundzVjb#scrollTo=Gd0pr38hoT7S) Let’s now check some visualizations to better grasp the final results. The following figures show the answers to Question 4: “Have you integrated or built any internal tools to support LLMs in your org? If so what?”. The users provided two types of information: the purpose for integrating LLMs (first figure), and the tools used to support the integration (second figure). ![Question 4 (Purposes): Have you integrated or built any internal tools to support LLMs in your org? If so what?](/static/img/articles/fixing-the-mlops-survey-on-llms-with-chatgpt-api-lessons-learned/03.webp) Question 4 (Purposes): Have you integrated or built any internal tools to support LLMs in your org? If so what? ![Question 4 (Solutions): Have you integrated or built any internal tools to support LLMs in your org? If so what?](/static/img/articles/fixing-the-mlops-survey-on-llms-with-chatgpt-api-lessons-learned/04.webp) Question 4 (Solutions): Have you integrated or built any internal tools to support LLMs in your org? If so what? #### Lesson Learned: - **Integration was trivial with JSON Lines**. I love the capacity of ChatGPT to output both natural language and code with the same system. - **ChatGPT has excellent summarization capabilities**. It was able to summarize complex sentences in a few standard keywords with ease. - **Having a good pipeline is key for iterating quickly on the use case**. This is why my first notebook has several helper functions to manipulate ChatGPT API with ease. ### Evaluations and Conclusions For the final step, I wanted to evaluate the performance of ChatGPT compared to my manual review. To do so, I extracted the values from the spreadsheet 1) based on the raw output of ChatGPT API and 2) following my manual review. I then performed a side-by-side comparison of all values. **Evaluation Process**: - Exclude cases where both values are missing -\> 0 - When ChatGPT and I disagreed (strict inequality) -\> -1 - When ChatGPT and I agreed (strict equality, excluding missing) -\> +1 You can find [the evaluation notebook at the following address](https://colab.research.google.com/drive/17WViQH-qTVec1okQHqGywndwxXWW52WJ#scrollTo=P_CE_SXMmBLO). Note that this is a harsh evaluation for ChatGPT, as I didn’t take into consideration results that are partially good. Either the model gave good results (+1), or I had to change something to get the desired results (-1). There is no in-between, even if the model provides some added value in the process. [**Google Colaboratory**\ *MLOps Survey: Evaluation Notebook*colab.research.google.com](https://colab.research.google.com/drive/17WViQH-qTVec1okQHqGywndwxXWW52WJ#scrollTo=P_CE_SXMmBLO "https://colab.research.google.com/drive/17WViQH-qTVec1okQHqGywndwxXWW52WJ#scrollTo=P_CE_SXMmBLO") [](https://colab.research.google.com/drive/17WViQH-qTVec1okQHqGywndwxXWW52WJ#scrollTo=P_CE_SXMmBLO) We can see in the plot below the number of good and bad answers per information extracted. Some information was easily fixed by ChatGPT (e.g., topics, tools, approaches, tasks, …) while others were more challenging to the model (e.g., challenges, reasons, title, …). My conclusions for this evaluation are 1) it’s easier for the model to extract common knowledge for low-variety answers (e.g., NLP tasks), and 2) high-variety answers such as titles and reasons are more diverse and thus more open to interpretation. ![Fixing the MLOps Survey on LLMs with ChatGPT API: Lessons Learned](/static/img/articles/fixing-the-mlops-survey-on-llms-with-chatgpt-api-lessons-learned/05.webp) The plot below shows the final evaluation of the model for my use case. Overall, ChatGPT API gave more good answers (514) than bad ones (289). Qualitatively, even bad results contained relevant values that improved the process. This made the manual review less painful than I first expected. ![Fixing the MLOps Survey on LLMs with ChatGPT API: Lessons Learned](/static/img/articles/fixing-the-mlops-survey-on-llms-with-chatgpt-api-lessons-learned/06.webp) #### Conclusions (for my use case): - **ChatGPT is productive.** This work took 6 hours to develop. While it may have been less time-consuming to do a full manual review, the approach based on ChatGPT was much more enjoyable! - **ChatGPT is cost-efficient.** This experience cost me \$0.15. I’m pleasantly surprised by how inexpensive ChatGPT is for building quick prototypes. - **ChatGPT was accurate enough.** Whenever the model outputs were good or bad, they provided significant help to the review process. #### Extrapolations (for other use cases): - **The sweet spot for ChatGPT is chat applications**. [The lack of good batch options was hard to overcome](https://community.openai.com/t/batching-with-chatcompletion-not-possible-like-it-was-in-completion/81647). I think ChatGPT is better suited for interactive use cases, such as [the application provided by default](https://chat.openai.com/chat). - **ChatGPT is a great tool for building quick prototypes**. Fine-tuning smaller models might be more efficient at dealing with specific use cases. Thus, I think ChatGPT is analogous to [Auto ML for machine learning](https://www.automl.org/automl/): great for prototyping, but limited in exploring problems in-depth. - **Integration and fine-tuning are important for reliability**. This is good news for our community, as this means ML and MLOps engineers will be key in enabling use cases based on LLMs. Humans are not out of the equation yet! ![Machine Learning: https://xkcd.com/1838/](/static/img/articles/fixing-the-mlops-survey-on-llms-with-chatgpt-api-lessons-learned/07.webp) Machine Learning: [https://xkcd.com/1838/](https://xkcd.com/1838/) ### [We need POSIX for MLOps](https://www.fmind.dev/articles/we-need-posix-for-mlops/) If you work on MLOps, you must navigate [an ever-growing landscape of tools and solutions](https://mattturck.com/mad2023/). This is both an intense source of [stimulation](https://mlops.community/) and [fatigue](https://dshersh.medium.com/too-many-mlops-tools-c590430ba81b) for MLOps practitioners. ![Machine Learning, Artificial Intelligence & Data Landscape — MAD 2023](/static/img/articles/we-need-posix-for-mlops/cover.webp) [Machine Learning, Artificial Intelligence & Data Landscape — MAD 2023](https://mattturck.com/mad2023/) Vendors and users face the same problem: **How can we combine all these tools without the** [**combinatorial complexity**](https://en.wikipedia.org/wiki/Combinatorial_explosion) **of** [**creating custom integrations**](https://zenml.io/integrations) **?** ```python import math # number of AI/ML tools -> number of possible integrations print({n: math.comb(n, 2) for n in range(10, 100+10, 10)}) {10: 45, 20: 190, 30: 435, 40: 780, 50: 1225, 60: 1770, 70: 2415, 80: 3160, 90: 4005, 100: 4950} ``` In this article, I propose a solution analogous to [POSIX](https://en.wikipedia.org/wiki/POSIX) to address this challenge. First, I motivate the creation of common protocols and schemas for combining MLOps tools. Second, I present a high-level architecture to support implementation. Third, I conclude with the benefits and limitations of standardizing MLOps. ### What is POSIX? [POSIX](https://en.wikipedia.org/wiki/POSIX) (Portable Operating System Interface**)** is a set of standards specified by the [IEEE](https://www.ieee.org/) for defining a level of compatibility between operating systems (e.g., Linux, MacOS, BSD, …). More concretely, [**POSIX**](https://en.wikipedia.org/wiki/POSIX) **is the foundation that allows end users to implement new applications and ensure they can communicate with each other**. This can be done with shell [commands](https://en.wikipedia.org/wiki/List_of_Unix_commands) (e.g., `ls`, `df`, `pwd`, …) and [pipelines](https://en.wikipedia.org/wiki/Pipeline_%28Unix%29) (e.g., `fd | sort | unique`), or with more complex interfaces such as [network sockets](https://en.wikipedia.org/wiki/Network_socket). POSIX is also linked with the [Unix Philosophy](https://en.wikipedia.org/wiki/Unix_philosophy), an approach that favors [composability](https://en.wikipedia.org/wiki/Composability "Composability") over [monolithic design](https://en.wikipedia.org/wiki/Monolithic_application "Monolithic application"). To quote [Doug McIlroy](https://en.wikipedia.org/wiki/Douglas_McIlroy) (1978): 1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”. 2. Expect the output of every program to become the input to another, as a yet unknown, program. Don’t clutter the output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input. 3. Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them. 4. Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them. ### Why do we need POSIX for MLOps? As ML Engineers we have 2 possibilities for implementing an AI/ML solution: either [go all-in on a set of tools](https://en.wikipedia.org/wiki/Monolithic_application) or [create interfaces to combine, remove, and replace tools](https://en.wikipedia.org/wiki/Software_design_pattern) that are part of the solution. Going all-in is often the easier way to go. For instance, we can start a new project with [MLflow](https://mlflow.org/) for experiment tracking, [TensorFlow](https://www.tensorflow.org/) as our ML framework, and [Great Expectations](https://greatexpectations.io/) to validate our data. But wait, now the team wants to switch to [Neptune](https://neptune.ai/), [PyTorch Lightning](https://pytorch-lightning.readthedocs.io/), and [Evidently](https://www.evidentlyai.com/) … That's a lot of rewrite and rework! [Our goal as an engineer is to create abstractions](https://en.wikipedia.org/wiki/Abstraction_%28computer_science%29) and [protocols](https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview) to avoid such hassle. On the web, we can use any web browser (e.g., Chrome, Firefox, Edge), with any web server (e.g., NGINX, Apache, Gunicorn) without any rewrite or custom integration. If something wants to create a new web program, this entity can do it without asking for permission or requiring changes from other actors. [**AI/ML is a complex field where new solutions are constantly added to solve an increasing number of user cases**](https://paperswithcode.com/) **. We should not limit nor slow the growth of AI/ML because of inconvenient software design.** ### How should we implement it? My proposal is to massively leverage [message brokers](https://en.wikipedia.org/wiki/Message_broker) like [Apache Kafka](https://kafka.apache.org/), [Redis](https://redis.com/solutions/use-cases/messaging), or [ZeroMQ](https://zeromq.org/) to exchange metadata and instructions between AI/ML components. The main benefit of message brokers is to minimize the mutual awareness between components and decouple information sharing between producers and consumers. As [Rich Hickey](https://en.wikipedia.org/wiki/Rich_Hickey) explained, this kind of architecture supports the emergence of a [Language of the System](https://www.youtube.com/watch?v=ROor6_NGIWU). This is also [the paradigm behind the design of the Erlang language](https://www.erlang.org/blog/message-passing/). For MLOps, this means separating each component such as Experiment Tracker, Model Training, or Pipeline Monitoring (e.g., using [Python module](https://docs.python.org/3/tutorial/modules.html) or [Docker container](https://www.docker.com/resources/what-container/)), and exchanging information only through message brokers. This design seeks to implement the [SOLID principles](https://en.wikipedia.org/wiki/SOLID): - [**S**ingle-responsibility principle](https://en.wikipedia.org/wiki/Single-responsibility_principle "Single-responsibility principle"): “There should never be more than one reason for a [class](https://en.wikipedia.org/wiki/Class_%28computer_programming%29 "Class (computer programming)") to change.” - The [**O**pen–closed principle](https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle "Open–closed principle"): “Software entities should be open for extension, but closed for modification.” - The [**L**iskov substitution principle](https://en.wikipedia.org/wiki/Liskov_substitution_principle "Liskov substitution principle"): “Functions that use pointers or references to base classes must be able to use objects of derived classes without knowing it.” - The [**I**nterface segregation principle](https://en.wikipedia.org/wiki/Interface_segregation_principle "Interface segregation principle"): “Clients should not be forced to depend upon interfaces that they do not use.” - The [**D**ependency inversion principle](https://en.wikipedia.org/wiki/Dependency_inversion_principle "Dependency inversion principle"): “Depend upon abstractions, \[not\] concretions.” ![High-level architecture: decoupling between an ML Pipeline and MLOps components](/static/img/articles/we-need-posix-for-mlops/02.webp) High-level architecture: decoupling between an ML Pipeline and MLOps components Each MLOps component should work like a [micro-service](https://en.wikipedia.org/wiki/Microservices). When the component receives an instruction (e.g., log a parameter, get the location of a model, …), it should reply to the sending process with the information requested. A message can [be dispatched to several components](https://cloud.google.com/pubsub/docs/overview) (i.e., fan out), and the component can acknowledge that the message has been processed with a status or error message. The components should be as [loosely coupled](https://en.wikipedia.org/wiki/Loose_coupling) as possible, and they must be configured with a [creational pattern](https://en.wikipedia.org/wiki/Creational_pattern) to swap the components with soft code (e.g., configuration files). In addition, [common schemas](https://schema.org/) (i.e., [names for things](https://en.wikipedia.org/wiki/Ontology_%28computer_science%29)) are required to enable global integrations between MLOps tools. The main benefit of [POSIX](https://en.wikipedia.org/wiki/POSIX) is not only to provide design concepts (e.g., [file descriptors](https://en.wikipedia.org/wiki/File_descriptor)), but also standard names to facilitate the collaboration between actors (e.g., /dev/std{in,err,out} for [standard streams](https://en.wikipedia.org/wiki/Standard_streams)). ### Conclusions MLOps is a complex field and we should not make this field even more complex through [accidental complexity](https://wiki.c2.com/?AccidentalComplexity). MLOps actors need to facilitate the interoperability of the tools we build so we can focus on the real problems: deliver value to our organizations. The high-level architecture proposed in this article is an attempt to answer this problem through common protocols and standard naming. However, there is a hard truth I acknowledge: nobody got rich or famous from creating a standard. While this contribution would be beneficial for all MLOps actors, no individual has the incentive to create this initiative on its own. The alternative is to wait for [Darwinism](https://en.wikipedia.org/wiki/Darwinism) to cull the MLOps tools available, but I'm not sure [the best solutions will emerge from this process](https://en.wikipedia.org/wiki/Worse_is_better). Finally, there is also another thing you need to know about standards. But I think XKCD does a better job than me at explaining it 😄. ![Standards — https://xkcd.com/927/](/static/img/articles/we-need-posix-for-mlops/03.webp) Standards — [https://xkcd.com/927/](https://xkcd.com/927/) ### [How to install Kubeflow Pipelines v2 on Apple Silicon](https://www.fmind.dev/articles/how-to-install-kubeflow-on-apple-silicon/) **Kubeflow Pipelines (KFP) is a powerful platform for building machine learning pipelines at scale with Kubernetes**. The platform is well supported on major cloud platforms such as GCP ([Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)) or AWS ([Kubeflow on AWS](https://awslabs.github.io/kubeflow-manifests/)). However, installing KFP on Apple Silicon (macOS 12.5.1 with Apple M1 Pro) proved to be more challenging than I imagined. Thus, I wanted to share my experience and tips to install KFP as easily as possible on your shiny Mac. **In this article, I present 4 steps to install Kubeflow on Apple Silicon**, using [Rancher Desktop](https://rancherdesktop.io/) for setting up Docker/Kubernetes. In the end, I list the problems I encountered during the installation of Kubeflow Pipelines. ![How to install Kubeflow Pipelines v2 on Apple Silicon](/static/img/articles/how-to-install-kubeflow-on-apple-silicon/cover.webp) ### Step 1: Install Rancher Desktop to setup Kubernetes [**Rancher Desktop**](https://rancherdesktop.io/) **is the most friendly solution I found to install Docker and Kubernetes**. While Docker Desktop is also a popular option, this solution now has a significant price tag for companies with more than 250 employees OR \$10 million in annual revenue. Once you install Rancher Desktop, you need to configure it using the startup window below. Select the stable version of Kubernetes and let Rancher Desktop configures the path automatically. In this case, we use containerd [as this is the default container runtime for Kubernetes](https://blog.tilt.dev/2022/03/04/rancher-desktop-container-runtimes.html). ![How to install Kubeflow Pipelines v2 on Apple Silicon](/static/img/articles/how-to-install-kubeflow-on-apple-silicon/02.webp) I also recommend you to adjust the \# CPUs and the Memory (GB) in `Preferences > Virtual Machine` to avoid Out of Memory errors (OOM). **You show now be able to access the** **`kubectl`** **command in your terminal**. If not, check that your PATH environment variable includes `$HOME/.rd/bin` where Rancher Desktop binaries are installed by default. ### Step 2 : Install Kubeflow Pipelines The official instructions to install Kubeflow Pipelines are available on the [documentation website](https://www.kubeflow.org/docs/components/pipelines/v1/installation/localcluster-deployment/#deploying-kubeflow-pipelines) for KFP v1. You can also find the latest instructions on [Kubeflow Pipelines GitHub repository](https://github.com/kubeflow/pipelines/tree/58052eafc10d42dcbd3ab38abf5c944d25af0e4b/manifests/kustomize) for KFP v2. **As you can see, there is no easy one-click solution available at the time, but we can easily adapt the instructions.** You need to make two choices for installing Kubeflow Pipelines: - **The version of KFP (KFP_VERSION)**: We select the latest v2 version (v2.0.0b4) for the installation. This is a beta release at the time of this writing, but I found this version stable enough for my use case. - **The platform for KFP (KFP_PLATFORM)**: This configuration defines the Argo Workflow execution engine to choose from. I select the new emissary platform ([platform-agnostic-emissary](https://github.com/kubeflow/pipelines/tree/58052eafc10d42dcbd3ab38abf5c944d25af0e4b/manifests/kustomize/env/platform-agnostic-emissary "platform-agnostic-emissary")) that [is now shipped by default in the new versions of KFP](https://www.kubeflow.org/docs/components/pipelines/v1/installation/choose-executor/).   ```bash # set the variables for the installation KFP_PLATFORM=platform-agnostic-emissary KFP_VERSION=2.0.0b4 # start the installation using kubectl apply kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$KFP_VERSION" kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/$KFP_PLATFORM?ref=$KFP_VERSION" ``` The last command may take several minutes to complete, as Kubernetes needs to download the required container images and create the pods for Kubeflow Pipelines. You can monitor the deployment happening in the `kubeflow` namespace using this command: ```bash kubectl get pods -n kubeflow --watch ``` **Once all Kubeflow Pipelines Pods have the RUNNING Status, you can move to the next step**. ![How to install Kubeflow Pipelines v2 on Apple Silicon](/static/img/articles/how-to-install-kubeflow-on-apple-silicon/03.webp) ### Step 3: Port Forwarded **We need to forward the port of KFP UI to your system to access the main dashboard from your browser.** To do so, set a port forward from the `ml-pipeline-ui` Pod to a port on your machine (e.g., 8443 in the example below). ```bash kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8443:80 ``` **You should now be able to access Kubeflow Pipelines at this address:** [**http://localhost:8443/#/pipelines**](http://localhost:8443/#/pipelines) ![How to install Kubeflow Pipelines v2 on Apple Silicon](/static/img/articles/how-to-install-kubeflow-on-apple-silicon/04.webp) You can see in the screenshot above the pipelines shipped by default with Kubeflow Pipelines. ### Step 4: Run a Pipeline **To ensure the system is working properly, we are going to run the pipelines available by default on a new installation**. As a reminder, a KFP Pipeline is a Directed Acyclic Graph (DAG) of components that generates outputs from the inputs and parameters given. The screenshot below shows the structure of the \[Demo\] XGBBoost — Iterative model training pipeline. ![Demo: XGBBoost — Iterative model training pipeline](/static/img/articles/how-to-install-kubeflow-on-apple-silicon/05.webp) I tried all the pipelines available, but some of them had issues that do not seem related to the installation procedure: - 🛑 **\[Demo\] XGBoost — Iterative model training**: the container that trains the model (Xgboost train) does not include the CMake utility. - 🛑 **\[Demo\] TFX — Taxi prediction model training**: the pipeline only works on GCP as indicated in the description of the pipeline. ![How to install Kubeflow Pipelines v2 on Apple Silicon](/static/img/articles/how-to-install-kubeflow-on-apple-silicon/06.webp) ### (Optional) What went wrong This guide was written after many trials and errors. You can find below the issues I encountered during the installation of Kubeflow Pipelines v2. - **Constant ErrImagePull and ImagePullBackoff during installation**: I was not able to install KFP with the docker (moby) engine because of this error. One workaround is to connect directly to the Cluster Node and pull the image directly to fix this error:   ```bash # open a shell to the node executing your pods rdctl shell # install the required container images manually docker pull ... ``` - **Slow UI with Rancher Desktop port forwarding**: Rancher Desktop provides a GUI to set up port forwarding. However, the KFP UI was not responsive when I follow this method. ![How to install Kubeflow Pipelines v2 on Apple Silicon](/static/img/articles/how-to-install-kubeflow-on-apple-silicon/07.webp) - **Solutions not compatible with Apple Silicon**: other alternatives to Rancher Desktop like [minikube](https://minikube.sigs.k8s.io/docs/start/) and [kind](https://kind.sigs.k8s.io/) didn’t work on my system (although I did not investigate these issues further). ### Conclusions While Kubeflow Pipelines is a nice system to use out of the box on Cloud platforms, I find its installation cumbersome for local development. I hope this guide will help you set up KFP on your system, and create amazing machine learning models! Happy Artificial Intelligence ✌️! ### [3P Principle: Purpose, Productivity, Performance](https://www.fmind.dev/articles/3p-principle-purpose-productivity-performance/) ### The Programming Trade-Off: Purpose, Productivity, Performance ![Components of the programming trade-off](/static/img/articles/3p-principle-purpose-productivity-performance/cover.webp) Components of the programming trade-off As programmers, we are continuously looking for languages that are performant, productive, and general purpose. Is there any programming language that currently satisfies these properties? Can we ever create one? In this article, I present a fundamental trade-off that affects the design of programming languages and the success of software projects. ### Definitions **The programming trade-off dictates that a programming language cannot satisfy the following desirable properties at the same time:** - **(General) Purpose**: a programming language can be applied to a wide range of tasks and problems (e.g., web programming, computer security, theorem proving, data analysis …). - **Productivity**: a programming language that provides several mechanisms to deliver effective programs under time constraints (e.g., dynamic typing, introspection, meta-programming, REPL development …). - **Performance**: a programming language that imposes a set of constraints to produce code which is both fast and efficient to execute by a computer (e.g., static typing, compilation, manual memory management, unboxed types …). ### Trade-offs Let’s focus first on the trade-off between performance and productivity. One the one hand, some languages like C and Assembly work closer to the hardware and are more performant than their counterparts. On the other hand, other languages such as LISP and Python provide many convenient features to boost the productivity of programmers. We can infer the following rules to generalize these observations: > **A programming language is performant if and only if the language provides a set of features close to the logic of a computer.** > **A programming language is productive if and only if the language provides a set of features close to the logic of a programmer.** > **Since the logic of a computer is different from the logic of a programmer, the task of optimizing both the performance and the productivity of a programming language is not possible.** This dichotomy, [already described by John Ousterhout](https://en.wikipedia.org/wiki/Ousterhout%27s_dichotomy), is a well-known divergence for programming languages. However, the programming trade-off states that this limitation can be addressed if language designers choose to limit the scope of their language. It is only at this condition that language features cam become more essential both for the programmer to leverage and for the computer to execute. We can see that the programming trade-off is a direct application of the [divide-and-conquer technique](https://en.wikipedia.org/wiki/Divide_and_rule) to the creation of programming languages. For instance, SQL has one and only goal: managing database information based on relational algebra semantics. Other Domain-Specific Languages (DSL) are created to deal with narrow tasks such as web templating (Jinja), matrix manipulation (numpy), or logic programming (Prolog). > **By reducing the scope of programming languages, programmers have the opportunity to improve both the performance and the productivity of their solutions.** ### Take-home message If ignored, the programming trade-off can harm the success of software projects. When I was working as a web developer, my mission was to deliver full-featured applications as fast as I can. But as the user base grown, my tasks shifted more and more toward improving the performance of the application. [Facebook](https://code.fb.com/web/hiphop-for-php-move-fast/) and [Twitter](https://www.infoq.com/articles/twitter-java-use) faced the same problem and had to scale their websites without sacrificing their ability to develop new features. In all these cases, most programming languages do not help as they are either too slow to execute (e.g. Python, R) or too slow to develop with(Java, C++). **The most common solution to address this issue is often to rewrite the application with a different (and often incompatible) language.** Others solutions can mitigate the need to rewrite existing applications: - **Create and explore specific-purpose programming languages** (e.g., query languages, data pipelines, programming paradigms …) - **Embrace** [**Polyglot Programming**](https://deanwampler.github.io/polyglotprogramming/) to combine the strength of several languages (e.g., TensorFlow relies both on C++ for performance and Python productivity). - **Remember that** [**premature optimization is the root of all evil**](http://wiki.c2.com/?PrematureOptimization), it is often more important to explore the edge cases of a problem before optimizing a narrow path. - **Remember that the requirements of a project will change over time**, the ability to adapt to new requirements remains important through the course of a project.