# Structured LLM Outputs Made Easy: Building a Slack Digest Agent with Pydantic AI

> As developers and AI practitioners, many of us belong to vibrant online communities, like the MLOps Community on Slack. These platforms are…

- Published: 2025-04-15
- Reading time: 7 min
- Tags: Agent, LLM, Python
- URL: https://www.fmind.dev/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/
- Also published at: https://medium.com/@fmind/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai-d61df262e63d

As developers and AI practitioners, many of us belong to vibrant online communities, like the [MLOps Community on Slack](https://go.mlops.community/slack). These platforms are fantastic hubs for knowledge sharing, discussion, and collaboration. However, they can also become overwhelming sources of information. Keeping up with dozens of channels and hundreds of messages daily is a real challenge!

![Structured LLM Outputs Made Easy: Building a Slack Digest Agent with Pydantic AI](https://www.fmind.dev/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/cover.webp)

Wouldn’t it be great to have a personal assistant that could sift through the noise, identify the most relevant conversations, and summarize them neatly for you?

That’s exactly what I set out to build with the [**MLOps Digester**](https://github.com/fmind/mlops-digester) project — a simple agent designed to fetch interesting Slack threads and generate concise summaries. And the secret sauce making this manageable and reliable? [**Pydantic AI**](https://ai.pydantic.dev/).

![Agent Framework / shim to use Pydantic with LLMs: https://ai.pydantic.dev/](https://www.fmind.dev/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/02.webp)

_Agent Framework / shim to use Pydantic with LLMs:_ [https://ai.pydantic.dev/](https://ai.pydantic.dev/)

In this post, we’ll dive into how Pydantic AI helps bridge the gap between the often unstructured, free-form text generated by Large Language Models (LLMs) and the structured data formats we need for robust applications. We’ll use the [MLOps Digester](https://github.com/fmind/mlops-digester) as a practical example.

### The Challenge: Getting Structured Data from LLMs

LLMs like [OpenAI’s GPT models](https://ai.pydantic.dev/api/models/openai/) or [Google’s Gemini](https://ai.pydantic.dev/api/models/gemini/) are incredibly powerful at understanding and generating human-like text. You can ask them to summarize conversations, extract key points, or classify information. However, getting the output _exactly_ in the format you need (like specific JSON fields) often requires complex prompt engineering and can still be brittle. The LLM might hallucinate fields, miss required ones, or return data in an inconsistent structure.

![Slack Channels on the MLOps Community](https://www.fmind.dev/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/03.webp)

Slack Channels on the MLOps Community

### Enter Pydantic AI: Validation Meets Language Models

If you’re a Python developer, you’ve likely encountered [Pydantic](https://docs.pydantic.dev/latest/). It’s a fantastic library for [data validation](https://docs.pydantic.dev/latest/concepts/validators/) and [settings management](https://docs.pydantic.dev/latest/concepts/pydantic_settings/) using Python [type annotations](https://docs.python.org/3/library/typing.html). You define how your data _should_ look using standard Python classes, and Pydantic handles the parsing, validation, and error handling.

**Pydantic AI** cleverly extends this concept to LLM interactions. It allows you to:

1. **Define your desired output structure** using a standard Pydantic model.
2. **Provide this model alongside your prompt** to an LLM (via Pydantic AI’s interface).
3. **Pydantic AI guides the LLM** to generate output that conforms to your defined model.
4. **It automatically parses and validates** the LLM’s response against your Pydantic model.

Essentially, Pydantic AI acts as a mediator, ensuring the LLM’s creativity adheres to the structural requirements you define.

![High-Level Components of the MLOps Digester](https://www.fmind.dev/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/04.webp)

High-Level Components of the MLOps Digester

### Use Case: The MLOps Digester Agent

In the [`mlops-digester`](https://github.com/fmind/mlops-digester) project, the goal is to process transcripts of Slack threads and generate a structured `Digest` containing summaries of the most relevant ones. We need a consistent output format to reliably display or further process these digests.

![Project Structure of the MLOps Digester](https://www.fmind.dev/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/05.webp)

Project Structure of the MLOps Digester

**1. Defining the Structure:**

First, we define our desired output structure using Pydantic models. In [`src/mlops_digester/results.py`](https://github.com/fmind/mlops-digester/blob/main/src/mlops_digester/results.py), we have something like this:

```python
import pydantic as pdt

class SlackThreadDigest(pdt.BaseModel):
    """Digest of a Slack thread."""

    title: str
    summary: str
    takeaways: list[str]
    tags: list[str]
    links: list[str]
    tools: list[str]


class SlackWorkspaceDigest(pdt.BaseModel):
    """Digest of a Slack workspace."""

    tops: list[str]
    flops: list[str]
    moods: list[str]
    topics: list[str]
    sharing: list[str]
```

Here, the _SlackThreadDigest_ defines the fields we want for _each_ summarized thread. On the other hand, the _SlackWorkspaceDigest_ summarizes the key points of a whole Slack Workspace.

**2. Creating the Agent:**

Next, in [`src/mlops_digester/agents.py`](https://github.com/fmind/mlops-digester/blob/main/src/mlops_digester/agents.py), we create an agent that uses Pydantic AI. This involves initializing `PydanticAI` with an LLM client (like OpenAI's or Gemini's) and then using it to process the input text based on our desired Pydantic model.

```python
def to_slack_thread_digest_agent(
    slack_thread_digest_agent_settings: settings.SlackThreadDigesterAgentSettings,
) -> SlackThreadDigestAgent:
    """Create an agent for digesting Slack threads from settings."""
    agent = pdtai.Agent(
        name=slack_thread_digest_agent_settings.name,
        model_settings=slack_thread_digest_agent_settings.model_settings,
        deps_type=depends.SlackThreadDepends,
        result_type=results.SlackThreadDigest,
    )

    @agent.system_prompt
    def agent_system_prompt(ctx: pdtai.RunContext[depends.SlackThreadDepends]) -> str:
        """Define the system prompt for the agent."""
        slack_channel_name = ctx.deps.channel_name
        system_prompt = f"""{slack_thread_digest_agent_settings.system_prompt}

        The Slack Channel Name is: {slack_channel_name}
        """
        return system_prompt

    return agent
```

3\. **Running the Main Tasks**

The magic happens in [`src/mlops_digester/tasks.py`](https://github.com/fmind/mlops-digester/blob/main/src/mlops_digester/tasks.py). We provide the raw Slack transcript within a prompt and tell Pydantic AI we expect an output conforming to the `Digests` model. Pydantic AI handles the interaction with the LLM, the parsing, the validation, and potentially the retries, finally returning a validated `Digest` object (or raising an error if it fails).

```python
def fetch_slack_content(
    fetch_slack_content_step_settings: settings.FetchSlackContentStepSettings,
    slack_client: slack.WebClient,
) -> SlackContent:
    """Fetch MLOps content from Slack channels, messages, and replies."""
    # dates
    end_date = datetime.datetime.now()
    start_date = end_date - datetime.timedelta(
        days=fetch_slack_content_step_settings.since_last_days
    )
    # channels
    slack_channels = {}
    if fetch_slack_content_step_settings.channels:
        for slack_channel_id in fetch_slack_content_step_settings.channels:
            slack_channel_result = slack_client.conversations_info(channel=slack_channel_id)
            slack_channel = slack_channel_result["channel"]  # extract channel data
            slack_channels[slack_channel["id"]] = slack_channel
    else:
        slack_conversations = slack_client.conversations_list(
            limit=fetch_slack_content_step_settings.max_channels_per_workspace,
            exclude_archived=fetch_slack_content_step_settings.exclude_archived_channels,
        )
        for slack_channel in slack_conversations["channels"]:
            slack_channels[slack_channel["id"]] = slack_channel
    # messages
    for slack_channel_id, slack_channel in slack_channels.items():
        slack_channel_messages = slack_channel.setdefault("messages", {})
        try:
            slack_conversation_history = slack_client.conversations_history(
                channel=slack_channel_id,
                oldest=str(start_date.timestamp()),
                limit=fetch_slack_content_step_settings.max_messages_per_channel,
            )
            for slack_channel_message in slack_conversation_history["messages"]:
                slack_channel_messages[slack_channel_message["ts"]] = slack_channel_message
        except slack_errors.SlackApiError as slack_api_error:
            logger.warning(f"Error while Fetching Slack Channel Messages: {slack_api_error}")
    # replies
    for slack_channel_id, slack_channel in slack_channels.items():
        for slack_message_ts, slack_message in slack_channel["messages"].items():
            slack_message_replies = slack_message.setdefault("replies", {})
            slack_message_thread = slack_client.conversations_replies(
                ts=slack_message_ts,
                channel=slack_channel_id,
                limit=fetch_slack_content_step_settings.max_replies_per_message,
            )
            for slack_message_reply in slack_message_thread["messages"]:
                slack_message_replies[slack_message_reply["ts"]] = slack_message_reply
    return slack_channels
```

This gives us confidence that `digest_output` will have all the required fields (`title`, `summary`, `takeaways`) and nested structures (SlackThreadDigest and SlackWorkspaceDigest) we defined, ready for use in our application.

![User Interface of the MLOps Digester showing the key information of a MLOps Community channel](https://www.fmind.dev/static/img/articles/structured-llm-outputs-made-easy-building-a-slack-digest-agent-with-pydantic-ai/06.webp)

User Interface of the MLOps Digester showing the key information of a MLOps Community channel

### Pydantic AI: A Quick Review

Based on my experience using it in this project and exploring its capabilities:

**Strengths:**

- **Reliable Structured Output:** This is its killer feature. Significantly increases the reliability of getting JSON or other structured data from LLMs compared to prompt engineering alone.
- **Leverages Pydantic’s Power:** Uses the familiar and powerful validation features of Pydantic. If you already use Pydantic, the learning curve is gentle.
- **Improved Developer Experience:** Reduces the need for complex output parsing and validation logic in your application code. Type hints provide great editor support.
- **LLM Agnostic:** Designed to work with various LLM providers (OpenAI, Gemini, Bedrock, etc.), giving you flexibility.

**Weaknesses:**

- **Abstraction Leak:** Pydantic AI relies on the underlying LLM’s ability to understand the request and the structure. If the LLM fundamentally struggles with the task complexity or the data, Pydantic AI can only enforce the _format_, not guarantee the _semantic correctness_ beyond what the LLM provides.
- **Overhead:** Adds another dependency to your project. For _extremely_ simple, single-field extractions, it might feel like overkill (though still often worth it for the validation).
- **Prompting Still Matters:** While it handles _structure_, you still need to write clear prompts to guide the LLM on _what_ content to generate for the fields. The descriptions in your Pydantic models become part of this implicit prompt.

### Conclusion

[**Pydantic AI**](https://ai.pydantic.dev/) is a powerful tool for any developer working with LLMs who needs reliable, structured output. By combining the robust data validation of Pydantic with intelligent LLM interaction, it significantly simplifies the process of integrating generative AI into applications that require predictable data formats.

The [MLOps Digester](https://github.com/fmind/mlops-digester) project demonstrates how Pydantic AI can turn a potentially messy task — summarizing noisy Slack conversations — into a structured, manageable output. It streamlines development, improves reliability, and lets you focus more on the application logic rather than fighting with inconsistent LLM responses.

If you’re building applications that consume LLM output, especially for data extraction, classification, or structured content generation, I highly recommend giving Pydantic AI a try.

**Link to the GitHub Repository**: [https://github.com/fmind/mlops-digester/tree/main](https://github.com/fmind/mlops-digester/tree/main)

# An error occurred.

Unable to execute JavaScript.

Build a Slack Agent with Pydantic AI \[Step-by-Step Tutorial\]