Structured Outputs with Ollama¶

Open-source Large Language Models (LLMs) are rapidly gaining popularity in the AI community. With the recent release of Ollama's OpenAI compatibility layer, it has become possible to obtain structured outputs using JSON schema from these open-source models. This development opens up exciting possibilities for developers and researchers alike.

In this blog post, we'll explore how to effectively utilize the Instructor library with Ollama to harness the power of structured outputs with Pydantic models. We'll cover everything from setup to implementation, providing you with practical insights and code examples.

Why use Instructor?¶

Instructor offers several key benefits:

Simple API with Full Prompt Control: Instructor provides a straightforward API that gives you complete ownership and control over your prompts. This allows for fine-tuned customization and optimization of your LLM interactions. Explore Concepts
Reasking and Validation: Automatically reask the model when validation fails, ensuring high-quality outputs. Leverage Pydantic's validation for robust error handling. Learn about Reasking
Streaming Support: Stream partial results and iterables with ease, allowing for real-time processing and improved responsiveness in your applications. Learn about Streaming
Powered by Type Hints: Leverage Pydantic for schema validation, prompting control, less code, and IDE integration. Learn more
Simplified LLM Interactions: Support for various LLM providers including OpenAI, Anthropic, Google, Vertex AI, Mistral/Mixtral, Anyscale, Ollama, llama-cpp-python, Cohere, and LiteLLM. See Hub

For more details on these features, check out the Concepts section of the documentation.

Patching¶

Instructor's patch enhances an openai api with the following features:

response_model in create calls that returns a pydantic model
max_retries in create calls that retries the call if it fails by using a backoff strategy

Learn More

To learn more, please refer to the docs. To understand the benefits of using Pydantic with Instructor, visit the tips and tricks section of the why use Pydantic page.

Ollama¶

Start by downloading Ollama, and then pull a model such as Llama 3 or Mistral.

Make sure you update your ollama to the latest version!

ollama pull llama3

from openai import OpenAI
from pydantic import BaseModel, Field
from typing import List

import instructor


class Character(BaseModel):
    name: str
    age: int
    fact: List[str] = Field(..., description="A list of facts about the character")


# enables `response_model` in create call
client = instructor.from_openai(
    OpenAI(
        base_url="http://localhost:11434/v1",
        api_key="ollama",  # required, but unused
    ),
    mode=instructor.Mode.JSON,
)

resp = client.chat.completions.create(
    model="llama3",
    messages=[
        {
            "role": "user",
            "content": "Tell me about the Harry Potter",
        }
    ],
    response_model=Character,
)
print(resp.model_dump_json(indent=2))
"""
{
  "name": "Harry James Potter",
  "age": 37,
  "fact": [
    "He is the chosen one.",
    "He has a lightning-shaped scar on his forehead.",
    "He is the son of James and Lily Potter.",
    "He attended Hogwarts School of Witchcraft and Wizardry.",
    "He is a skilled wizard and sorcerer.",
    "He fought against Lord Voldemort and his followers.",
    "He has a pet owl named Snowy."
  ]
}
"""

This example demonstrates how to use Instructor with Ollama, a local LLM server, to generate structured outputs. By leveraging Instructor's capabilities, we can easily extract structured information from the LLM's responses, making it simpler to work with the generated data in our applications.