Question

How I Connected a Dockerized Python API to a Local Mistral Model

I’m currently building the AI foundation for MyZubster, and one of my first goals was to create a local development environment that could run without relying on paid external AI APIs.

The stack uses:

• Python and Flask for the application API
• Docker Compose for orchestration
• Ollama for local model inference
• Mistral as the language model
• Qdrant for vector storage
• Open WebUI for manual testing

This post explains how the services are connected and highlights a few issues I encountered along the way.

The architecture

The planned AI workflow is:

User request
    ↓
MyZubster Flask API
    ↓
Semantic search in Qdrant
    ↓
Relevant MyZubster data
    ↓
Mistral through Ollama
    ↓
Generated response

Qdrant will provide the retrieval part of the RAG pipeline, while Mistral will generate an answer using the retrieved context.

Why Ollama runs on the host

Ollama was already installed and running directly on Windows at:

http://localhost:11434

The mistral:latest model was also already downloaded.

I decided not to add another Ollama container because that would duplicate model storage, increase memory usage and conflict with the port already occupied by the host process.

Docker Desktop provides the special hostname:

host.docker.internal

Containers can use it to connect to services running on the host machine.

Inside the MyZubster API container, the Ollama address therefore becomes:

http://host.docker.internal:11434

Verifying the connection

Before changing the application, I tested the connection using a temporary curl container:

docker run --rm curlimages/curl:8.12.1 `
  http://host.docker.internal:11434/api/tags

Ollama returned the installed model:

{
  "models": [
    {
      "name": "mistral:latest",
      "parameter_size": "7.2B",
      "quantization_level": "Q4_K_M"
    }
  ]
}

This confirmed that Docker could communicate with Ollama on the Windows host.

Docker Compose stack

Here is the Compose configuration used for the local environment:

services:
  api:
    build:
      context: .
    init: true
    ports:
      - "5000:5000"
    environment:
      MYZUBSTER_HOST: "0.0.0.0"
      MYZUBSTER_PORT: "5000"
      MYZUBSTER_OBSERVATIONS_FILE: "/data/observations.json"
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
      OLLAMA_MODEL: "mistral:latest"
      QDRANT_URL: "http://qdrant:6333"
      QDRANT_COLLECTION: "myzubster"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - observations-data:/data
    depends_on:
      qdrant:
        condition: service_started
    restart: unless-stopped

  qdrant:
    image: qdrant/qdrant:latest
    ports:
      - "127.0.0.1:6333:6333"
      - "127.0.0.1:6334:6334"
    volumes:
      - qdrant-data:/qdrant/storage
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      OLLAMA_BASE_URL: "http://host.docker.internal:11434"
      WEBUI_AUTH: "true"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui-data:/app/backend/data
    restart: unless-stopped

volumes:
  observations-data:
  qdrant-data:
  open-webui-data:

networks:
  default:
    name: myzubster-ai-network

The API communicates with Qdrant using its Compose service name:

http://qdrant:6333

It communicates with Ollama through the host gateway:

http://host.docker.internal:11434

Starting the environment

I first validate the Compose file:

docker compose config

Then I pull the external images and start the stack:

docker compose pull
docker compose down
docker compose up -d --build

The service status can be checked with:

docker compose ps

Useful logs are available through:

docker compose logs --tail 100 api
docker compose logs --tail 100 qdrant
docker compose logs --tail 100 open-webui

Testing Ollama from the application container

A successful request from Windows does not necessarily prove that the application container can reach Ollama.

I tested the full connection from inside the API container:

docker compose exec api python -c "import json,os,urllib.request; url=os.environ['OLLAMA_BASE_URL']+'/api/generate'; data=json.dumps({'model':os.environ['OLLAMA_MODEL'],'prompt':'Reply only with: MyZubster connected','stream':False}).encode(); req=urllib.request.Request(url,data=data,headers={'Content-Type':'application/json'}); result=json.loads(urllib.request.urlopen(req,timeout=120).read()); print(result['response'].strip())"

The container received a generated response, confirming that the networking and environment variables were configured correctly.

A PowerShell trap

One mistake I made was pasting YAML directly into PowerShell:

services:
  api:
    build:
      context: .

PowerShell interpreted each YAML key as a command and returned errors such as:

The term 'services:' is not recognized as the name of a cmdlet

A Compose configuration must be saved as docker-compose.yml; it cannot be executed directly.

To open the file in Visual Studio Code:

code docker-compose.yml

Always validate it afterward:

docker compose config

Available services

After startup, the local tools are available at:

• MyZubster API: http://localhost:5000
• Open WebUI: http://localhost:3000
• Qdrant dashboard: http://localhost:6333/dashboard
• Ollama API: http://localhost:11434

Qdrant and Open WebUI are bound to 127.0.0.1, so their ports are not exposed to other devices on the network.

What comes next

The containers provide the infrastructure, but the actual RAG pipeline still needs to be implemented.

The next development steps are:

1. Install an embedding model in Ollama.
2. Create a Qdrant collection with the correct vector dimensions.
3. Generate embeddings for MyZubster content.
4. Store vectors together with useful metadata.
5. Embed incoming user questions.
6. Retrieve the most relevant records from Qdrant.
7. Add the retrieved context to the Mistral prompt.
8. Expose the workflow through an API endpoint.
9. Add integration tests and response validation.

One possible endpoint will be:

POST /api/ai/ask

Example request:

{
  "question": "Which products use sustainable packaging?"
}

The application will search its own indexed data and ask Mistral to generate an answer grounded in those results.

Final thoughts

This setup creates a practical local AI development environment while keeping data and model inference on the developer’s machine.

The most important lesson was to verify each connection independently:

• Host to Ollama
• Docker to Ollama
• API to Qdrant
• Browser to Open WebUI

Once those connections work, debugging the application layer becomes much easier.

The MyZubster AI integration is still evolving, but the infrastructure is now ready for the next phase: building and testing the RAG workflow.

What local model and vector database are you using for your AI projects?

Docker #ArtificialIntelligence #Python #Ollama #Mistral #Qdrant #RAG