Deploying a Production-Ready Go Scraper to Kubernetes with Prometheus: The Complete Implementation

Deploying a Production-Ready Go Scraper to Kubernetes with Prometheus: The Complete Implementation

●1 ●6 ●15
calendar_today • schedule3 min read

Most Go tutorials stop at go run main.go. Real production starts after that.

I had a working Go Jobs Tracker with Prometheus metrics, but it only ran on my laptop. If I close the lid, production dies. That is not production.

This is how I took it from localhost to a scalable, observable Kubernetes deployment with 12MB images, health checks, and zero-downtime.


  1. The Problem: Why go run is Not Production
  • No isolation
  • No auto-restart
  • No scaling
  • No observability in cluster
  • Secrets in .env

We need Docker plus Kubernetes plus Prometheus.

  1. Production Dockerfile - Multi-Stage (12MB vs 1GB)

This is the Dockerfile big tech uses. Not the junior FROM golang.

Create Dockerfile in root:

Stage 1: Builder - we compile here
FROM golang:1.22-alpine AS builder

WORKDIR /app

Cache dependencies first - faster rebuilds
COPY go.mod go.sum ./
RUN go mod download

COPY . .

Build static binary - no CGO, no libc dependency
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
  -ldflags="-w -s" \
  -o /go-jobs-tracker ./cmd/scraper

Stage 2: Final - distroless-level small
FROM alpine:3.19

WORKDIR /app

RUN apk --no-cache add ca-certificates wget

Copy only the binary
COPY --from=builder /go-jobs-tracker .

Non-root user - security best practice
RUN adduser -D appuser
USER appuser

EXPOSE 2112

K8s needs this to know if you are alive
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:2112/healthz || exit 1

CMD ["./go-jobs-tracker"]
Why this is production grade:
- Multi-stage results in 12MB final image
- -ldflags="-w -s" strips debug symbols
- CGO_ENABLED=0 produces static binary
- Non-root user for security
- HEALTHCHECK for Docker and K8s

Build it:
docker build -t go-jobs-tracker:latest .
docker images | grep go-jobs-tracker
3. Full Observability Stack with docker-compose

One command to run everything like in production.

Create `docker-compose.yml`:
version: '3.8'
services:
  scraper:
    build: .
    ports:
            - "2112:2112"
    restart: unless-stopped
    environment:
            - SERPAPI_KEY=${SERPAPI_KEY}
    networks:
            - monitoring

  prometheus:
    image: prom/prometheus:v2.52.0
    ports:
            - "9090:9090"
    volumes:
            - ./prometheus.yml:/etc/prometheus/prometheus.yml
            - prometheus_data:/prometheus
    command:
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--storage.tsdb.path=/prometheus'
    networks:
            - monitoring

  grafana:
    image: grafana/grafana:10.4.0
    ports:
            - "3000:3000"
    environment:
            - GF_SECURITY_ADMIN_USER=admin
            - GF_SECURITY_ADMIN_PASSWORD=admin
            - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
            - grafana_data:/var/lib/grafana
    depends_on:
            - prometheus
    networks:
            - monitoring

volumes:
  prometheus_data:
  grafana_data:

networks:
  monitoring:
    driver: bridge
Create `prometheus.yml`:
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
    - job_name: 'go-jobs-tracker'
    static_configs:
            - targets: ['scraper:2112']
    metrics_path: /metrics
    scrape_interval: 5s
Run it:
docker-compose up --build
Your stack:
- App on 2112 slash metrics
- Prometheus on 9090
- Grafana on 3000 with admin slash admin

4. Kubernetes - Real Production

This is what separates Senior from Staff.

Create folder `k8s` and file `k8s/deployment.yaml`:
apiVersion: v1
kind: Secret
metadata:
  name: scraper-secrets
type: Opaque
stringData:
  serpapi-key: YOUR_SERPAPI_KEY_HERE

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-jobs-tracker
  labels:
    app: go-jobs-tracker
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: go-jobs-tracker
  template:
    metadata:
      labels:
        app: go-jobs-tracker
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2112"
        prometheus.io/path: "/metrics"
    spec:
      containers:
            - name: scraper
        image: ghcr.io/aureliopires186/go-jobs-tracker:latest
        imagePullPolicy: Always
        ports:
                - name: metrics
          containerPort: 2112
        env:
                - name: SERPAPI_KEY
          valueFrom:
            secretKeyRef:
              name: scraper-secrets
              key: serpapi-key
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "200m"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 2112
          initialDelaySeconds: 15
          periodSeconds: 20
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /healthz
            port: 2112
          initialDelaySeconds: 5
          periodSeconds: 10
          failureThreshold: 3
---
apiVersion: v1
kind: Service
metadata:
  name: go-jobs-tracker-service
  labels:
    app: go-jobs-tracker
spec:
  selector:
    app: go-jobs-tracker
  ports:
    - name: metrics
    port: 2112
    targetPort: 2112
    protocol: TCP
  type: ClusterIP
Deploy to K8s:
kubectl apply -f k8s/deployment.yaml
kubectl get pods -w
kubectl logs -f deployment/go-jobs-tracker
kubectl port-forward svc/go-jobs-tracker-service 2112:2112
5. The Metrics Code - What Makes It Observable

Your metrics endpoint should expose this:
// metrics/metrics.go
package metrics

import (
  "github.com/prometheus/client_golang/prometheus"
  "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
  JobsScraped = promauto.NewCounterVec(prometheus.CounterOpts{
    Name: "jobs_scraped_total",
    Help: "Total number of jobs scraped",
  }, []string{"engine", "status"})

  ScrapeDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "scrape_duration_seconds",
    Help:    "Duration of scrape jobs",
    Buckets: prometheus.DefBuckets,
  }, []string{"engine"})

  ScraperErrors = promauto.NewCounterVec(prometheus.CounterOpts{
    Name: "scraper_errors_total",
    Help: "Total errors in scraper",
  }, []string{"type"})
)
And in http://main.go:
http.Handle("/metrics", promhttp.Handler())
http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request){
  w.WriteHeader(200)
  w.Write([]byte("ok"))
})
go http.ListenAndServe(":2112", nil)
6. Result: What You Get

After this tutorial you have:

- 12MB Docker image, not 1GB
- Non-root, secure, production-ready
- docker-compose with full monitoring
- K8s with 2 replicas, zero-downtime rolling update
- Liveness and Readiness probes
- Resource limits to prevent OOM in prod
- Prometheus auto-discovery
- Secrets, not hardcoded keys

This is how real companies deploy Go in production.

Production-ready Go is not about code, it is about deployment.


•Author : Pires Aurélio 

1 Comment

0 votes
🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

Your Kubernetes Requests Are a Guess. groundcover Wants an Agent to Keep Fixing Them.

Tom Smithverified - Nov 3

Your AI Agent Doesn't Need Your Token: Edera's Ariadne Conill on Object Capabilities

Tom Smithverified - Nov 5

Production-Ready Go Scraper with Prometheus - Complete Implementation Building a scraper

Pires Aurelio - Oct 3

Kamal vs Kubernetes: An Honest Comparison for Teams Who Don’t Need 1,000 Services

Alexandre Vazquez - Jul 24

Sovereign Intelligence: The Complete 25,000 Word Blueprint (Download)

Pocket Portfolio - Apr 1
chevron_left
368 Points • 22 Badges
6Posts
5Comments
6Connections
Go Developer | Open Source | API Integration`/ Independent Go Developer building open-source project... Show more

Related Jobs

View all jobs →

Commenters (This Week)

4 comments
3 comments
2 comments

Contribute meaningful comments to climb the leaderboard and earn badges!