Production-Ready Go Scraper with Prometheus - Complete Implementation Building a scraper

Production-Ready Go Scraper with Prometheus - Complete Implementation Building a scraper

●1 ●6 ●15
calendar_today • schedule1 min read

Building a scraper that works is easy. Building a scraper that you can observe in production is senior work.

I added Prometheus to my Go Jobs Tracker and here's the full working implementation.

PART 1: METRICS DEFINITION

Create file metrics/metrics.go:

package metrics

import (

"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"

)

var (

JobsScraped = promauto.NewCounterVec(
    prometheus.CounterOpts{
        Name: "jobs_scraped_total",
        Help: "Total number of jobs scraped per engine and status",
    },
    []string{"engine", "status"},
)

ScrapeDuration = promauto.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "scrape_duration_seconds",
        Help: "Duration of scrape jobs",
        Buckets: prometheus.DefBuckets,
    },
    []string{"engine"},
)

ErrorsTotal = promauto.NewCounterVec(
    prometheus.CounterOpts{
        Name: "scraper_errors_total",
        Help: "Total errors by type",
    },
    []string{"type"},
)

)

PART 2: MAIN.GO INTEGRATION - 100% WORKING

package main

import (

"net/http"
"time"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/Aureliopires186/go-jobs-tracker/metrics"

)

func main() {

go func() {
    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(200)
        w.Write([]byte("ok"))
    })
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("go-jobs-tracker running. check /metrics and /healthz"))
    })
    http.ListenAndServe(":2112", nil)
}()

start := time.Now()

// Simulate your scraping logic here
err := ScrapeGoogleJobs()

duration := time.Since(start)

if err != nil {
    metrics.ErrorsTotal.WithLabelValues("scrape_failed").Inc()
} else {
    metrics.JobsScraped.WithLabelValues("google_jobs", "success").Inc()
    metrics.ScrapeDuration.WithLabelValues("google_jobs").Observe(duration.Seconds())
}

select {}

}

func ScrapeGoogleJobs() error {

// your logic
return nil

}

PART 3: GO.MOD DEPENDENCY

Run:
go get github.com/prometheus/client_golang@v1.19.0
go mod tidy

PART 4: HOW TO TEST IT LOCALLY

Terminal 1:
go run main.go

Terminal 2:
curl http://localhost:2112/metrics
curl http://localhost:2112/healthz

You will see:
jobs_scraped_total
scrape_duration_seconds
scraper_errors_total

PART 5: PROMETHEUS CONFIG

prometheus.yml

global:
scrape_interval: 15s
scrape_configs:

- job_name: 'go-jobs-tracker'
static_configs:
        - targets: ['localhost:2112']

This setup is production ready, K8s ready, and used by real companies.

No more blind scraping.

•Author :Pires Aurélio

•Go Developer

1 Comment

1 vote
🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

Deploying a Production-Ready Go Scraper to Kubernetes with Prometheus: The Complete Implementation

Pires Aurelio - Oct 3

Make Background Image Jobs Observable: States, Retries and Recovery

WAQAS AHMAD - Nov 3

Sovereign Intelligence: The Complete 25,000 Word Blueprint (Download)

Pocket Portfolio - Apr 1

Two rejected series stalled all my metrics

Artem Senenko - Nov 12

How to Deploy an Open Source LLM Reliably (on Kubernetes)

heyyayushh - May 3
chevron_left
368 Points • 22 Badges
6Posts
5Comments
6Connections
Go Developer | Open Source | API Integration`/ Independent Go Developer building open-source project... Show more

Related Jobs

View all jobs →

Commenters (This Week)

1 comment
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!