Building a scraper that works is easy. Building a scraper that you can observe in production is senior work.
I added Prometheus to my Go Jobs Tracker and here's the full working implementation.
PART 1: METRICS DEFINITION
Create file metrics/metrics.go:
package metrics
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
JobsScraped = promauto.NewCounterVec(
prometheus.CounterOpts{
Name: "jobs_scraped_total",
Help: "Total number of jobs scraped per engine and status",
},
[]string{"engine", "status"},
)
ScrapeDuration = promauto.NewHistogramVec(
prometheus.HistogramOpts{
Name: "scrape_duration_seconds",
Help: "Duration of scrape jobs",
Buckets: prometheus.DefBuckets,
},
[]string{"engine"},
)
ErrorsTotal = promauto.NewCounterVec(
prometheus.CounterOpts{
Name: "scraper_errors_total",
Help: "Total errors by type",
},
[]string{"type"},
)
)
PART 2: MAIN.GO INTEGRATION - 100% WORKING
package main
import (
"net/http"
"time"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/Aureliopires186/go-jobs-tracker/metrics"
)
func main() {
go func() {
http.Handle("/metrics", promhttp.Handler())
http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200)
w.Write([]byte("ok"))
})
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("go-jobs-tracker running. check /metrics and /healthz"))
})
http.ListenAndServe(":2112", nil)
}()
start := time.Now()
// Simulate your scraping logic here
err := ScrapeGoogleJobs()
duration := time.Since(start)
if err != nil {
metrics.ErrorsTotal.WithLabelValues("scrape_failed").Inc()
} else {
metrics.JobsScraped.WithLabelValues("google_jobs", "success").Inc()
metrics.ScrapeDuration.WithLabelValues("google_jobs").Observe(duration.Seconds())
}
select {}
}
func ScrapeGoogleJobs() error {
// your logic
return nil
}
PART 3: GO.MOD DEPENDENCY
Run:
go get github.com/prometheus/client_golang@v1.19.0
go mod tidy
PART 4: HOW TO TEST IT LOCALLY
Terminal 1:
go run main.go
Terminal 2:
curl http://localhost:2112/metrics
curl http://localhost:2112/healthz
You will see:
jobs_scraped_total
scrape_duration_seconds
scraper_errors_total
PART 5: PROMETHEUS CONFIG
prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'go-jobs-tracker'
static_configs:
- targets: ['localhost:2112']
This setup is production ready, K8s ready, and used by real companies.
No more blind scraping.
•Author :Pires Aurélio
•Go Developer