KJA·2026 · Case study

Go Scraper AI

Generalized Concurrent Web Scraper and AI Structuring Pipeline

GoBackendWeb ScrapingAI IntegrationAPI DevelopmentLearning Project

Year

2026

Role

Go Backend Developer

Stack

Go · goquery · Google GenAI SDK · Ollama · PostgreSQL · pgx · OpenAPI · GitHub Actions

Duration

1 Week

Status

Completed

Type

Solo

View code

The brief

A testing/learning project built to practice Go by implementing a production-style scraper pipeline with concurrent workers, dynamic JSON-schema extraction, Gemini/Ollama provider support, PostgreSQL persistence, and API/CLI interfaces.

Approach

A concurrent worker pool handles URL jobs with partial-failure isolation; runtime JSON schemas get passed to an AI provider (Gemini or local Ollama) for extraction, with env-driven provider selection and retry logic.

Outcome

Structured logging, an OpenAPI spec, tests, and a CI workflow — built to the same standard as a shipped product, not a throwaway script.

Key features

Dual Interface (CLI + API)

Exposes the same scraping pipeline via command line and REST API endpoints.

Dynamic Schema Extraction

Accepts runtime JSON schema and returns generalized structured outputs without hardcoded domain structs.

Concurrent Worker Pool

Processes URL jobs concurrently with robust partial-failure handling.

Gemini/Ollama Provider Strategy

Supports Gemini and Ollama with env-driven provider selection, retry logic, and fallback behavior.

Persistence and Reporting

Stores extracted payloads in PostgreSQL (JSONB) and generates Markdown summaries.

Production-Style Tooling

Includes structured logging (slog), OpenAPI spec, tests, Makefile commands, and CI workflow.

Team

Kalhara JADeveloper

Gallery