Go Scraper AI
Generalized Concurrent Web Scraper and AI Structuring Pipeline
COMPLETEDSOLO
Overview
A testing/learning project built to practice Go by implementing a production-style scraper pipeline with concurrent workers, dynamic JSON-schema extraction, Gemini/Ollama provider support, PostgreSQL persistence, and API/CLI interfaces.
Gallery
Project Details
ClientPersonal Learning Project
RoleGo Backend Developer
TimelineApr 2026 - Apr 2026
Duration1 Week
Technologies
GogoqueryGoogle GenAI SDKOllamaPostgreSQLpgxOpenAPIGitHub Actions
Key Features
- Dual Interface (CLI + API): Exposes the same scraping pipeline via command line and REST API endpoints.
- Dynamic Schema Extraction: Accepts runtime JSON schema and returns generalized structured outputs without hardcoded domain structs.
- Concurrent Worker Pool: Processes URL jobs concurrently with robust partial-failure handling.
- Gemini/Ollama Provider Strategy: Supports Gemini and Ollama with env-driven provider selection, retry logic, and fallback behavior.
- Persistence and Reporting: Stores extracted payloads in PostgreSQL (JSONB) and generates Markdown summaries.
- Production-Style Tooling: Includes structured logging (slog), OpenAPI spec, tests, Makefile commands, and CI workflow.