- The live Vercel app shows the React search interface and result cards.
- The project notes show the React frontend, Go spider, SQLite index, and Turso upload path.
- The diagrams below show how a seed URL becomes indexed rows, then browser-searchable results.
- The current page does not claim public source repo access, active production crawl coverage, or a maintained public search index.
Try it in the browser!
The search frontend is linked below. The project notes show the crawler, indexing path, and React app that feed it.
Open in full page
View the React app notes
View the Go spider notes
What Visitors Can Inspect
Description
Challenges: Building the Go spider as a standalone CLI first, then designing the data layer so the React frontend could query indexed content efficiently. The main design decision was keeping the spider and frontend decoupled: the spider writes crawled pages into a local SQLite database, then the frontend reads the uploaded index through Turso's browser-reachable HTTP API.
Result: A two-part project: a Go spider crawls from a seed URL and indexes page titles, body text, and links into SQLite; a React frontend deployed on Vercel searches that index from the browser. The live app and project notes are linked above so visitors can inspect both sides of the build.
How It Works
Go Spider — BFS Web Crawl
════════════════════════════
seed URL
│
▼
┌──────────────┐
│ URL Queue │◄──────────────────────────────┐
└──────┬───────┘ │
│ dequeue next URL │
▼ │
┌────────────────┐ in visited set? │
│ visited set │──────► skip │
│ (hash map) │ │
└──────┬─────────┘ new URL │
│ │
▼ │
┌──────────────────┐ │
│ net/http GET │ follow redirects once │
└──────┬───────────┘ │
│ │
▼ │
┌────────────────────────────────┐ │
│ golang.org/x/net/html │ │
│ parse HTML tree │ │
│ extract: title, body text │ │
│ extract: all <a href=...> │───────────────┘
└──────────────┬─────────────────┘ enqueue new links
│
▼
┌──────────────────────┐
│ SQLite (local) │ INSERT pages(url, title, body)
│ → upload_db.py │ push magellan.sq3 → Turso
└──────────────────────┘
Visited-URL hash set was the key fix — without it the spider followed redirect loops indefinitely.
React Frontend — Search Query Flow
═════════════════════════════════════
user types in search box
│ onChange → debounced fetch
▼
Turso HTTP API (libSQL over HTTPS — reachable from browser)
│
│ SELECT url, title, body FROM pages
│ WHERE body LIKE '%query%'
│ OR title LIKE '%query%'
▼
result rows → render as card list
Why Turso instead of local SQLite?
────────────────────────────────────
Browser JS cannot open a local .db file.
Turso exposes SQLite over a REST/HTTP API
that the browser can reach with fetch().
Spider and frontend are fully decoupled — either part can be replaced independently.
In Action
Search frontend proof points
- React search UI queries indexed pages through a browser-reachable Turso endpoint.
- Debounced input avoids firing a query for every keystroke.
- Result cards render title, URL, and body matches from the same indexed record.
Spider pipeline proof points
- Go crawler fetches pages and parses links with standard HTTP and HTML packages.
- Visited-URL tracking prevents redirect loops and repeated crawls.
- Crawling, indexing, and querying stay decoupled so each layer can be replaced.
Dev Notes
Problems Solved
I wanted to understand how web indexing works end-to-end without delegating to a third-party search API. The goal was to build every layer — crawling, indexing, and querying — from scratch so nothing was a black box.
Errors & Fixes
The spider followed redirect loops and re-crawled the same URLs indefinitely — fixed with a visited-URL hash set. The React frontend couldn't query local SQLite directly; solved by migrating from local SQLite to Turso (libSQL) over HTTP, which the browser could reach.
What I Learned
Go's net/http and golang.org/x/net/html for crawling, SQLite schema design for full-text search, and how to architect a decoupled spider + frontend system where each part is independently usable. Also learned the Turso/libSQL HTTP API for browser-accessible databases.
Back to top