root-me-api/README.md

scoreboard-api

A small Go daemon that scrapes the first pages of the root-me.org scoreboard and serves them over HTTP/JSON and gRPC.

root-me.org is fronted by Anubis, a proof-of-work anti-scraper proxy. This daemon includes a native Go Anubis client — no headless browser — that transparently solves the challenge and caches the resulting auth cookie.

How it works

1. Anubis (internal/anubis) — an http.RoundTripper that detects the interstitial, solves the challenge, calls pass-challenge, stores the auth cookie, and replays the original request. root-me uses Anubis's preact challenge: the answer is SHA256(randomData) plus a short server-enforced wait (it is not a nonce proof-of-work). A nonce solver for the fast/slow challenges is also included for portability.
2. Scraper (internal/scraper) — fetches the public, login-free Rankings fragment (?page=structure&inc=modeles/classement&lang=en&ajah=1&debut_classement=N), 50 rows per page, and parses each row into an Entry.
3. Store + refresher (internal/scoreboard) — an atomically-swapped in-memory snapshot with byRank/byUsername indexes, refreshed on an interval. The last good snapshot keeps serving if a refresh fails.
4. Servers (internal/server) — gRPC and a net/http JSON mux, both reading the same store (no scraping in the request path).

Rank is the canonical 1-based position in score-descending order (root-me's own displayed rank is noisy around ties, so it is not used for ordering).

Build & run

go build ./cmd/scoreboard-apid
./scoreboard-apid                 # HTTP :8080, gRPC :9090, refresh every 10m

Configuration (flags or env)

Flag	Env	Default
-http-addr	HTTP_ADDR	:8080
-grpc-addr	GRPC_ADDR	:9090
-refresh-interval	REFRESH_INTERVAL	10m
-pages	SCOREBOARD_PAGES	4 (→ top 200)
-user-agent	USER_AGENT	a Firefox UA
-request-timeout	REQUEST_TIMEOUT	30s
-log-level	LOG_LEVEL	info

HTTP endpoints

curl localhost:8080/readyz
curl 'localhost:8080/v1/scoreboard?limit=10&offset=0'
curl localhost:8080/v1/scoreboard/rank/1
curl localhost:8080/v1/scoreboard/user/skav

gRPC

Server reflection is enabled:

grpcurl -plaintext localhost:9090 list scoreboard.v1.ScoreboardService
grpcurl -plaintext -d '{"limit":2}' localhost:9090 scoreboard.v1.ScoreboardService/ListScoreboard
grpcurl -plaintext -d '{"rank":1}'  localhost:9090 scoreboard.v1.ScoreboardService/GetByRank
grpcurl -plaintext -d '{"username":"skav"}' localhost:9090 scoreboard.v1.ScoreboardService/GetByUsername

Development

go test ./...
go generate ./...   # regenerate gen/ from proto/ (needs protoc + plugins)

cmd/anubis-smoke is a small helper that solves Anubis for a URL and dumps the page — handy for refreshing the parser test fixture.

Notes

• An official JSON API exists at https://api.www.root-me.org/classement (not behind Anubis) but requires an api_key. This daemon deliberately uses the public HTML fragment so no credentials are needed.
• Be a good citizen: the default 10-minute refresh keeps load on root-me low.