88 lines
3.3 KiB
Markdown
88 lines
3.3 KiB
Markdown
# scoreboard-api
|
|
|
|
A small Go daemon that scrapes the first pages of the
|
|
[root-me.org](https://www.root-me.org) scoreboard and serves them over **HTTP/JSON**
|
|
and **gRPC**.
|
|
|
|
root-me.org is fronted by [Anubis](https://github.com/TecharoHQ/anubis), a
|
|
proof-of-work anti-scraper proxy. This daemon includes a **native Go Anubis
|
|
client** — no headless browser — that transparently solves the challenge and
|
|
caches the resulting auth cookie.
|
|
|
|
## How it works
|
|
|
|
1. **Anubis** (`internal/anubis`) — an `http.RoundTripper` that detects the
|
|
interstitial, solves the challenge, calls `pass-challenge`, stores the auth
|
|
cookie, and replays the original request. root-me uses Anubis's `preact`
|
|
challenge: the answer is `SHA256(randomData)` plus a short server-enforced
|
|
wait (it is *not* a nonce proof-of-work). A nonce solver for the `fast`/`slow`
|
|
challenges is also included for portability.
|
|
2. **Scraper** (`internal/scraper`) — fetches the public, login-free Rankings
|
|
fragment
|
|
(`?page=structure&inc=modeles/classement&lang=en&ajah=1&debut_classement=N`),
|
|
50 rows per page, and parses each row into an `Entry`.
|
|
3. **Store + refresher** (`internal/scoreboard`) — an atomically-swapped
|
|
in-memory snapshot with `byRank`/`byUsername` indexes, refreshed on an
|
|
interval. The last good snapshot keeps serving if a refresh fails.
|
|
4. **Servers** (`internal/server`) — gRPC and a `net/http` JSON mux, both reading
|
|
the same store (no scraping in the request path).
|
|
|
|
`Rank` is the canonical 1-based position in score-descending order (root-me's
|
|
own displayed rank is noisy around ties, so it is not used for ordering).
|
|
|
|
## Build & run
|
|
|
|
```sh
|
|
go build ./cmd/scoreboard-apid
|
|
./scoreboard-apid # HTTP :8080, gRPC :9090, refresh every 10m
|
|
```
|
|
|
|
### Configuration (flags or env)
|
|
|
|
| Flag | Env | Default |
|
|
|------|-----|---------|
|
|
| `-http-addr` | `HTTP_ADDR` | `:8080` |
|
|
| `-grpc-addr` | `GRPC_ADDR` | `:9090` |
|
|
| `-refresh-interval` | `REFRESH_INTERVAL` | `10m` |
|
|
| `-pages` | `SCOREBOARD_PAGES` | `4` (→ top 200) |
|
|
| `-user-agent` | `USER_AGENT` | a Firefox UA |
|
|
| `-request-timeout` | `REQUEST_TIMEOUT` | `30s` |
|
|
| `-log-level` | `LOG_LEVEL` | `info` |
|
|
|
|
## HTTP endpoints
|
|
|
|
```sh
|
|
curl localhost:8080/readyz
|
|
curl 'localhost:8080/v1/scoreboard?limit=10&offset=0'
|
|
curl localhost:8080/v1/scoreboard/rank/1
|
|
curl localhost:8080/v1/scoreboard/user/skav
|
|
```
|
|
|
|
## gRPC
|
|
|
|
Server reflection is enabled:
|
|
|
|
```sh
|
|
grpcurl -plaintext localhost:9090 list scoreboard.v1.ScoreboardService
|
|
grpcurl -plaintext -d '{"limit":2}' localhost:9090 scoreboard.v1.ScoreboardService/ListScoreboard
|
|
grpcurl -plaintext -d '{"rank":1}' localhost:9090 scoreboard.v1.ScoreboardService/GetByRank
|
|
grpcurl -plaintext -d '{"username":"skav"}' localhost:9090 scoreboard.v1.ScoreboardService/GetByUsername
|
|
```
|
|
|
|
## Development
|
|
|
|
```sh
|
|
go test ./...
|
|
go generate ./... # regenerate gen/ from proto/ (needs protoc + plugins)
|
|
```
|
|
|
|
`cmd/anubis-smoke` is a small helper that solves Anubis for a URL and dumps the
|
|
page — handy for refreshing the parser test fixture.
|
|
|
|
## Notes
|
|
|
|
- An official JSON API exists at `https://api.www.root-me.org/classement`
|
|
(not behind Anubis) but requires an `api_key`. This daemon deliberately uses
|
|
the public HTML fragment so no credentials are needed.
|
|
- Be a good citizen: the default 10-minute refresh keeps load on root-me low.
|