FicHub — Rust Backend + SvelteKit SPA

Self-hosted fanfiction archive and download server (fichub.net replacement).
Scrapes fanfiction sites, generates EPUB, HTML, MOBI, PDF, AZW3, TXT, Markdown. Serves a full community platform via HTTP API: user accounts, bookmarks, 5-star ratings, reviews, threaded comments, Fic Requests (prompt board), reading lists, follows + updates feed, series/author pages, RSS/Atom feeds, in-browser reader, roadmap consensus, and an anti-bot defense system.

Repository: https://opencommit.eu/MagicZhang/fichub.git (public mirror)
Stack: Rust + Axum 0.8 + SQLx 0.9 + PostgreSQL + Redis + Ollama (embeddings) + SvelteKit 5 (frontend)

---

Quick Start

Prerequisites: PostgreSQL, Redis, Ollama (for consensus/auto-tagger), Rust toolchain, Node.

cp .env.example .env     # Edit with your DB/Redis URLs
cargo run                # Starts on :8000 (or $PORT)

Architecture

User/Script → Axum HTTP server (:8000)
        │
        ├─► Redis ─────────────── (rate limiter token bucket, shadowban)
        ├─► PostgreSQL ────────── (works, fic metadata, social, search, requests, lists)
        ├─► Ollama ────────────── (nomic-embed-text: roadmap consensus, auto-tagger, rec embeddings)
        ├─► Scraper subsystem ─── (FanFicFare CLI: 40+ sites; AO3/FFN blocked from this host)
        ├─► EPUB/TXT/MD builder ── (pure Rust, epub-builder + txt/md export)
        ├─► Calibre sidecar ───── (MOBI/PDF/AZW3 conversion via Docker)
        ├─► Rec strategy registry (REC_ENGINE_MODE=pluggable: RRF-blended
        │    strategies — cooccur/decay/embeddings/mf/hybrid/author_graph/
        │    tag_graph/sequential/clusters/bandit/external; legacy mode
        │    preserves the current engine exactly)
        └─► Filesystem cache ─── (hash-based directory tree)
        └─► ServeDir fallback ─── (frontend/build SPA, single origin)

Feature Highlights

• 5-star ratings + in-depth reviews — positive-only public surface; no dislikes shown; reviews/reactions feed the recommendation engine (work_feedback_signals).
• Web reader (/read/[urlId]) — typography prefs, chapter nav, scroll progress, position save, "Next Up" panel (sequel → community pick → readers-also-bookmarked).
• Fic Requests (/requests) — prompt board; works-only answers (work_id or a pasted fic URL — URL-ingest via the scraper registry), community fit votes, request-level upvotes, requester accept + notifications on answer/accept; "Request similar" button on every fic page.
• Reading lists + shelves (/lists, /shelves) — curated bundles, positioned items, blurbs.
• Follows + updates feed — follow fics/authors/users; /api/v1/feed; refresh-fic re-scrape with follower notifications.
• Series & author pages (/series/[id], /authors/[id]) — ordered works, next-in-series.
• RSS/Atom feeds — /feed.xml (new arrivals), /feed/follows.xml (token), per-fic feeds.
• PWA offline reader — service worker caches reader responses (stale-while-revalidate) + offline banner.
• Roadmap consensus (/roadmap) — MaxDiff/Elo arena over pgvector-embedded feature clusters (Ollama nomic-embed-text, 768-dim); public consensus leaderboard + most-controversial list.
• Advanced search — boolean AND/OR/NOT, phrases, fielded search, main_char_attr (AO3 "Dark Harry" semantics), typo tolerance, tag filters, kudos/comment bounds.
• Ask the Archive (/ask) — natural-language queries ("completed slow-burn Dramione over 50k, no major character death") converted by an LLM into search filters, with graceful fallback when the model is down.
• Similar-fic suggestions — on every fic page, suggest in-archive stories (autocomplete) or paste a link to be scraped; readers up/down-vote so the best matches rank first.
• Pluggable recommendation platform — strategy registry (co-occurrence, time-decayed SAR, pgvector embeddings, implicit MF, tag/author graphs, sequential, bandit exploration, curator prior) behind a single config knob; today's engine is the default cooccur strategy.
• Transparent modlog (/modlog) — every moderator/curator/admin action recorded and readable by any logged-in user; moderation is fully transparent.
• Non-PII usage analytics (/admin/analytics) — unique daily/weekly/monthly visitors, active vs view-only users, action timeline, search→export conversion; anonymous client IDs only, never IPs.
• Admin UI — /admin/* pages for auto-tag review, upload moderation, blacklist, bots, comment triage, scrapers, analytics, translation review (approve/reject/edit ML translations) and metadata correction (fix canonical title/author/status/description, synced to the default source).
• Site-as-cache — every scraped fic body persisted on the attached drive (BODY_CACHE_DIR), instant repeat exports, curator peer-voted body fixes.
• Anti-bot — honeypot traps, tiered rate limits, Redis shadowban, proof-of-work, hourly bot-scorer, /admin/bots.
• API route-walk (qa/api-walk.js) — scriptable e2e audit that enumerates every route from the router, asserts auth gates, and flags 5xx/stub responses (97/97 green).

Unified Works Model

FicHub uses a work-centric architecture where a "work" is the abstract story and a "source" is a concrete URL (e.g. AO3, FFN). The same story posted on multiple sites appears as a single canonical entry.

• works table: auto-increment PK, canonical title/author, description
• fic_info: sources linked to works via work_id FK
• Auto-merge: exact title+author match with word count tolerance
• Curator proposals: community-driven merge/split with voting
• Reputation: contributors earn points, auto-promote to curator

API Endpoints

Method	Endpoint	Purpose
GET	/api/epub?q=<url>	Export EPUB for a fic URL
GET	/api/health	Health check (DB + Redis ping)
GET	/api/meta?q=<url>	Get fic metadata
GET	/api/search?q=...	Advanced search with filters
POST	/api/search/ask	Ask the Archive: NL query → search filters → results
GET	/api/recommendations?url_id=&n=	Collaborative-filtering recs
GET	/api/recommendations/personal	Personal recs (strategy registry when enabled)
GET	/api/v1/works/{url_id}/also-bookmarked	Co-occurrence rec anchors
GET	/api/fic-suggestions?url_id=	Per-fic similar-fic suggestions (votes)
POST	/api/fic-suggestions	Suggest a similar fic (in-DB or URL scrape)
GET	/api/roadmap/consensus	Public feature consensus leaderboard
POST	/api/bookmarks	Add bookmark (work_id)
GET	/api/bookmarks	List user's bookmarks
DELETE	/api/bookmarks/{work_id}	Remove bookmark
POST	/api/ratings	Rate a work (1..=5)
GET	/api/ratings/{work_id}	Get aggregate ratings (positive-only)
POST	/api/reviews	Upsert in-depth review
GET	/api/works/{id}/reviews	List reviews
POST	/api/comments	Post comment (work_id)
GET	/api/comments/{work_id}	List comments
POST	/api/follows	Follow user/work/author
GET	/api/follows	List my follows
GET	/api/v1/feed	Follow updates feed
POST	/api/requests	Create fic request
GET	/api/requests	List requests (top/new; top ranks by upvotes)
POST	/api/requests/{id}/answers	Add work answer (work_id OR a fic URL — URL-ingest)
POST	/api/requests/{id}/answers/{aid}/vote	Vote on answer
POST	/api/requests/{id}/upvote	Upvote a request (toggle, no self-vote)
POST	/api/requests/{id}/accept/{aid}	Accept answer
GET	/api/requests/{id}/candidates	Engine suggestions for answering a request (seed work)
GET	/api/modlog	Public moderation log (any logged-in user; ?action= filter)
GET	/api/admin/analytics	Usage analytics dashboard (role ≥ 10)
GET	/api/admin/search-analytics	Search analytics: zero-result queries, trope popularity, search→export conversion
GET	/api/admin/translations	Translation review queue (role ≥ 10; approve/reject/edit)
GET	/api/admin/works/{id}/metadata	Fetch work metadata for correction
PUT	/api/admin/works/{id}/metadata	Correct canonical work metadata (syncs default source)
GET	/api/curator/content/{url_id}	Inspect cached body (curator)
POST	/api/curator/content/{url_id}/propose	Propose a body fix (peer-voted)
POST	/api/curator/content/proposals/{id}/vote	Vote on a body-fix proposal
GET	/api/series/{id}	Series detail + works
GET	/api/authors/by-name/{name}	Author bibliography
GET/POST	/api/lists	Reading lists
GET	/api/reader/{url_id}	Reader HTML bundle
GET	/feed.xml	New arrivals Atom feed
GET	/feed/follows.xml	Follows Atom feed
GET	/api/roadmap/arena	Roadmap consensus arena
POST	/api/roadmap/vote	Cast arena comparison
GET	/opds/*	OPDS catalog feeds

See docs/SPECIFICATION.md for the complete endpoint reference.

Supported Sites (Scrapers)

• FanFicFare (primary) — 40+ sites via fanficfare CLI
• Archive of Our Own (AO3) — archiveofourown.org
• FanFiction.Net — fanfiction.net
• FictionPress — fictionpress.com
• AdultFanFiction — adultfanfiction.org
• HPFanFic — hpfanfic.com
• XenForo forums — generic (SpaceBattles, SufficientVelocity)

Host limitation: from this host, only RoyalRoad/Quotev/Wattpad are reachable (AO3 returns 525, FFN 403). Wayback/relay fallback is a strategic gap.

Spec

See SPECIFICATION.md for the complete system specification — frontend, backend, database schema, deployment, API endpoints, and technical debt inventory.

Testing & Coverage

# Backend (Rust)
cargo test --lib                 # unit tests (fast, no DB)
cargo test -- --include-ignored  # + DB-gated integration tests (needs .env DB/Redis)
cargo llvm-cov --lib             # line coverage report (install: cargo install cargo-llvm-cov)

# Frontend (SvelteKit / vitest)
cd frontend
npm test                         # unit tests
npm run test:e2e                 # E2E/integration suite (real layout + nav)
npm run coverage                 # coverage report + gate (thresholds in frontend/vite.config.ts)

# QA harness (local-first, deterministic; see docs/AGENTS.md)
node qa/run.js

CI coverage gate (.forgejo/workflows/ci.yml):

• frontend-coverage — npm run coverage; fails below 65% lines / 55% funcs / 60% branches on src/lib/**.
• backend-coverage — cargo llvm-cov --lib; fails below 30% line coverage.
• Both publish reports as downloadable artifacts (frontend-coverage, backend-coverage) and a combined summary on each run. Raise thresholds as coverage grows toward the 80/70 target.

Deployment

cargo build --release
sudo systemctl restart fichub.service   # User=alvaro, WorkingDirectory=/personal/documents/code/rust/fichub

The fichub.service systemd unit runs the release binary on port 8000 (see PORT in .env), serving the API (/api/*), docs (/docs/*), and the static SvelteKit frontend (FRONTEND_DIR=./frontend/build) on one port. The public domain fichub.polarisocial.xyz is fronted by Cloudflare.

Frontend rebuild (after frontend/src changes):

cd frontend && npm run build    # writes frontend/build (served by the binary)

Database migrations

Migrations live in migrations/ and are applied by sqlx automatically on service start (sqlx::migrate!). Never insert rows into _sqlx_migrations by hand: sqlx validates each migration's checksum against the recorded value and panics with Migrate(VersionMismatch(N)) if they differ (this happened with migration 008 — a manually inserted row with an empty checksum broke startup until it was deleted so sqlx could re-apply the file). If you hit that, verify the file is what you want, then:

DELETE FROM _sqlx_migrations WHERE version = N;
-- restart the service; sqlx re-applies migrations/NNN_*.sql (keep it idempotent)

New migrations must be idempotent (CREATE TABLE IF NOT EXISTS, ON CONFLICT DO NOTHING) so a re-apply is safe.

License

AGPL-3.0-only