fanfic-scrapers

General-purpose fanfiction scraping library (AGPL-3.0), extracted from the FicHub codebase and made site-agnostic so any project can use it.

What it does

• Shared, site-agnostic metadata model: FicMetadata, Chapter, ExtractedTag, ScrapeError.
• Pluggable adapter trait SiteScraper — implement it for your own site and register it.
• A Registry that routes URLs to the right adapter (native scrapers preferred, FanFicFare as catch-all).
• Native adapters: AO3, FanFiction.net, FictionPress, RoyalRoad, XenForo boards (+XenForo2: alternatehistory, althistory, the-sietch), HPFanficArchive, adultfanfiction.org, FimFiction, Literotica, ScribbleHub, WordPress-novel sites, the eFiction family (19 niche archives), the OTW-family siblings (adastrafanfic, cfaa, squidgeworld, superlove), FicBook, Wattpad, SpiritFanfiction, Syosetu, AsianFanFics, DeviantArt, Twisting the Hellmouth, StoriesOnline family (storiesonline.net, scifistories, storyroom), Quotev, Fanfictions.fr, Kakuyomu, MediaMiner, ChiReads, PMDFanFiction, LCFanFic, FireflyFans, FicWad, FictionMania, FanFiktion.de, TouchFluffyTail, Fanfics.me, FanFicAuthors, FictionHunt, InkBunny, SoFurry, Dokuga, PhoenixSong, StoriesOfArda, Fictionalley, NovelAll, MassEffect2.in, ReadonlyMind, UtopiaStories, ASexStories, AnEroticStory, MCStories, HentaiFoundry, BDSMLibrary, and the eFiction-variant family (26 viewstory.php?sid= archives: psychfic, wolverineandrogue, sycophanthex sites, walkingtheplank, themasque, ksarchive, twilighted, whofic, etc.) — full FanFicFare adapter parity (all 107 real sites). Login support for author/adult-gated sites (fanfics.me, fictionhunt, inkbunny, sofurry, dokuga) + is_adult flag for adult archives (readonlymind, utopiastories, asexstories, aneroticstory, mcstories, hentaifoundry, bdsmlibrary).
• FanFicFare integration via its --meta-only CLI output (covers 100+ sites through FFF's adapter set).
• Optional LLM-assisted author-profile-link extraction (author_link module) — own generic HTML patterns first, LLM fallback for unknown sites. The LLM interface is a tiny LlmClient trait so hosts plug in their own provider (Ollama, OpenAI, ...).

Usage

use fanfic_scrapers::{Registry, NoopHealHook};
use fanfic_scrapers::sites;

let mut registry = Registry::new();
// Catch-all first: FanFicFare handles any URL.
registry.register(Box::new(sites::fanficfare::FanFicFareScraper));
// Native scrapers preferred for sites they know.
registry.register(Box::new(sites::ao3::Ao3Scraper));
registry.register(Box::new(sites::ffnet::FfNetScraper));
// ...

let client = reqwest::Client::new();
let meta = registry
    .lookup(&client, "https://archiveofourown.org/works/21845264", None)
    .await?;

let chapters = registry.fetch_chapters(&client, &meta).await?;

License

AGPL-3.0-or-later. The library itself carries no FicHub code or types; it is a standalone reusable component.