opencommit/roadmap

Watch

Deploy Anti AI Scraper Measures #15

New issue

Open

opened 2026-04-14 21:52:43 +02:00 by mvdkleijn · 2 comments

mvdkleijn commented

2026-04-14 21:52:43 +02:00

Owner

1. What

The implementation of a multi-layered defensive perimeter around OpenCommit to identify, challenge, and mitigate unauthorized automated data extraction by AI-related crawlers (e.g., GPTBot, CCBot, AnthropicAI). This includes deploying technical controls at the application level (via robots.txt), the network/proxy level (via User-Agent filtering or rate limiting), and, if required, an identity challenge layer (such as Anubis) to verify human-driven traffic.

2. Why

To protect the intellectual property and privacy of the repositories hosted on this instance by preventing uncompensated use of source code in Large Language Model (LLM) training sets. Additionally, this initiative aims to reduce infrastructure resource consumption and "noise" caused by high-frequency automated requests, ensuring higher availability and performance for legitimate human users and authorized integrations.

3. Boundaries

In-Scope:
- Configuration of robots.txt directives. (Robots.txt Traefik plugin?)
- Implementation of reverse proxy rules (Traefik) for User-Agent blocking or rate limiting.
- Evaluation and deployment of challenge-based utilities (e.g., Anubis).
- Configuration of "Good Bot" allowlists to ensure no loss of critical service functionality.
Out-of-Scope:
- Modification of the Forgejo core codebase or internal application logic.
- Blocking of legitimate human users or authenticated developers.
- Decommissioning of search engine visibility (unless specifically identified as a risk during implementation).
- Any measures that would break essential CI/CD webhooks or third-party integrations.

4. Definition of Done

Deployment: A multi-layered defense strategy is active and operational across the edge/proxy layer.
Shadowing: Initial deployment of measures is in a "shadow" or "log-only" mode so administrators can verify that normal traffic remains unaffected.
Validation: Verification testing confirms that known AI scrapers (e.g., GPTBot) are either blocked or successfully challenged by the new layers.
Integrity Check: An audit of "Good Bot" traffic (Search Engines, CI/CD webhooks, and Archive bots) confirms they remain unaffected and can still access necessary resources.
Observability: Monitoring or logging is configured to track the frequency of blocked attempts and any impact on server latency/resource usage.
Documentation: The new security architecture, including the "allowlist" of permitted bots, is documented for future maintenance.

## 1. What The implementation of a multi-layered defensive perimeter around OpenCommit to identify, challenge, and mitigate unauthorized automated data extraction by AI-related crawlers (e.g., GPTBot, CCBot, AnthropicAI). This includes deploying technical controls at the application level (via `robots.txt`), the network/proxy level (via User-Agent filtering or rate limiting), and, if required, an identity challenge layer (such as Anubis) to verify human-driven traffic. ## 2. Why To protect the intellectual property and privacy of the repositories hosted on this instance by preventing uncompensated use of source code in Large Language Model (LLM) training sets. Additionally, this initiative aims to reduce infrastructure resource consumption and "noise" caused by high-frequency automated requests, ensuring higher availability and performance for legitimate human users and authorized integrations. ## 3. Boundaries * **In-Scope:** * Configuration of `robots.txt` directives. (Robots.txt Traefik plugin?) * Implementation of reverse proxy rules (Traefik) for User-Agent blocking or rate limiting. * Evaluation and deployment of challenge-based utilities (e.g., Anubis). * Configuration of "Good Bot" allowlists to ensure no loss of critical service functionality. * **Out-of-Scope:** * Modification of the Forgejo core codebase or internal application logic. * Blocking of legitimate human users or authenticated developers. * Decommissioning of search engine visibility (unless specifically identified as a risk during implementation). * Any measures that would break essential CI/CD webhooks or third-party integrations. ## 4. Definition of Done * [ ] **Deployment:** A multi-layered defense strategy is active and operational across the edge/proxy layer. * [ ] **Shadowing:** Initial deployment of measures is in a "shadow" or "log-only" mode so administrators can verify that normal traffic remains unaffected. * [ ] **Validation:** Verification testing confirms that known AI scrapers (e.g., GPTBot) are either blocked or successfully challenged by the new layers. * [ ] **Integrity Check:** An audit of "Good Bot" traffic (Search Engines, CI/CD webhooks, and Archive bots) confirms they remain unaffected and can still access necessary resources. * [ ] **Observability:** Monitoring or logging is configured to track the frequency of blocked attempts and any impact on server latency/resource usage. * [ ] **Documentation:** The new security architecture, including the "allowlist" of permitted bots, is documented for future maintenance.

mvdkleijn added the

labels

2026-04-14 21:52:43 +02:00

mvdkleijn added this to the Highlevel Roadmap project

2026-04-14 21:52:43 +02:00

mvdkleijn self-assigned this

2026-05-21 16:19:56 +02:00

benoit commented

2026-09-19 15:01:53 +02:00

Owner

Is this an idea? Wish we were able to test it, maybe just with a simple setup before we use it within opencommit?

Recommended layout

Put Anubis between your public Ingress and Forgejo’s HTTP service:

Internet
  → Traefik Ingress (TLS)
  → Anubis Service / Deployment
  → existing Forgejo ClusterIP Service
  → Forgejo pods

Do not expose the Forgejo Service directly through an Ingress once Anubis is enabled. Your public hostname, opencommit.eu, should route to Anubis instead.

Anubis is an HTTP reverse proxy that presents a JavaScript proof-of-work challenge to traffic matching its bot policies, then proxies accepted requests upstream. It does not protect SSH Git access, so git@… / port 22 needs its own access and rate-limit policy. The project describes it as deliberately broad protection that may also affect legitimate crawlers. See Anubis.

1. Inspect the current Forgejo Service

First identify the exact namespace, service name, and port that currently reaches Forgejo:

kubectl get namespaces
kubectl -n <forgejo-namespace> get svc
kubectl -n <forgejo-namespace> get ingress

You need the HTTP Forgejo service name and port—not the SSH port. In the manifests below, replace:

<forgejo-namespace> with your namespace
<forgejo-http-service> with the existing Forgejo HTTP Service name
<forgejo-http-port> with the existing Service port
<anubis-version> with a reviewed, pinned Anubis release version

Avoid latest; pin a release and upgrade deliberately.

2. Deploy Anubis

Save the following as anubis.yaml after replacing the placeholders:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: anubis
  namespace: <forgejo-namespace>
  labels:
    app.kubernetes.io/name: anubis
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: anubis
  template:
    metadata:
      labels:
        app.kubernetes.io/name: anubis
    spec:
      containers:
        - name: anubis
          image: ghcr.io/techarohq/anubis:<anubis-version>
          imagePullPolicy: IfNotPresent
          env:
            - name: BIND
              value: ":8080"
            - name: TARGET
              value: "http://<forgejo-http-service>:<forgejo-http-port>"
            - name: SERVE_ROBOTS_TXT
              value: "true"
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            tcpSocket:
              port: http
          livenessProbe:
            tcpSocket:
              port: http
          resources:
            requests:
              cpu: 50m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
  name: anubis
  namespace: <forgejo-namespace>
spec:
  selector:
    app.kubernetes.io/name: anubis
  ports:
    - name: http
      port: 80
      targetPort: http

Apply it:

kubectl apply -f anubis.yaml
kubectl -n <forgejo-namespace> rollout status deployment/anubis
kubectl -n <forgejo-namespace> get pods -l app.kubernetes.io/name=anubis
kubectl -n <forgejo-namespace> logs deployment/anubis

3. Point `opencommit.eu` at Anubis

Create or modify the Ingress for opencommit.eu so that its backend is anubis, not Forgejo.

This is a standard K3s/Traefik-compatible Ingress using a pre-existing TLS secret:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: opencommit-eu
  namespace: <forgejo-namespace>
spec:
  ingressClassName: traefik
  tls:
    - hosts:
        - opencommit.eu
      secretName: <existing-tls-secret>
  rules:
    - host: opencommit.eu
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: anubis
                port:
                  name: http

Apply it:

kubectl apply -f opencommit-ingress.yaml
kubectl -n <forgejo-namespace> describe ingress opencommit-eu

If your TLS is currently managed by Traefik/cert-manager annotations rather than an existing secret, retain those existing annotations and TLS settings; change only the backend Service from Forgejo to anubis.

4. Keep Forgejo aware it is behind proxies

Your effective proxy chain will be:

Traefik → Anubis → Forgejo

Confirm these details in your Forgejo deployment/configuration:

ROOT_URL remains https://opencommit.eu/.
Forgejo is configured to trust the reverse proxies that sit in front of it.
Forgejo’s original Service remains ClusterIP; it should not also be public through another Ingress, LoadBalancer, or NodePort.
Anubis sees the original visitor address through forwarded client-IP headers from Traefik.

The client IP is important: if Anubis only sees Traefik/pod IPs, users can get inconsistent challenges or share challenge state incorrectly.

5. Test before making it your only route

Run these checks after applying:

curl -I https://opencommit.eu/
git ls-remote https://opencommit.eu/<owner>/<repository>.git
kubectl -n <forgejo-namespace> logs deployment/anubis --tail=200

Also test in a normal browser:

An initial page request should receive an Anubis challenge if its policy decides to challenge it.
After completing the challenge, Forgejo’s UI should load normally.
Log in, browse repositories, view commits/files, clone over HTTPS, and push to a test repository.
Test Forgejo Actions/CI callbacks, webhooks, Git LFS, package/OCI registry use, and any API clients you operate.

Important Forgejo caveats

HTTP Git and API clients are not browsers. Ensure Anubis policy does not make non-browser Git tooling, Forgejo runners, webhooks, package clients, or internal automation solve a JavaScript challenge.
OCI/package registry: Forgejo’s registry uses /v2/. An Anubis issue reports this route being challenged in at least one version, which breaks normal OCI tooling because it expects JSON, not an HTML challenge page. Treat /v2/ as a route requiring explicit validation or a bypass policy. See Forgejo / OCI registry regression.
Open Graph previews: Anubis has a documented concern where Open Graph fetching could cause the Forgejo upstream to be contacted despite a challenge. The referenced mitigation is disabling Open Graph passthrough if you encounter it. See Anubis issue #435.
Current compatibility: There is also a June 2026 Forgejo-related issue describing repeated challenges and failed browser fetches under a default setup. Test the exact Anubis release you pin against your Forgejo version in a staging host before switching production traffic. See issue #1687.

Suggested rollout

Deploy Anubis and its Service without changing public traffic.
Use a temporary staging hostname that routes through Anubis to validate normal Forgejo behaviour.
Change the opencommit.eu Ingress backend to anubis.
Monitor Anubis and Forgejo logs.
Keep a copy of the previous Ingress manifest so rollback is only a backend-Service change.

The core configuration is just BIND, TARGET, and SERVE_ROBOTS_TXT; those variables are used in a known Forgejo + Anubis deployment example, where Anubis proxies to Forgejo and the external reverse proxy forwards the real client IP. See this Forgejo integration example.

Is this an idea? Wish we were able to test it, maybe just with a simple setup before we use it within opencommit? ### Recommended layout Put Anubis **between your public Ingress and Forgejo’s HTTP service**: ```text Internet → Traefik Ingress (TLS) → Anubis Service / Deployment → existing Forgejo ClusterIP Service → Forgejo pods ``` Do **not** expose the Forgejo Service directly through an Ingress once Anubis is enabled. Your public hostname, `opencommit.eu`, should route to Anubis instead. Anubis is an HTTP reverse proxy that presents a JavaScript proof-of-work challenge to traffic matching its bot policies, then proxies accepted requests upstream. It does not protect SSH Git access, so `git@…` / port 22 needs its own access and rate-limit policy. The project describes it as deliberately broad protection that may also affect legitimate crawlers. See [Anubis](https://github.com/TecharoHQ/anubis). ### 1. Inspect the current Forgejo Service First identify the exact namespace, service name, and port that currently reaches Forgejo: ```bash kubectl get namespaces kubectl -n <forgejo-namespace> get svc kubectl -n <forgejo-namespace> get ingress ``` You need the **HTTP** Forgejo service name and port—not the SSH port. In the manifests below, replace: - `<forgejo-namespace>` with your namespace - `<forgejo-http-service>` with the existing Forgejo HTTP Service name - `<forgejo-http-port>` with the existing Service port - `<anubis-version>` with a reviewed, pinned Anubis release version Avoid `latest`; pin a release and upgrade deliberately. ### 2. Deploy Anubis Save the following as `anubis.yaml` after replacing the placeholders: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: anubis namespace: <forgejo-namespace> labels: app.kubernetes.io/name: anubis spec: replicas: 2 selector: matchLabels: app.kubernetes.io/name: anubis template: metadata: labels: app.kubernetes.io/name: anubis spec: containers: - name: anubis image: ghcr.io/techarohq/anubis:<anubis-version> imagePullPolicy: IfNotPresent env: - name: BIND value: ":8080" - name: TARGET value: "http://<forgejo-http-service>:<forgejo-http-port>" - name: SERVE_ROBOTS_TXT value: "true" ports: - name: http containerPort: 8080 readinessProbe: tcpSocket: port: http livenessProbe: tcpSocket: port: http resources: requests: cpu: 50m memory: 128Mi limits: cpu: 500m memory: 512Mi --- apiVersion: v1 kind: Service metadata: name: anubis namespace: <forgejo-namespace> spec: selector: app.kubernetes.io/name: anubis ports: - name: http port: 80 targetPort: http ``` Apply it: ```bash kubectl apply -f anubis.yaml kubectl -n <forgejo-namespace> rollout status deployment/anubis kubectl -n <forgejo-namespace> get pods -l app.kubernetes.io/name=anubis kubectl -n <forgejo-namespace> logs deployment/anubis ``` ### 3. Point `opencommit.eu` at Anubis Create or modify the Ingress for `opencommit.eu` so that its backend is `anubis`, not Forgejo. This is a standard K3s/Traefik-compatible Ingress using a pre-existing TLS secret: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: opencommit-eu namespace: <forgejo-namespace> spec: ingressClassName: traefik tls: - hosts: - opencommit.eu secretName: <existing-tls-secret> rules: - host: opencommit.eu http: paths: - path: / pathType: Prefix backend: service: name: anubis port: name: http ``` Apply it: ```bash kubectl apply -f opencommit-ingress.yaml kubectl -n <forgejo-namespace> describe ingress opencommit-eu ``` If your TLS is currently managed by Traefik/cert-manager annotations rather than an existing secret, **retain those existing annotations and TLS settings**; change only the backend Service from Forgejo to `anubis`. ### 4. Keep Forgejo aware it is behind proxies Your effective proxy chain will be: ```text Traefik → Anubis → Forgejo ``` Confirm these details in your Forgejo deployment/configuration: - `ROOT_URL` remains `https://opencommit.eu/`. - Forgejo is configured to trust the reverse proxies that sit in front of it. - Forgejo’s original Service remains `ClusterIP`; it should not also be public through another Ingress, LoadBalancer, or NodePort. - Anubis sees the original visitor address through forwarded client-IP headers from Traefik. The client IP is important: if Anubis only sees Traefik/pod IPs, users can get inconsistent challenges or share challenge state incorrectly. ### 5. Test before making it your only route Run these checks after applying: ```bash curl -I https://opencommit.eu/ git ls-remote https://opencommit.eu/<owner>/<repository>.git kubectl -n <forgejo-namespace> logs deployment/anubis --tail=200 ``` Also test in a normal browser: 1. An initial page request should receive an Anubis challenge if its policy decides to challenge it. 2. After completing the challenge, Forgejo’s UI should load normally. 3. Log in, browse repositories, view commits/files, clone over HTTPS, and push to a test repository. 4. Test Forgejo Actions/CI callbacks, webhooks, Git LFS, package/OCI registry use, and any API clients you operate. ### Important Forgejo caveats - **HTTP Git and API clients are not browsers.** Ensure Anubis policy does not make non-browser Git tooling, Forgejo runners, webhooks, package clients, or internal automation solve a JavaScript challenge. - **OCI/package registry:** Forgejo’s registry uses `/v2/`. An Anubis issue reports this route being challenged in at least one version, which breaks normal OCI tooling because it expects JSON, not an HTML challenge page. Treat `/v2/` as a route requiring explicit validation or a bypass policy. See [Forgejo / OCI registry regression](https://github.com/TecharoHQ/anubis/issues/1231). - **Open Graph previews:** Anubis has a documented concern where Open Graph fetching could cause the Forgejo upstream to be contacted despite a challenge. The referenced mitigation is disabling Open Graph passthrough if you encounter it. See [Anubis issue #435](https://github.com/TecharoHQ/anubis/issues/435). - **Current compatibility:** There is also a June 2026 Forgejo-related issue describing repeated challenges and failed browser fetches under a default setup. Test the exact Anubis release you pin against your Forgejo version in a staging host before switching production traffic. See [issue #1687](https://github.com/TecharoHQ/anubis/issues/1687). ### Suggested rollout 1. Deploy Anubis and its Service without changing public traffic. 2. Use a temporary staging hostname that routes through Anubis to validate normal Forgejo behaviour. 3. Change the `opencommit.eu` Ingress backend to `anubis`. 4. Monitor Anubis and Forgejo logs. 5. Keep a copy of the previous Ingress manifest so rollback is only a backend-Service change. The core configuration is just `BIND`, `TARGET`, and `SERVE_ROBOTS_TXT`; those variables are used in a known Forgejo + Anubis deployment example, where Anubis proxies to Forgejo and the external reverse proxy forwards the real client IP. See [this Forgejo integration example](https://eigenwijsje.dev/til/protect-forgejo-with-anubis/).

mvdkleijn commented

2026-09-19 15:16:06 +02:00

Author

Owner

This is definitely an idea. I had Anubis in my head. We could probably develop and test this setup on a local kind cluster?

I was also looking at: https://codeberg.org/gone/go-away

Not sure what Codeberg is using nowadays. I think they moved away from Anubis because it was deemed too inflexible. Maybe we should check their infra repo.

This is definitely an idea. I had Anubis in my head. We could probably develop and test this setup on a local kind cluster? I was also looking at: https://codeberg.org/gone/go-away Not sure what Codeberg is using nowadays. I think they moved away from Anubis because it was deemed too inflexible. Maybe we should check their infra repo.