JackPrince/GoBDLogBook
JackPrince/GoBDLogBook
Watch
1
1
Fork
You've already forked GoBDLogBook
 0
Code Issues 3 Pull requests Projects Releases Wiki Activity Actions

Feature: Proper GEO Coding and Maps #1

New issue
Closed
opened 2026-09-03 13:37:01 +02:00 by JackPrince · 2 comments
JackPrince commented 2026-09-03 13:37:01 +02:00
Owner
Copy link

Background & Problem

The current geocoding and map tile implementation is basic and lacks flexibility. More importantly, research and community feedback suggest that our current usage does not strictly adhere to the OpenStreetMap Foundation's Usage Policy.

If we continue to use the public OSM endpoints without proper rate limiting, caching, and identifiable User-Agents, our server IP risks being permanently blocked. This needs to be addressed immediately to ensure the stability of the application.

Proposed Solution

We will refactor the geocoding and map tile logic by implementing a robust Manager pattern combined with caching and usage guards. This architecture will support a multi-tenant setup, allowing organization admins (or a super admin) to configure their preferred commercial or self-hosted map provider.

Architecture Highlights

  1. Manager Pattern (GeoManager): Dynamically resolves the correct driver based on tenant settings (with a fallback to system defaults).
  2. Usage Guard (GeoUsageGuard): Acts as a middleware/throttler to ensure we never hit API limits (e.g., strictly enforcing Nominatim's 1 req/s rule) and tracks usage metrics.
  3. Caching Layer (CachedGeocoder): Prevents redundant API calls by caching coordinates. Coordinates will be rounded (e.g., to 4 decimals) to significantly increase cache hit rates for recurring parking spots/routes.

Architecture Flow

                                  Client (React / Leaflet / Vue)
                                      │              │
                       GET /api/geo/reverse   GET /api/geo/tile-config
                       GET /api/geo/search           │
                                      ▼              ▼
                               ┌──────────────────────────────────┐
                               │       GeoController              │
                               └─────────────┬────────────────────┘
                                             │
                                             ▼
                               ┌──────────────────────────────────┐
                               │       CachedGeocoder             │
                               │  (Cache-Lookup: 4 Decimals, TTL) │
                               └─────────────┬────────────────────┘
                                             │ Cache Miss
                                             ▼
                               ┌──────────────────────────────────┐
                               │       GeoUsageGuard              │
                               │  - Pre-emptive Quota Check       │
                               │  - Nominatim 1 req/s Throttler   │
                               │  - Usage Metrics Counter         │
                               └─────────────┬────────────────────┘
                                             │
                                             ▼
                               ┌──────────────────────────────────┐
                               │       GeoManager (Manager)       │
                               │  (Tenant Setting -> System Fallb)│
                               └──────┬──────┬──────┬──────┬──────┘
                                      │      │      │      │
            ┌─────────────────────────┼──────┼──────┼──────┴──────────────────────────┐
            ▼                         ▼      ▼      ▼                                 ▼
      NominatimDriver            LocationIQ  OpenCage  Geoapify / Stadia / MapTiler   NullDriver
 (OSM User-Agent + 1 req/s)     (Drop-In)  (Structured)      (GeoJSON / Vector)    (Offline/Test)

Tasks

  • Add map_settings JSON column to the tenant/organization database model.
  • Define the GeocoderInterface.
  • Implement GeoManager to resolve drivers based on tenant context.
  • Implement core drivers: NominatimDriver (enforcing User-Agent) and at least one commercial fallback like OpenCageDriver or LocationIQDriver.
  • Implement NullDriver for automated testing.
  • Implement CachedGeocoder wrapper to handle rounding and Redis/DB caching.
  • Implement GeoUsageGuard to handle rate-limiting (especially the strict 1 req/s for Nominatim).
  • Update frontend settings UI to allow admins to select their provider and enter API keys/contact emails.
  • Migrate existing geocoding calls in jobs/controllers to use the new architecture.
## Background & Problem The current geocoding and map tile implementation is basic and lacks flexibility. More importantly, research and community feedback suggest that our current usage does not strictly adhere to the [OpenStreetMap Foundation's Usage Policy](https://operations.osmfoundation.org/policies/). If we continue to use the public OSM endpoints without proper rate limiting, caching, and identifiable User-Agents, our server IP risks being permanently blocked. This needs to be addressed immediately to ensure the stability of the application. ## Proposed Solution We will refactor the geocoding and map tile logic by implementing a robust **Manager pattern** combined with caching and usage guards. This architecture will support a multi-tenant setup, allowing organization admins (or a super admin) to configure their preferred commercial or self-hosted map provider. ### Architecture Highlights 1. **Manager Pattern (`GeoManager`):** Dynamically resolves the correct driver based on tenant settings (with a fallback to system defaults). 2. **Usage Guard (`GeoUsageGuard`):** Acts as a middleware/throttler to ensure we never hit API limits (e.g., strictly enforcing Nominatim's 1 req/s rule) and tracks usage metrics. 3. **Caching Layer (`CachedGeocoder`):** Prevents redundant API calls by caching coordinates. Coordinates will be rounded (e.g., to 4 decimals) to significantly increase cache hit rates for recurring parking spots/routes. ## Architecture Flow ```text Client (React / Leaflet / Vue) │ │ GET /api/geo/reverse GET /api/geo/tile-config GET /api/geo/search │ ▼ ▼ ┌──────────────────────────────────┐ │ GeoController │ └─────────────┬────────────────────┘ │ ▼ ┌──────────────────────────────────┐ │ CachedGeocoder │ │ (Cache-Lookup: 4 Decimals, TTL) │ └─────────────┬────────────────────┘ │ Cache Miss ▼ ┌──────────────────────────────────┐ │ GeoUsageGuard │ │ - Pre-emptive Quota Check │ │ - Nominatim 1 req/s Throttler │ │ - Usage Metrics Counter │ └─────────────┬────────────────────┘ │ ▼ ┌──────────────────────────────────┐ │ GeoManager (Manager) │ │ (Tenant Setting -> System Fallb)│ └──────┬──────┬──────┬──────┬──────┘ │ │ │ │ ┌─────────────────────────┼──────┼──────┼──────┴──────────────────────────┐ ▼ ▼ ▼ ▼ ▼ NominatimDriver LocationIQ OpenCage Geoapify / Stadia / MapTiler NullDriver (OSM User-Agent + 1 req/s) (Drop-In) (Structured) (GeoJSON / Vector) (Offline/Test) ``` # Tasks - [x] Add map_settings JSON column to the tenant/organization database model. - [x] Define the GeocoderInterface. - [x] Implement GeoManager to resolve drivers based on tenant context. - [x] Implement core drivers: NominatimDriver (enforcing User-Agent) and at least one commercial fallback like OpenCageDriver or LocationIQDriver. - [x] Implement NullDriver for automated testing. - [x] Implement CachedGeocoder wrapper to handle rounding and Redis/DB caching. - [x] Implement GeoUsageGuard to handle rate-limiting (especially the strict 1 req/s for Nominatim). - [x] Update frontend settings UI to allow admins to select their provider and enter API keys/contact emails. - [x] Migrate existing geocoding calls in jobs/controllers to use the new architecture.
JackPrince commented 2026-09-04 11:08:49 +02:00
Author
Owner
Copy link

Implementation Summary & Architecture Evolution

During implementation in feat/geo-provider-pools, the architecture evolved beyond a basic manager pattern into a full Multi-Tenant Provider Pool System to properly address data isolation and security:

  1. Separation of Concerns:
    • Geocoding (address resolution) is strictly routed through tenant provider pools.
    • Map Tiles are decoupled and resolved via GET /api/geo/tile-config (with osm_public as default).
  2. Multi-Tenant Hierarchy & Encryption:
    • Super-Admins manage a global System Pool.
    • Tenants can inherit the system pool or configure their own isolated pool.
    • API keys are securely encrypted at rest in companies.map_settings_secret.
  3. Provider Support & Failover:
    • Supports 6 providers: Nominatim, OpenCage, LocationIQ, Geoapify, Stadia Maps, and MapTiler.
    • Enforces Nominatim's strict 1 req/s policy and contact User-Agent header.
    • Automatic failover if a provider hits daily quota limits.
  4. Caching:
    • 4-decimal coordinate rounding with 30-day cache for recurring parking/stops.

Merged into dev via GitLab CI with full unit and feature test coverage. Detailed architecture documented in docs/geo-provider-pools.md.

### Implementation Summary & Architecture Evolution During implementation in `feat/geo-provider-pools`, the architecture evolved beyond a basic manager pattern into a full **Multi-Tenant Provider Pool System** to properly address data isolation and security: 1. **Separation of Concerns:** - **Geocoding** (address resolution) is strictly routed through tenant provider pools. - **Map Tiles** are decoupled and resolved via `GET /api/geo/tile-config` (with `osm_public` as default). 2. **Multi-Tenant Hierarchy & Encryption:** - Super-Admins manage a global **System Pool**. - Tenants can inherit the system pool or configure their own isolated pool. - API keys are securely encrypted at rest in `companies.map_settings_secret`. 3. **Provider Support & Failover:** - Supports 6 providers: **Nominatim**, **OpenCage**, **LocationIQ**, **Geoapify**, **Stadia Maps**, and **MapTiler**. - Enforces Nominatim's strict 1 req/s policy and contact User-Agent header. - Automatic failover if a provider hits daily quota limits. 4. **Caching:** - 4-decimal coordinate rounding with 30-day cache for recurring parking/stops. Merged into `dev` via GitLab CI with full unit and feature test coverage. Detailed architecture documented in `docs/geo-provider-pools.md`.
JackPrince commented 2026-09-04 13:45:57 +02:00
Author
Owner
Copy link

✅ Resolved and Merged into main

The geocoding and map tile provider management feature has been fully implemented, tested, and merged into the production main branch (commit 174fb224).

📦 Deliverables Summary

  1. Provider Pools & Fallback Chains

    • Drivers Implemented: OpenStreetMap / Nominatim (default zero-config), MapTiler, OpenCage, Geoapify, LocationIQ, and Stadia Maps.
    • Multi-tier Pool Logic: Configurable primary, secondary, and fallback providers with automatic failover upon network errors, HTTP 429 rate limits, or monthly quota exhaustion.
    • Tenant Isolation: Per-company provider selection and encrypted credential storage (map_settings_secret).

  2. Security & Privacy Safeguards

    • Tile reverse-proxy endpoint (/api/geo/tile/{z}/{x}/{y}) prevents client-side third-party API key exposure.
    • Per-company daily and monthly quota guards with automatic audit metrics logged to geo_usage_metrics.

  3. Frontend & User Experience

    • Dedicated Geo & Map Settings management page (/admin/geo-settings) with live provider connectivity test buttons.
    • Interactive OpenStreetMap route visualizer embedded in Trip Details.
    • Optimized paginated trip list view with indexed database lookups for smooth scrolling and responsive rendering.

  4. Testing & Quality Gates

    • 155 automated PHPUnit and Vitest test suites passing across all provider drivers, fallback scenarios, and API endpoints.
    • Full OpenAPI specification updated (docs/openapi.yaml).
    • Production Docker images built and pushed to the GitLab Container Registry.

Closing this issue as complete.

### ✅ Resolved and Merged into `main` The geocoding and map tile provider management feature has been fully implemented, tested, and merged into the production `main` branch ([commit `174fb224`](https://gitlab.est-in.eu/daniel/gobd-logbook/-/commit/174fb224)). --- #### 📦 Deliverables Summary 1. **Provider Pools & Fallback Chains** - **Drivers Implemented**: OpenStreetMap / Nominatim (default zero-config), MapTiler, OpenCage, Geoapify, LocationIQ, and Stadia Maps. - **Multi-tier Pool Logic**: Configurable primary, secondary, and fallback providers with automatic failover upon network errors, HTTP 429 rate limits, or monthly quota exhaustion. - **Tenant Isolation**: Per-company provider selection and encrypted credential storage (`map_settings_secret`). 2. **Security & Privacy Safeguards** - Tile reverse-proxy endpoint (`/api/geo/tile/{z}/{x}/{y}`) prevents client-side third-party API key exposure. - Per-company daily and monthly quota guards with automatic audit metrics logged to `geo_usage_metrics`. 3. **Frontend & User Experience** - Dedicated **Geo & Map Settings** management page (`/admin/geo-settings`) with live provider connectivity test buttons. - Interactive OpenStreetMap route visualizer embedded in Trip Details. - Optimized paginated trip list view with indexed database lookups for smooth scrolling and responsive rendering. 4. **Testing & Quality Gates** - 155 automated PHPUnit and Vitest test suites passing across all provider drivers, fallback scenarios, and API endpoints. - Full OpenAPI specification updated (`docs/openapi.yaml`). - Production Docker images built and pushed to the GitLab Container Registry. --- **Closing this issue as complete.**
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
dev
No results found.
Labels
Clear labels   
Compat/Breaking
Breaking change that won't be backward compatible

  
Kind/Bug
Something is not working

  
Kind/Documentation
Documentation changes

  
Kind/Enhancement
Improve existing functionality

  
Kind/Feature
New functionality

  
Kind/Security
This is security issue

  
Kind/Testing
Issue or pull request related to testing

  
Priority
Critical
 The priority is critical

  
Priority
High
 The priority is high

  
Priority
Low
 The priority is low

  
Priority
Medium
 The priority is medium

  
Reviewed
Confirmed
 Issue has been confirmed

  
Reviewed
Duplicate
 This issue or pull request already exists

  
Reviewed
Invalid
 Invalid issue

  
Reviewed
Won't Fix
 This issue won't be fixed

  
Status
Abandoned
 Somebody has started to work on this but abandoned work

  
Status
Blocked
 Something is blocking this issue or pull request

  
Status
Need More Info
 Feedback is required to reproduce issue or to continue work

No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
JackPrince
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
JackPrince/GoBDLogBook#1
No description provided.