Aves dataset — refined tasks¶

Source idea (verbatim from the request):

we want to enrich our map with this imense dataset - but allow rich controls (e.g. filtering layers for different marker types) and ensure a proper performance.

What the dataset actually is¶

data/avesmaps.map-features.json is a single ~1.2M-line JSON document with four top-level keys:

Facts verified by inspection, not assumed:

• Geometry is only Point and LineString. No Polygon, MultiPolygon, MultiLineString, MultiPoint or GeometryCollection occurs.
• feature_type has six values, not four as first assumed: location (2717), crossing (792), junction (1239), path (5720), plus two missed on first inspection — label (659, text labels for regions/mountains/ rivers/seas etc., geometry Point, 24 subtypes such as gebirge, fluss, meer, wueste) and powerline (163, magical ley lines between locations, geometry LineString, single subtype powerline). Task 1's shards and manifest cover all six.
• feature_subtype is the useful axis: metropole, grossstadt, stadt, kleinstadt, dorf, gebaeude, crossing, Pfad, Weg, Reichsstrasse, Seeweg, Flussweg, Gebirgspass, …
• layer is unreliable. It carries the German display groupings (Metropolen, Großstädte, Städte, Dörfer, Kreuzungen, Pfade, Meerwege, Flusswege, Wüstenpfade, …) but a large tail of features has no layer key at all — e.g. the Gebirgspass / Karawanenroute block around line 600000. This is why the filter is keyed on feature_subtype.
• Property shape is not uniform. A metropole carries wiki_settlement, coat, images[], political.hierarchy[], wiki_url, territory_*; a synthesized crossing carries eight scalars. A path may instead carry transport_domain and allowed_transports[].
• Coordinates max out at ~1022 on both axes, and public/tiles/avesmaps/0/ is a 4×4 grid of 256px tiles = 1024×1024px at z = 2 (= minZoom). The transform is therefore probably map.unproject([x, y], 2), but this is an inference and must be verified — see task 2.
• Image/coat URLs are host-relative to avesmaps.de (/uploads/wappen/…, /uploads/siedlungen/…), not to this app's public/uploads/.

Blocking decision (not resolved)¶

The request was to import features into the map's POI list as editable, persisted entries. That is not viable against the current persistence without a separate decision, for two concrete reasons:

1. projects.server.ts reads and rewrites all of data/projects.json on every single POI mutation. Folding tens of thousands of features into map.pois makes every marker edit a multi-megabyte rewrite.
2. PointOfInterest in app/lib/poi.ts models a single position only. A LineString route has no representation in it at all.

Tasks 1–4 below are deliberately scoped so they do not depend on how this is answered. See the open questions in task 1.

1. Split the Aves dataset into typed shards with a taxonomy manifest¶

Goal: The dataset is available as a set of small files that can be read individually, instead of one document that must be parsed whole.

Scope

• In: A repeatable ingest step that reads data/avesmaps.map-features.json and emits shards plus a manifest describing the taxonomy and per-shard counts.
• In: Shards are split so that a single feature_subtype can be loaded without touching the rest.
• In: source_catalog, feature_sources and in_settlement_places are emitted as their own files, unchanged in meaning.
• Out: Any rendering, filtering or UI. Nothing in app/ changes.
• Out: Coordinate conversion — shards keep source coordinates verbatim (task 2).
• Out: Writing anything into data/projects.json.

Acceptance criteria

• A documented command regenerates every derived file from the source file, and is idempotent — running it twice produces byte-identical output. pnpm ingest:avesmaps runs scripts/ingest-avesmaps-data.ts; it clears and rewrites data/avesmaps/ on every run and emits no timestamps or other non-deterministic fields.
• No emitted shard exceeds 5 MB. Each subtype's features are chunked to a 3.5 MB target and every written file's actual size is checked against a 5 MB hard limit, throwing if exceeded. feature_sources (~8 MB unsplit) also gets this treatment — chunked by dict entry into feature-sources.1/2/3.json — since it hit the limit in practice; source_catalog and in_settlement_places use the same chunking helper but fit in one file each.
• A manifest file lists every feature_type, every feature_subtype under it, a human-readable label, the feature count, and the shard path. See data/avesmaps/manifest.json → featureTypes[].subtypes[].
• Manifest counts sum to the total feature count of the source file, and the feature_type values in the manifest are exactly location, crossing, junction, path, label, powerline — six, not the four assumed before inspection (see the taxonomy note above). The script throws if any feature has a feature_type outside that set, or if the manifest total doesn't match the source feature count.
• Every feature in every shard retains its original id / public_id, so feature_sources lookups still resolve. Features are grouped and written unchanged, no field stripping.
• Whether the source file stays in the repo or is treated as an input artifact is decided and written down. Decided: data/ is already entirely git-ignored (see .gitignore), so both the source file and everything under the new data/avesmaps/ directory are generated/local artifacts, not committed — consistent with public/tiles/ and build/client/tiles/. No .gitignore change was needed.

Affected areas

• data/avesmaps.map-features.json — the input.
• package.json — the ingest command lives alongside build / typecheck.
• app/lib/*.server.ts — not touched by this task; per its own scope, task 1 makes no app/ changes. A server-only reader for the shards is for whichever task first needs to load them (task 2).

Open questions

• Should the derived shards be committed, or generated during pnpm build and git-ignored like build/client/tiles/? Resolved above — data/ is already git-ignored wholesale, so this didn't need a build-step decision.
• Does the answer to the blocking decision above (editable POIs vs. read-only reference layer) change what the shards need to contain? If features become editable, the shards are a seed and need a stable import key; if not, they are the runtime source of truth.

2. Render Aves point features on the map, with the coordinate transform verified¶

Goal: Opening a map backed by the avesmaps tile set shows the dataset's settlements sitting on the correct spots on the tiles.

Scope

• In: Point features only (location, crossing, junction).
• In: Establishing and verifying the transform from dataset coordinates to Leaflet CRS.Simple positions.
• In: Viewport culling — only features intersecting the current view are rendered — and a per-subtype minimum zoom so dense classes such as dorf and crossing do not render when zoomed out.
• Out: Filter UI (task 3), LineStrings (task 4), detail panel (task 5), clustering (task 6).
• Out: Editing, dragging, deleting dataset features.
• Out: Maps backed by kind: "upload" sources or other tile sets.

Acceptance criteria

• Gareth (source coordinates 551.625, 532.969) renders on the Gareth settlement drawn in the tile imagery, at every zoom level from minZoom to maxZoom, with no drift when zooming.
• At least two further settlements far apart on the map are checked the same way and land correctly, ruling out a coincidental fit.
• The transform and the evidence for it are written down; if the 1024×1024 / z=2 inference is wrong, the actual basis is recorded instead.
• Dataset features are visually distinguishable from user-created POIs.
• Panning and zooming across the whole map stays responsive; the number of markers in the DOM stays bounded regardless of how far out you zoom.
• Existing user POIs, POI placement, and POI editing behave exactly as before on the same map.
• A map whose source is an upload, or a tile set other than avesmaps, is unaffected and shows no dataset features.

Affected areas

• app/lib/tile-sets.ts — the avesmaps grid tile set the dataset is aligned to; the association between a tile set and its dataset belongs near here. Added datasetId?: string, set to "avesmaps" on that entry.
• app/lib/avesmaps-dataset.ts — new shared module: the AvesmapsPointFeature type and the per-feature_subtype minimum-zoom table.
• app/lib/avesmaps-dataset.server.ts — new server module reading the task 1 shards for the location, crossing and junction feature types.
• app/components/map.tsx — new AvesmapsFeatureLayer, rendered beside PoiLayer for grid tile sets that carry a datasetId; culls to the current viewport bounds plus a per-subtype minimum zoom.
• app/routes/project-map.tsx — the loader now returns datasetFeatures (empty unless the map's tile set has datasetId === "avesmaps").
• app/lib/poi.ts — untouched; the dataset layer reuses UNKNOWN_POI_TYPE.color for a muted fill so it reads as "reference data", not a user POI.

The transform, verified

map.unproject([x, tileSize * 2**minZoom - y], minZoom), i.e. for the avesmaps tile set (tileSize: 256, minZoom: 2): lng = x / 4, lat = y / 4 - 256.

This is the 1024×1024 / z=2 basis from the note above, plus one correction: dataset y turns out to increase northward (mathematical convention, origin at the south edge), not downward from the top as guessed — so on top of CRS.Simple's own lat = -y/scale negation, y must first be flipped within the 1024px-tall native grid (y' = 1024 - y) before unprojecting.

Evidence (no other source file in this workspace fixes the coordinate space, so this was checked against two independent things):

1. The avesmaps.de source itself. js/config.js in valentin-schwind/avesmaps defines TILE_SIZE = 256, IMG_WIDTH = IMG_HEIGHT = 1024, confirming the 1024×1024 basis. js/app/bootstrap.js seeds its default view from const AVESMAPS_DEFAULT_MAP_CENTER = [497.28, 520.5]; // [lat, lng] = [y, x] — i.e. their own map assigns dataset y to lat and x to lng directly, with their own L.map({ crs: L.CRS.Simple, minZoom: 0, bounds: [[0,0],[1024,1024]] }). Combined with CRS.Simple's built-in lat = -y/scale, matching their direct (unnegated) assignment requires the extra flip captured above.
2. Five named settlements already pinned by hand in this app's own data/projects.json (project aventurien, map Kontinent-Karte), compared against the same names' dataset coordinates:

All five land within ~0.1 map units of the hand-placed pin (consistent with the imprecision of clicking a point on screen), across settlements spread across the whole map — ruling out a coincidental fit for any single one.

Open questions

• Confirmation of the coordinate space is pending an answer from avesmaps.de. Resolved above via the avesmaps.de source and the five-pin cross-check.
• Is y measured downward from the top (SVG convention)? No — resolved above: dataset y increases northward, and the required flip is 1024 - y, not a no-op.

3. Filter Aves features by subtype¶

Goal: A user can choose which classes of dataset feature are on the map.

Scope

• In: A control listing every feature_subtype, grouped under its feature_type header, driven by the manifest from task 1 — not a hardcoded list.
• In: Per-subtype toggles, plus toggling a whole group at once.
• In: The selection survives a page reload for that map.
• Out: Reordering layers, opacity, or per-layer styling.
• Out: Filtering user-created POIs.
• Out: Free-text search.

Acceptance criteria

• Every subtype present in the manifest appears exactly once in the control, under its feature_type group, with its feature count shown.
• Turning a subtype off removes exactly those features and leaves all others on the map.
• Toggling a group toggles every subtype under it.
• Reloading the page restores the previous selection for that map; a different map is unaffected.
• A subtype that is off is never fetched or parsed on the client.
• Toggling a subtype does not reset the map's pan or zoom.
• The control is reachable in view mode, not only edit mode.

Affected areas

• app/lib/avesmaps-dataset.ts — added AvesmapsManifest/AvesmapsManifestFeatureType/ AvesmapsManifestSubtype and getAvesmapsLayerKey(featureType, subtype). The key is "<feature_type>:<feature_subtype>", not the bare subtype: the manifest's junction feature_type reuses the crossing feature_subtype for a different (and much larger) set of features, so the two need to be toggled and fetched independently.
• app/lib/avesmaps-dataset.server.ts — added getAvesmapsManifest() (a point-feature-only, shards-stripped view for the client) and getAvesmapsPointFeaturesByLayerKeys(layerKeys). The previously-exported getAvesmapsPointFeatures() (task 2) is now a private loadAllAvesmapsPointFeatures() — the full parsed set stays cached server-side, but only the requested layers are ever put on the wire.
• app/routes/avesmaps-features.tsx — new resource route (GET /avesmaps/features?layers=<key>,<key>,…) the client fetches from whenever the enabled layer set changes. Registered in app/routes.ts.
• app/routes/project-map.tsx — the loader now returns datasetManifest instead of the full feature list. The component owns the enabled-layers state (useAvesmapsLayerSelection) and re-fetcher.load()s /avesmaps/features when it changes; datasetFetcher.data.features (not a loader field) is what reaches <Map>. The layer control renders as a floating panel over the map, positioned outside the mode === "edit" sidebar so it shows in view mode too.
• app/components/avesmaps-layer-control.tsx — new: useAvesmapsLayerSelection (the localStorage-backed on/off state, keyed avesmaps.layers.<mapId>) and AvesmapsLayerControl (the grouped-checkbox panel UI).
• app/components/map.tsx — unchanged by this task; AvesmapsFeatureLayer already just renders whatever features prop it's given (task 2), so server-side layer filtering "just works" underneath it.

Decisions

• Persistence is localStorage, keyed avesmaps.layers.<mapId> (resolves the open question below): sufficient for now, doesn't grow data/projects.json, and doesn't need to be shareable.
• The control lists location, crossing and junction only (task 2's scope) — not path, label or powerline, which nothing renders yet. Task 4 extends getAvesmapsManifest()'s filter to include path once route rendering exists, which is what "join the same filter taxonomy" refers to.
• "Off" is stored as the disabled set, not the enabled set, so a manifest that gains a new subtype later defaults it to visible instead of hidden.

Open questions

• Where does the selection persist — URL search params, localStorage, or the map record in data/projects.json? Resolved above: localStorage.

4. Render Aves route features¶

Goal: Roads, paths, river routes and sea routes are drawn on the map.

Scope

• In: LineString features (feature_type: "path") rendered as polylines.
• In: Route subtypes join the same filter taxonomy from task 3.
• In: The same viewport culling and per-subtype zoom thresholds as task 2.
• Out: Route editing, drawing new routes, snapping to crossings/junctions.
• Out: Pathfinding, and any use of transport_domain / allowed_transports beyond storing them.
• Out: Connecting routes to the crossing / junction points as a graph.

Acceptance criteria

• Route features render as lines aligned to the roads and rivers in the tile imagery, using the same verified transform as task 2.
• Route subtypes (Reichsstrasse, Seeweg, Flussweg, Gebirgspass, Weg, Pfad, …) are individually toggleable via the task 3 control.
• Route subtypes are visually distinguishable from one another.
• Panning across a dense region with all route subtypes enabled stays responsive.
• Clicking a route does not interfere with POI placement in edit mode.

Affected areas

• app/lib/avesmaps-dataset.ts — added AvesmapsRouteFeature, extended the per-subtype minimum-zoom table with the 8 path subtypes (no key collision with the point subtypes), and added getAvesmapsPathColor(subtype).
• app/lib/avesmaps-dataset.server.ts — added loadAllAvesmapsRouteFeatures() / getAvesmapsRouteFeaturesByLayerKeys(), reading the task 1 path shards and keeping their LineString coordinates verbatim as points. getAvesmapsManifest() now also includes the path feature type (was point-only through task 3) — the decision task 3 flagged as pending this task.
• app/routes/avesmaps-features.tsx — the resource route now returns routes alongside features, filtered by the same layers param.
• app/components/map.tsx — new AvesmapsRouteLayer: same transform, viewport culling and per-subtype minimum zoom as AvesmapsFeatureLayer, but culls each route by whether its bounding box overlaps the viewport (not a single point), and renders <Polyline interactive={false}> so a route never intercepts a click meant for POI placement or deselection.
• app/routes/project-map.tsx — passes datasetFetcher.data.routes through to <Map> as datasetRoutes.
• app/lib/poi.ts — untouched; line geometry has no representation there and didn't need one — AvesmapsRouteFeature (a dataset-only, read-only type) lives in avesmaps-dataset.ts instead.

Open questions

• None.

5. Aves feature detail panel¶

Goal: Clicking a dataset feature shows what the source knows about it.

Scope

• In: A panel showing name, class, the wiki_settlement fields, the political.hierarchy chain, the coat of arms, the images[] gallery, and the source references resolved through feature_sources → source_catalog.
• In: Handling the non-uniform property shape — a synthesized crossing has almost none of these fields.
• In: Deciding and implementing how the host-relative /uploads/… image URLs are resolved.
• Out: Editing any of this data.
• Out: in_settlement_places (no geometry; needs its own decision).

Acceptance criteria

• Clicking Gareth shows its population, ruler, region, description, wiki link, coat of arms, image gallery, and its four-level political hierarchy.
• Clicking a synthesized crossing shows a panel with no empty sections and no broken image placeholders.
• Source references render with the source label and page numbers, and link out where source_catalog provides a URL.
• External URLs in the data cannot inject markup or script into the panel, and non-http(s) URLs are not rendered as links.
• Images do not block the panel from rendering, and a missing image does not break the layout.
• The panel does not conflict with the existing POI detail surface in project-map.tsx.

Affected areas

• app/lib/avesmaps-dataset.ts — added AvesmapsFeatureDetail, AvesmapsPoliticalEntity, AvesmapsFeatureSourceReference, resolveAvesmapsAssetUrl() and isSafeHttpUrl().
• app/lib/avesmaps-dataset.server.ts — loadAllAvesmapsPointFeatures() (task 2) now also fills a Map<id, rawFeature> cache, so the full non-uniform properties are available without a second file scan. Added getAvesmapsFeatureDetail(id), plus private readers for feature-sources.*.json / source-catalog.json (paths come from the manifest's auxiliary block, not hardcoded) and a getAvesmapsFeatureSources helper that looks features up as settlement:<id> — crossings/junctions resolve to [], which the panel renders as no "Sources" section.
• app/routes/avesmaps-feature-detail.tsx — new resource route (GET /avesmaps/features/:id), 404s for an unknown id. Registered in app/routes.ts.
• app/components/avesmaps-feature-detail-panel.tsx — new AvesmapsFeatureDetailPanel; every section (settlement fields, political hierarchy, images, sources) is conditionally rendered only when it has data. A SafeImage wrapper hides itself on onError instead of showing a broken image icon.
• app/components/map.tsx — AvesmapsFeatureLayer's CircleMarkers now take an onSelectRequest, mirroring how PoiMarker already does it.
• app/routes/project-map.tsx — new selectedDatasetFeatureId state and a useFetcher that loads the detail route when it changes. Selecting a dataset feature clears the selected POI and vice versa (mutual exclusion, not two panels stacking), and only does anything in view mode — dataset features aren't editable, so there's nothing to show them for in edit mode.

Decisions

• Image/coat URLs are hotlinked directly as https://avesmaps.de<path> (resolveAvesmapsAssetUrl), not mirrored into this app's public/uploads/. This resolves the open question below: mirroring a third party's asset library is out of scope for a read-only reference layer, and the acceptance criteria already require handling a missing/blocked image gracefully, which SafeImage's onError handles regardless of why the image failed to load.
• XSS/link-injection defense is two-layered: isSafeHttpUrl rejects any URL whose scheme isn't http/https before it's ever put in the detail JSON, and the panel only ever puts a URL in an href attribute — it never uses dangerouslySetInnerHTML, so React's normal text escaping covers names, descriptions and source notes (which can contain wiki-style [[...]] markup) automatically.

Open questions

• Are the avesmaps.de image URLs publicly fetchable and are we permitted to hotlink them, or must they be mirrored into public/uploads/? Resolved above: hotlinked, with graceful failure if that turns out to be wrong.
• What is in_settlement_places for — a future submap feature, or a list to show inside a settlement's detail panel? Still open; out of this task's scope.

6. Cluster dense point features¶

Goal: Zoomed-out views of dense regions stay readable and fast.

Scope

• In: Clustering for point features, replacing or complementing the per-subtype zoom thresholds from task 2.
• Out: Clustering routes.
• Out: Clustering user-created POIs.

Acceptance criteria

• With every point subtype enabled, the map is interactive at all zoom levels and marker count in the DOM stays bounded.
• A cluster shows how many features it contains and expands on click.
• Clusters respect the active subtype filter — a disabled subtype is not counted.
• Any new dependency is added with pnpm and is client-only, consistent with how Leaflet is lazy-loaded.

Affected areas

• app/components/map.tsx — the dataset point layer.
• package.json — no clustering dependency exists today.

Open questions

• None.

Recommended order¶

Task 1 first. Nothing else can be built while the dataset is a single 1.2M-line document — task 2 needs shards to load, task 3 needs the manifest to build its control from, and task 5 needs feature_sources addressable on its own. It is also the only task that changes no application code, so it can land without touching the map.

Then 2 (proves the transform — everything visual is wrong until this is right), 3, 4, 5, 6.