Dev#6
Open
mretallack wants to merge 14 commits into
Open
Conversation
…any) UK TMC (Inrix) is proprietary and unavailable. Revised plan uses publicly available TMC tables from France, Germany, etc. 6 new subtasks for downloading, parsing, and generating TMC files from open data sources.
Complete map update needs more than just FBL: - 20.1-20.3: HNR routing (A/B major/minor from OSM highway tags) - 20.4-20.6: POI (from OSM amenity/shop/tourism tags) - 20.7-20.9: Speed cameras (from OSM enforcement data) - 20.10: TMC → see Task 16 (public location code lists) All formats already decoded. Task is to build generators. 157 completed, 46 open.
Replaced messy 952-line file with clean structure: - Completed section: summary of 157 done tasks (not listed individually) - Open tasks grouped by category: - FBL Enrichment (7 tasks) - Supporting files: HNR/POI/SPC (9 tasks) - TMC from public data (6 tasks) - Content download (6 tasks) - Head unit testing (3 tasks) - Low priority HNR DLL (4 tasks) - Key files reference table 35 open tasks (removed duplicates and stale cross-refs).
7 high-level tasks → 30 concrete subtasks: - Junction connectivity: 6 subtasks (detect shared nodes → encode → verify) - Road names: 6 subtasks (extract → index → encode section 15) - Shape points: 5 subtasks (identify junctions vs intermediate → split ways) - Section boundaries: 4 subtasks (analyse real FBL → add records) - Road attributes: 6 subtasks (speed/oneway/surface → encode) - Curve data section 1: 5 subtasks (bitstream encoding) 60 open tasks total.
19.4a: Find intersections in OSM data (nodes shared by 2+ ways) Monaco: 4,808 junctions (3,902 two-way, 756 three-way, 85 four-way) 19.4b: Assign sequential junction IDs (1-based, ordered by appearance) Also: split OSM ways at junctions into proper road segments Monaco: 3,464 ways → 6,389 segments after splitting 81% of segments have both ends at junctions Analysis of real FBL junction records: - 20 junction records in Monaco section 4 - IDs 1-4 (sequential counter, resets per line) - Graph builder writes opcode 0x85 with 2-byte junction ID - Junctions mark routing decision points, not every intersection Next: 19.4c-19.4f (encode + integrate + verify + test)
Junction connectivity is IMPLICIT via shared endpoint coordinates. The graph builder infers connections from matching coords (verified). Explicit 0x80080000 records are optional (require complex context setup). Changes to osm_to_fbl.py: - read_osm_xml() now detects junction nodes (shared by 2+ ways) - Ways are split at junctions into proper road segments - Monaco: 3,464 ways → 6,389 segments after splitting - 81% of segments have both ends at junctions - All 3 sections decode correctly, graph builder accepts output 318 tests passing.
FINDING: Street names are NOT stored as readable text in FBL files. Searched entire Monaco FBL (53KB) for known street names in ASCII, UTF-8, UTF-16, byte*2 — nothing found. Section 15 is a binary lookup table (260 bytes for Vatican, 540 for Monaco) with high-entropy data. The varint decoder produces numeric records, not character codes. 0x80070000 (name) records: ZERO found in any section of Monaco FBL. Names are compiled into the proprietary binary format by the NNG map compiler's rendering engine, not the graph builder. Status: 19.5c BLOCKED on name encoding reverse engineering. Roads will render without name labels — routing still works.
Key fix: ) (0x29) terminates processing when group_depth==0 (matches DLL behavior: return 0x7A error). Line match jumped from 28-34% to 47-51%. Also added (?...) group handler for junction records. Monaco: 37/72 lines exact match (51%), 4027/6327 records (63%). Remaining errors: 14x junction ID mismatch, 8x data, 6x PY long. 318 tests passing.
Fixes: - \0-\7 octal escapes: \0 = NUL (0), \012 = 10, etc. - (?...) group handler generates 0x80080000 junction records Monaco: 38/72 lines (52%), 4028/6327 records (63%) All files: 47-51% line match, 63-74% record accuracy Remaining gap analysis: - UC consumes 90% of varints via pattern matching - PY consumes 57% — the 33% gap is in # hash and \ escape handling - Each line has ~420 varints, UC outputs ~43, PY outputs ~238 - The pattern engine's consumption rules are the core blocker 318 tests passing.
Escape handling: positive table values output as data, negative values (road class) output raw escaped value, octal escapes (\0-\7), \Q/\E for quote mode. ) terminates when group_depth==0. (?...) generates junction records. Best accuracy: Vatican 74%, SanMarino 70%, Andorra 68%. 318 tests passing.
CRITICAL FIX: # hash delimiter is LF (0x0A), not NUL (0x00). Since we split by LF, there are no LF bytes in the line data. The hash scan reaches end of input and consumes EVERYTHING. This was the #1 remaining issue. param_4[0x24] = 1 (delimiter length), delimiter byte = 0x0A (LF) Set by the caller based on context[7] high word = 2 (LF mode). Monaco: 44/72 lines (61%), 4027/6327 records (63%) Gibraltar: 48/73 lines (65%) Vatican: 5/12 lines (41%) 318 tests passing.
The # hash consuming to end of line means road class escapes inside hash-consumed regions are no longer output as records. Test now checks for record count > 100 instead of specific types.
Fixes: - [ handler: only consume to ] if ] found within 100 bytes Otherwise treat [ as data (prevents consuming past #) - # hash consumes to end of line (LF delimiter, not NUL) - ) terminates when group_depth==0 Vatican: 77%, SanMarino: 74%, Andorra: 72%, Gibraltar: 71% Monaco: 65%, Malta: 70%, Liechtenstein: 68% 318 tests passing.
Line match: 59-67% (was 58-65%) Record accuracy: 65-77% Gibraltar: 67% lines, Vatican: 77% records 318 tests passing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.