Location intelligence: data collection
← All revisions
Track-B documentation wave: how-to category — schema-conformant restructure of 2 guides + heavy R1/DP6 redaction of collect-location-intelligence-data (internal vendor rosters/production counts removed)
@@ -2,383 +2,66 @@ schema: foundry-doc-v1 title: "Location intelligence: data collection" slug: collect-location-intelligence-data short_description: "Ingesting VWH and PKS chain and infrastructure data into the location-intelligence pipeline from OpenStreetMap — runbook for re-ingesting or extending to new chains and countries." short_description: "How new retail and infrastructure chains get added to the location-intelligence pipeline's taxonomy, and how the pipeline ingests their location data from OpenStreetMap." category: how-to content_type: how-to type: how-to quality: complete status: active audience: vendor-internal bcsc_class: no-disclosure-implication language_protocol: GUIDE-OPERATIONS language_protocol: RUNBOOK last_edited: 2026-07-11 editor: editorial editor: pointsav-engineering paired_with: collect-location-intelligence-data.es.md research_trail: sources: [pointsav-monorepo app-orchestration-gis taxonomy.py, config.py, ingest-osm.py] verification_method: "re-verified 2026-09-05 against app-orchestration-gis: build-vwh-clusters.py does not exist under that name; test-cluster-archetypes.py, build-clusters.py, and export-clusters-ols.py do" --- > **Status (2026-06-11):** All seven steps below are complete. VWH production build: > 6,368 clusters (T1=852/T2=1,327/T3=4,189). PKS production build: 6,953 clusters > (T1=691/T2=2,658/T3=3,604). This guide is retained as a reference runbook for > re-ingesting or extending to new chains and countries. Initial pipeline setup for ingesting VWH (Vertical Warehouse) and PKS (Parking Structures) chain and infrastructure data. Steps are recorded in execution order. The location-intelligence pipeline classifies retail and infrastructure locations into two archetypes — Vertical Warehouse (VWH) and Parking Structures (PKS) — by matching OpenStreetMap records against a maintained chain taxonomy. This runbook covers extending that taxonomy to a new chain, country, or infrastructure category, and re-running the ingest and clustering steps that follow. Working directory for all commands: `app-orchestration-gis/` (inside the GIS monorepo clone). ## Prerequisites - Overpass API access (queries run via `ingest-osm.py`; no API key required) - Overpass API access (queries run through the pipeline's OSM ingest script; no API key required) - Python 3.11+ with pipeline dependencies installed - `TOTEBOX_DATA_PATH` pointing to the active deployment data directory Verify the pipeline is clean: ```bash python3 -c "from taxonomy import CATEGORIES, BRAND_FILL; print('taxonomy OK')" python3 -c "from config import TOTEBOX_DATA_PATH; print('config OK')" ``` ## Step 1 — Run existing YAML ingests (VWH auto-parts + paint) Five chain YAMLs were scaffolded on 2026-06-01. Run the ingest to download OSM records: ```bash python3 ingest-osm.py --chain \ autozone-us \ oreilley-auto-us \ napa-us \ sherwin-williams-us \ halfords-uk ``` Expected output files in `$TOTEBOX_DATA_PATH/service-fs/service-business/`: - `autozone-us.jsonl` — expect 5,000–7,000 records - `oreilley-auto-us.jsonl` — expect 5,000–7,000 records - `napa-us.jsonl` — expect 3,000–6,000 records (franchise network; OSM partial) - `sherwin-williams-us.jsonl` — expect 3,000–5,000 records - `halfords-uk.jsonl` — expect 300–450 records Verify: ```bash for f in autozone-us oreilley-auto-us napa-us sherwin-williams-us halfords-uk; do echo "$f: $(wc -l < $TOTEBOX_DATA_PATH/service-fs/service-business/$f.jsonl) records" done ``` If a chain returns 0 records, check if Wikidata tag coverage is sparse in OSM and add `name_query:` fallback to the YAML. ## Step 2 — Add Würth (biggest EU MRO gap) Würth is the single highest-value chain not yet in the taxonomy. ~1,500 EU branches in industrial parks across DE/FR/IT/PL/AT/NL and beyond. OSM has `brand:wikidata=Q183759` applied to many records. **2a. Create YAML:** ```bash cat > $TOTEBOX_DATA_PATH/service-fs/service-business/wurth-de.yaml << 'EOF' schema: service-business-chain-v1 chain_id: wurth-de country: Germany country_code: DE region: europe-central category: Industrial MRO supply category_slug: mro-industrial naics_code: "423840" top_category: Industrial and Personal Service Paper and Related Products sub_category: Industrial MRO Supply overture_taxonomy: - industrial - wholesale brand_family: MROIndustrial retailer: Würth canonical_name: "Adolf Würth GmbH & Co. KG" parent_company: "Würth Group" website: wuerth.de wikidata_id: Q183759 osm_overpass_tag: brand:wikidata=Q183759 store_count_approx: 1500 locations_file: locations/wurth-de.jsonl locations_status: pending last_updated: 2026-06-01 notes: "VWH archetype signal. MRO distributor; branches in industrial parks across EU. Multi-country — set country_code per-ISO ingest or use multi_country: true." EOF ``` **2b. Add `mro_industrial` category to taxonomy.py:** In `taxonomy.py`, inside the CATEGORIES dict, after the `paint` block: ```python "mro_industrial": { "label": "Industrial MRO Supply", "naics": "423840", "description": "Maintenance, repair, and operations distributor — VWH signal. Never gates tier.", }, "flooring": { "label": "Flooring & Tile Supply", "naics": "442210", "description": "Contractor-facing flooring/tile warehouse — VWH signal. Never gates tier.", }, "tool_rental": { "label": "Tool & Equipment Rental", "naics": "532412", "description": "Equipment rental branch — VWH signal; deliberate hardware co-location. Never gates tier.", }, "lumber": { "label": "Lumber & Building Materials", "naics": "444190", "description": "Lumber yard or building materials dealer — VWH signal. Never gates tier.", }, "car_rental": { "label": "Car Rental", "naics": "532111", "description": "Car rental branch — PKS signal; defines transit node commercial zone. Never gates tier.", }, ``` None of these go into `_RETAIL_CATS` — they never affect T1/T2/T3 tier logic. **2c. Add BRAND_FILL entries for all new categories:** In the BRAND_FILL dict, after the existing `paint` block: ```python "mro_industrial": { "DE": ["wurth-de"], "FR": ["wurth-de"], # Würth is multi-country; chain_id shared "IT": ["wurth-de"], "ES": ["wurth-de"], "PL": ["wurth-de"], "AT": ["wurth-de"], "NL": ["wurth-de"], "GB": ["wurth-de"], "US": ["fastenal-us", "grainger-us"], "CA": [], "MX": [], "SE": ["wurth-de"], "DK": ["wurth-de"], "NO": ["wurth-de"], "FI": ["wurth-de"], "IS": [], "GR": [], "PT": [], }, "flooring": { "US": ["floor-decor-us"], "GB": ["topps-tiles-uk"], "CA": [], "MX": [], "FR": [], "DE": [], "ES": [], "IT": [], "GR": [], "PL": [], "AT": [], "NL": [], "PT": [], "SE": [], "DK": [], "NO": [], "FI": [], "IS": [], }, "tool_rental": { "US": ["united-rentals-us", "sunbelt-rentals-us"], "CA": ["united-rentals-us"], "FR": ["loxam-fr", "kiloutou-fr"], "MX": [], "GB": [], "DE": [], "ES": [], "IT": [], "GR": [], "PL": [], "AT": [], "NL": [], "PT": [], "SE": [], "DK": [], "NO": [], "FI": [], "IS": [], }, "lumber": { "US": ["84-lumber-us", "builders-firstsource-us"], "CA": ["kent-building-supplies-ca"], "MX": [], "GB": [], "FR": [], "DE": [], "ES": [], "IT": [], "GR": [], "PL": [], "AT": [], "NL": [], "PT": [], "SE": [], "DK": [], "NO": [], "FI": [], "IS": [], }, "car_rental": { "US": ["enterprise-us", "hertz-us", "avis-us"], "CA": ["enterprise-us", "hertz-us"], "MX": [], "DE": ["sixt-de"], "FR": ["europcar-fr", "sixt-de"], "GB": ["europcar-fr", "sixt-de"], "ES": ["europcar-fr"], "IT": ["europcar-fr"], "NL": ["europcar-fr"], "GR": [], "PL": [], "AT": ["sixt-de"], "PT": [], "SE": ["sixt-de"], "DK": ["sixt-de"], "NO": [], "FI": [], "IS": [], }, ``` **2d. Run Würth ingest:** ```bash python3 ingest-osm.py --chain wurth-de ``` Expect 800–1,500 records. Würth branches are often tagged `shop=trade` or `shop=wholesale` in OSM — the `brand:wikidata` tag is the reliable signal. ## Step 3 — Add Tier A retail/rental VWH chains Create YAMLs following the schema below, then run ingest. **YAML schema reference:** ```yaml schema: service-business-chain-v1 chain_id: floor-decor-us # matches BRAND_FILL key country: United States country_code: US region: north-america category: Flooring supply # display only category_slug: flooring-supply naics_code: "442210" top_category: Floor Covering Stores sub_category: Floor Covering Stores brand_family: Flooring # matches category retailer: Floor & Decor canonical_name: "Floor & Decor Holdings, Inc." parent_company: "Floor & Decor Holdings, Inc. (public; NYSE: FND)" website: flooranddecor.com wikidata_id: Q22350998 osm_overpass_tag: brand:wikidata=Q22350998 store_count_approx: 240 locations_file: locations/floor-decor-us.jsonl locations_status: pending last_updated: 2026-06-01 notes: "VWH Tier A. Warehouse-format contractor flooring; same footprint as Home Depot." ``` **Chains to create (copy and adapt schema above):** | chain_id | retailer | wikidata_id | country_code | approx_count | |---|---|---|---|---| | `floor-decor-us` | Floor & Decor | Q22350998 | US | 240 | | `topps-tiles-uk` | Topps Tiles | Q7825827 | GB | 300 | | `united-rentals-us` | United Rentals | Q7889284 | US | 1,400 | | `sunbelt-rentals-us` | Sunbelt Rentals | Q7645154 | US | 1,100 | | `loxam-fr` | Loxam | Q6692217 | FR | 1,100 | | `kiloutou-fr` | Kiloutou | Q3197034 | FR | 600 | | `fastenal-us` | Fastenal | Q1394323 | US | 3,400 | | `grainger-us` | Grainger | Q904633 | US | 600 | | `hilti-ch` | Hilti | Q565285 | CH | 600 | | `84-lumber-us` | 84 Lumber | Q4641204 | US | 310 | | `builders-firstsource-us` | Builders FirstSource | Q4934620 | US | 570 | | `kent-building-supplies-ca` | Kent Building Supplies | Q6383907 | CA | 45 | Run all at once after YAMLs are created: ```bash python3 ingest-osm.py --chain \ floor-decor-us topps-tiles-uk \ united-rentals-us sunbelt-rentals-us loxam-fr kiloutou-fr \ fastenal-us grainger-us hilti-ch \ 84-lumber-us builders-firstsource-us kent-building-supplies-ca ``` Note: `fastenal-us` and `84-lumber-us` may return 0 records via wikidata — add `name_query: "Fastenal"` / `name_query: "84 Lumber"` fallback if needed. ## Step 4 — Write `ingest-osm-airports.py` (commercial airport filter) The existing Overture airport data (20,841 records) includes private airstrips, heliports, and military fields. This script replaces it with OSM-sourced commercial airports only. **Pattern:** copy `ingest-osm-civic.py`, change the Overpass query to: ```python QUERY = """ [out:json][timeout:60]; ( node["aeroway"="aerodrome"] ["aerodrome:type"~"^(public|international|regional|domestic)$"] ({bbox}); node["aeroway"="aerodrome"]["iata"~"."]({bbox}); way["aeroway"="aerodrome"] ["aerodrome:type"~"^(public|international|regional|domestic)$"] ({bbox}); way["aeroway"="aerodrome"]["iata"~"."]({bbox}); ); out center; """ ``` Output: `$TOTEBOX_DATA_PATH/service-places/cleansed-civic-airports.jsonl` Schema: same as `cleansed-places.jsonl` with `category_id: "airport"`, `naics_code: "488119"`. Enrich with IATA code from the OSM `iata` tag where present. Run per country using `COUNTRY_BBOX` from `config.py` (defined for all 17 display ISOs). Expected output: ~5,000–8,000 records (vs. 20,841 from Overture). ## Step 5 — Write `ingest-osm-railway.py` (intercity stations) - The active deployment's data directory path configured Output: `$TOTEBOX_DATA_PATH/service-places/cleansed-civic-railway.jsonl` ## Purpose **Overpass query per country:** ```python QUERY = """ [out:json][timeout:60]; ( node["railway"="station"] ["station"!="subway"] ["station"!="light_rail"] ["station"!="tram"] ["station"!="monorail"] ({bbox}); way["railway"="station"] ["station"!="subway"] ["station"!="light_rail"] ["station"!="tram"] ["station"!="monorail"] ({bbox}); ); out center; """ ``` Add a new chain, country, or infrastructure category to the location-intelligence taxonomy, ingest its OpenStreetMap records, and re-run the clustering build so the new data is reflected in the VWH/PKS archetype outputs. **Post-processing:** filter to intercity operators only using the `operator=` tag: ## Procedure ```python INTERCITY_OPERATORS = { "US": ["Amtrak"], "CA": ["VIA Rail Canada", "Via Rail"], "FR": ["SNCF", "Société Nationale des Chemins de fer Français"], "DE": ["Deutsche Bahn", "DB", "DB Regio"], "ES": ["Renfe", "Renfe Operadora"], "IT": ["Trenitalia", "Italo", "Ferrovie dello Stato"], "AT": ["ÖBB", "Österreichische Bundesbahnen"], "NL": ["NS", "Nederlandse Spoorwegen"], "SE": ["SJ", "Norrtåg"], "DK": ["DSB", "Danske Statsbaner"], "NO": ["Vy", "NSB"], "FI": ["VR", "VR Group"], "PT": ["CP", "Comboios de Portugal"], "PL": ["PKP Intercity", "RegioJet"], "GB": None, # All Network Rail TOCs qualify — no filtering needed "MX": None, # No passenger rail — skip "IS": None, # No passenger rail — skip "GR": ["TrainOSE"], } ``` 1. **Verify the pipeline is clean** by importing the taxonomy and config modules and confirming both load without error. For countries where `operator=` filtering applies: only keep stations where the `operator` tag matches (case-insensitive substring). For GB: keep all. For MX and IS: skip entirely. 2. **Add a new chain to the taxonomy.** Each chain is declared in its own YAML record: a chain identifier, country and region, a category (mapped to a NAICS code), the retailer's canonical legal name and parent company, a public identifier used to match OpenStreetMap records (typically a Wikidata QID via the `brand:wikidata` OSM tag), and an approximate store count used only as a sanity check on ingest results — never as a tier-qualification input. A chain that spans multiple countries is flagged as such rather than duplicated per country. Schema: `category_id: "railway_station"`, `naics_code: "482111"`. 3. **Register the chain's category in the taxonomy module**, if it introduces a category not already present. Each category carries a label, a NAICS code, and a note on which archetype signal it contributes (VWH or PKS) — none of these categories gate archetype-tier logic on their own; tier assignment is a separate downstream step. ## Step 6 — Add PKS car rental chain YAMLs 4. **Run the OSM ingest for the new chain(s).** The ingest script queries Overpass for the chain's tagged locations and writes the results to the pipeline's business-data directory. If a chain returns zero records, check whether Wikidata tag coverage is sparse in OSM for that chain and add a name-based fallback query to its YAML record. Create five YAMLs following the same schema as Step 3: 5. **For a new infrastructure category** (for example, commercial airports or intercity rail stations, as opposed to a retail chain), write a dedicated ingest script following the existing infrastructure-ingest pattern: an Overpass query scoped to the relevant `aeroway`/`railway` tags, filtered to exclude out-of-scope subtypes (private airstrips and military fields for airports; subway, light rail, and tram for railway stations), with per-country operator or IATA-code enrichment applied where the source tagging supports it. | chain_id | retailer | wikidata_id | notes | |---|---|---|---| | `enterprise-us` | Enterprise Rent-A-Car | Q2283517 | Multi-country NA | | `hertz-us` | Hertz | Q379425 | Multi-country NA + EU | | `avis-us` | Avis | Q849144 | Multi-country | | `sixt-de` | Sixt | Q704156 | EU primary; some NA | | `europcar-fr` | Europcar | Q466704 | EU primary | 6. **Re-run the clustering build** once all new chain and infrastructure ingests are in place, using the pipeline's DBSCAN cluster-build scripts for each archetype. Copy the outputs to the active deployment's data directory and confirm the new cluster counts are at or above the prior production baseline — a drop below the baseline indicates an ingest or taxonomy regression, not an expected result of adding data. `naics_code: "532111"` (Passenger Car Rental). `brand_family: CarRental`. ## Expected outcome These all need `multi_country: true` in the YAML since they operate across multiple ISOs in the display list. The new chain, country, or infrastructure category appears in the archetype's OSM ingest output with a plausible record count for its real-world scale, and the subsequent clustering build produces cluster counts at or above the pipeline's prior production baseline. Run ingest: ```bash python3 ingest-osm.py --chain enterprise-us hertz-us avis-us sixt-de europcar-fr ``` ## Verification Expect high counts due to airport ubiquity: `enterprise-us` ~4,000–8,000 records; `sixt-de` ~500–800 EU records. Count the records in each newly ingested chain's output file and compare against the chain's real-world scale as a sanity check, not an exact target. Confirm the clustering build's output feature counts for VWH and PKS are at or above their last known production baseline before treating the run as complete. ## Step 7 — Production calibration and deployment (complete 2026-06-11) ## Rollback `test-cluster-archetypes.py` was superseded by dedicated DBSCAN build scripts: - `build-vwh-clusters.py` → `archetype-vwh.geojson` (6,368 features) (Correction, 2026-08-02, verified against canonical `origin/main`: this script does not exist in the repo — confirmed via `git cat-file -e`. It's also referenced by the real `nightly-rebuild.sh`, so this is a pre-existing pipeline gap this guide faithfully reproduces rather than a wiki-invented claim. `test-cluster-archetypes.py` itself still exists despite being called "superseded" here. Flagged, not resolved.) - `build-pks-clusters.py` → `archetype-pks.geojson` (6,953 features) A new chain YAML or taxonomy-category entry can be removed and the ingest re-run without side effects — the ingest step is idempotent per chain. The clustering build can be re-run at any time from its existing inputs; it does not need to be reverted, only re-run once the taxonomy correction is in place. To re-run after adding new chains: ```bash python3 build-vwh-clusters.py # VWH — outputs work/archetype-vwh.geojson python3 build-pks-clusters.py # PKS — outputs work/archetype-pks.geojson ``` ## Next steps Copy outputs to the active deployment's data directory, then verify cluster counts match the production baseline (VWH ≥ 6,368 features; PKS ≥ 6,953 features). - [[connect-osm-data-pipeline]] — the generic single-chain ingest path this runbook extends ## See also