Modelling & mapping suite
Stage 2 of the pipeline — the missing middle. Modelling turns an approved requirements-catalog.yaml (the discovery output) into the source-to-target mapping files the build suite consumes. On a traditional engagement this bridge between requirements and build is done by hand; here three agents chain together — SourceProfiler → Modeller → SourceToTargetMapper — passing the profile report, then the logical model, then the mapping files down the line. The newest of the three suites: its first scored runs landed on 12 July (see Runs).
The three agents
| Agent | Role |
|---|---|
SourceProfiler | reads and profiles the real source schemas (strictly read-only) to ground the catalogue's source and data-quality sections |
Modeller | turns approved requirements into entities, relationships, conformed dimensions and facts, keys, and SCD intent — the conceptual and logical model |
SourceToTargetMapper | combines the logical model with the profiled source schemas to emit the gold mapping files build consumes |
Design principles
- Source access is read-only by construction — a connector protocol with no write path: an in-memory SQLite connector for synthetic sources in tests, and a Fabric connector for real schemas and data.
- Profiling is deterministic code, not model output — null rates, cardinality, candidate keys, PII flags and referential gaps are computed, so the model reasons over facts rather than inventing them.
- The logical model schema mirrors the real catalogue — the same
logical_modelblock the Meridian worked example carries. - Fully testable offline — sources are mocked with synthetic SQLite shaped like the real systems, so the whole suite runs without a live Fabric connection.
Current limitations
The mapper currently emits gold-layer dimension and fact mappings — generating bronze-ingest and silver-conform mappings is the natural next step — and profiling-derived data-quality findings are not yet first-class typed citations. Both are tracked and honest: the run scorecards under Runs show exactly where the suite stands today.