Skip to content

Changelog

Notable user-facing changes are recorded here. Keep edition-specific behavior separate, note breaking changes and migrations, and link validation evidence when available. Do not describe a warehouse build as passing unless it was actually run.

[Unreleased]

  • Added a source-available evaluation licence (LICENSE), TheLook and synthetic-data attribution (NOTICE), and a best-effort, no-SLA support statement (SUPPORT.md), and a private security reporting policy (SECURITY.md). The rationale and the way to switch to Apache-2.0 or MIT are in decision 0001.
  • Added a private machine-readable migration acceptance report that summarizes existing run and Bronze reconciliation evidence without connecting to a destination. It excludes credentials, row values, and source/target identities, and always reports cutover_ready: false.
  • Release archive validation now rejects ZIP members outside the edition root before extraction. Edition archives include a Terraform lifecycle wrapper that supports both repository and extracted-archive layouts, with matching operator commands.
  • Added regression coverage for process interruption immediately after extraction; follow-up replacement is refused while the existing local DuckDB target remains unchanged.
  • Removed CSV content hashes from new migration manifests to avoid a second full-file read and deterministic equality leakage; manifests now record pre-extraction file names and sizes, clearly marked as descriptive metadata; neither field proves consumed bytes, and SQL schema/table names do not distinguish databases with identical names. The evidence index preserves the legacy identity-scope label, validates legacy hashes but omits them from its private evidence-index output and validates CSV basenames against entity names. Reconciliation now caps manifest reads at 10 MiB and creates reports exclusively without following symlinks. Binary SQL values fail closed when mapped to strings instead of becoming Python byte representations. Mapping files are opened without following final-component symlinks and parsed from a bounded 1 MiB read. Run manifests are created exclusively and status updates use owner-only temporary files with fsync and atomic replacement; an interrupted update preserves the prior complete manifest. Mapping configuration errors are safely actionable while malformed JSON details and content stay suppressed; opening device paths cannot set a controlling terminal. BigQuery Silver models now use canonical-field tie-breaks for stable duplicate-key selection; this is deterministic ordering, not proof of source-version semantics. State-retention previews use newest descendant timestamps, iterative symlink-safe traversal, deterministic path ordering, and unknown totals when a subtree cannot be inspected. The bounded value profile labels Python cast semantics separately from unverified warehouse castability. Shared private JSON readers open nonblocking to reject special files without hanging.
  • Added an opt-in, row/byte-capped retail value profile to migration dry-run output. It emits aggregate null, blank, cast-failure, and duplicate-key counts without source values and documents SQL scan costs and CSV inventory limits.
  • Tightened reviewed source mappings so aliases into modeled retail fields must use the canonical model cast.
  • Added sample-only relationship orphan counts to the bounded retail profile and a private, allowlisted evidence-index generator for migration manifests.
  • Added an age-based, read-only state-retention preview and explicit operator decision guidance for interrupted load recovery.
  • Restricted SQL sources to read-only SQLite snapshots and PostgreSQL via psycopg, with bounded connection/query settings; unsupported dialects and conflicting overrides fail closed.
  • Marked missing profile keys unavailable and use capped row-size estimation before serialization; added sampled relationship completeness labels and private evidence-index input/output validation.
  • Fixed SQL dlt mapper entity capture so synthetic extract-only rows retain the correct table lineage and budget accounting. One valid SQL alias/cast and one invalid-integer rejection were verified without loading; broader cast coverage and destination loading remain unvalidated.

BigQuery

  • Added an optional incremental daily sales mart with a configurable 90-day update lookback; it is disabled by default.
  • Added unit and singular tests for sales metrics and date/product grain.
  • Added a cost-check guide comparing Cloud Billing Reports with regional query-job byte metadata.
  • Added dbt query-job labels for edition/target/invocation attribution in BigQuery.
  • Changed the BigQuery dependency lock to a Dependabot-discoverable, hash-pinned requirements-lock.txt file.
  • Added a Looker Studio walkthrough with cost and access guidance.
  • Changed BigQuery CI to install the universal hash lock and support an opt-in manual WIF warehouse build on main.
  • Fixed sales mart clustering to use a column present in the model output.

Snowflake

  • Added an optional incremental order example with a three-day source-update lookback; it is disabled by default.
  • Added a data test for customer coverage and nonnegative completed metrics.
  • Changed pull request CI to credential-free parsing and trusted main pushes to the secrets-backed build.
  • Fixed the completed-order count to return zero when no orders are complete.
  • Added a Snowflake project-cost guide with Snowsight navigation, billed daily credits, hourly warehouse compute, access, and latency notes.
  • Added a configurable Snowflake query tag for statement attribution in query history.

Cross-edition and documentation

  • Added SQL source read-only and timeout controls for PostgreSQL via psycopg and existing SQLite files; unsupported dialects fail closed. SQLite's query deadline is cooperative, and neither source byte caps nor query timeouts bound all driver memory.
  • Added configurable per-row, per-table, and per-run row/byte budgets to the migration loader. Extraction and normalization stage before destination loading; rejected inputs record only private reason/count summaries. SQL extraction uses a one-row dlt chunk size, and dry-run readiness distinguishes preflight from unenforced extraction limits. Pending packages may remain after rejected extraction and require deliberate review. Runtime validation is pending.
  • Fixed migration dlt project-root and telemetry setup before imports, preserved caller-relative CSV paths, suppressed driver logs, and aligned DuckDB confirmation with dlt's <pipeline_name>.duckdb default.
  • Added an optional CSV/SQL full-load migration prototype with PostgreSQL driver support, destination-write-free --dry-run inventory, effective dlt destination and physical table confirmations, pre-created namespace checks before extraction and immediately before loading, private migration-local state/manifests, pending-package and load-result safeguards, and separate opt-in Bronze/Silver/Gold paths for BigQuery and Snowflake. Dry runs resolve destination configuration and may resolve credentials; they do not query or write to the destination. A residual race remains between the final namespace probe and the separate load API call. The prototype remains unvalidated end to end and has not been run against either warehouse.
  • Added an opt-in Bronze reconciliation report with explicit per-table and overall count-match, count-mismatch, and count-unavailable statuses. The report is labeled count-only diagnostic and always marks cutover readiness as not assessed/false. It uses application-counted CSV/SQL rows yielded during extraction and dlt 1.32.0 normalized/prepared row counts; current destination counts require explicit target/table confirmation. SQL preflight does not independently count source rows, and source/destination equality and Silver/Gold reconciliation remain unverified. This has not been validated end to end.
  • Added a read-only migration state inventory for pipeline-state and run-manifest paths, sizes, modification ages, and available ownership/permission warnings. It does not open file contents, modify files, or perform cleanup.
  • Added a credential-blind first-run preflight for Python/dbt installation, edition files, and local profile isolation; included it in scaffolded workspaces and edition archives.
  • Added a SHA-256 fingerprint of selected table and ordered column names to dry-run output and run manifests; extraction refuses a name-level schema mismatch. Type/value drift remains outside this fingerprint.
  • Added normalized type-family fingerprints for SQL schemas and an explicit untyped marker for CSV inference; unsupported SQL labels remain visible as other: families.
  • Added an opt-in, versioned JSON source mapping format that validates aliases/casts, preserves original source columns, adds canonical Bronze aliases, and rejects invalid casts before destination loading.
  • Updated BigQuery and Snowflake quick starts to run the preflight before dbt connection/build commands.
  • Added pinned Gitleaks behavior smoke checks using ephemeral fixtures and a clean release-archive build/audit workflow for both editions.
  • Pinned third-party GitHub Actions by full commit SHA and documented CI trust boundaries and dependency advisory handling.
  • Expanded the private release record with scanner, action, workflow, dependency, and archive evidence fields.
  • Added separate BigQuery and Snowflake Terraform modules for disposable non-production environments, with least-privilege scope, safe defaults, and interactive teardown.
  • Added a shared one-command Terraform lifecycle wrapper, offline wrapper tests, and credential-free Terraform validation CI.
  • BigQuery CI identity uses keyless workload identity federation; repository Actions variables are provisioned as part of Terraform.
  • Documented required owner-supplied cloud targets, permissions, private state handling, and destructive data deletion. No cloud resources were applied.
  • Added offline regression tests for credential filtering, scaffolding, package generation, and lock consistency.
  • Added workflow linting with checksum-verified Actionlint and redacted Gitleaks file scanning for pull requests and published releases.
  • Extended offline scaffolding and archive regression fixtures to cover the Snowflake edition path.
  • Added a repository script-test workflow and a pull-request template with validation and data/cost review prompts.
  • Added buyer discovery, offer, pilot, first-run, metric contract, operations, release, acquisition, support, source selection, AI gate, and compatibility materials.
  • Added edition scaffolding and ZIP packaging scripts that exclude local secrets and generated artifacts.
  • Added heuristic credential detection, a per-file archive manifest, and companion archive checksums.
  • Added a privacy-safe GitHub support form, weekly dependency updates, and CI timeout/concurrency guards.
  • Added source-change, archive-audit, and release-validation templates.
  • Added cross-cloud cost guardrails, a private cost review record, and links from the operations and root guides.
  • Changed setup instructions to use Python 3.11 hash locks and documented release/workspace procedures.

Dependencies and security

  • Dependency locks were regenerated as universal Python 3.11 hash files with a .txt extension for updater discovery; Dependabot proposals are configured but have not run.
  • Added CI checks that direct requirements have compatible exact pins in each edition's hash lock before installation.
  • Added heuristic credential filtering for scaffolding and package generation; this is not a full secret scan.
  • Pinned Actionlint 1.7.12 by release checksum and Gitleaks 8.30.1 by container digest in read-only repository checks.

Validation and known limitations

  • Validation recorded for earlier changes: both universal locks installed with --require-hashes in temporary Python 3.11 environments; lock-consistency checks passed; 14 offline repository utility tests passed at that time; credential-free dbt parse passed for both editions; all workflow YAML parsed; git diff --check passed. Read-only usage queries ran against the active default contexts.
  • The Actionlint/Gitleaks configuration change and its remote workflows were not run. The Actionlint and Gitleaks CI configuration is authored but needs a GitHub run; review redacted scanner findings before release.
  • Known limitations: no dbt warehouse tests/builds, dashboard, customer interview, or release was run; Snowflake usage could not be matched to the dbt profile; BigQuery billed dollars were not retrieved from Billing Reports; license and data attribution decisions remain open.
  • Required migration or operator action: configure CI variables/secrets and isolated targets before a warehouse run; review release terms before distribution.