Skip to content

Reliable Data Pipeline Starter Kit

Two independent dbt starter projects for retail and e-commerce analytics: one for BigQuery and one for Snowflake. Each shows a source → staging → marts layout with schema tests, cost guidance, and a pull-request CI workflow that needs no warehouse credentials.

It is aimed at data consultants and small analytics teams who want a working example to read, run in their own development environment, and adapt. It is a starting point, not a turnkey client deployment.

Choose an edition

You only need one edition to evaluate the kit. Start with BigQuery if you want realistic retail data; choose Snowflake if your team already works in Snowflake and a small seed-based example is enough.

BigQuery edition Snowflake edition
Example data Google's public TheLook eCommerce dataset (a fictional retailer), at order-item grain Synthetic CSV seeds: 3 customers and 4 orders, with no product or item rows
What you need A Google Cloud project; BigQuery Sandbox allows limited exploration without billing A Snowflake account with a warehouse, database, schema, and role you can create objects in
Where costs land Queries run in your project. The profile sets a 1 GB per-query bytes-billed guard, which is not a total spending cap Warehouse compute in your account. Use a small, auto-suspending development warehouse
Models stg_order_items, stg_products, stg_users → mart_daily_product_sales, mart_customer_value stg_customers, stg_orders → fct_customer_orders
Pull-request CI dbt parse, no credentials dbt parse, no credentials
Recorded warehouse build Not yet recorded Not yet recorded
Guides Setup, cost checks, Looker Studio walkthrough Setup, cost checks, operations

The two editions use different data and grains, so their outputs are not like-for-like. The mart contract lists which metrics match and which are edition-specific. The first-run walkthrough shows the data flow and example queries for both.

Current status

  • Pull-request CI runs a credential-free dbt parse for each edition, and the changelog records it passing. Parsing does not prove warehouse execution, and no authenticated warehouse build has been recorded for either edition; see the compatibility matrix.
  • Licence: source-available evaluation licence (LICENSE); production use, client delivery, and redistribution need a separate written licence. Support is best effort with no SLA (SUPPORT.md). Rationale: decision 0001.
  • The BigQuery edition queries TheLook in place and ships none of its rows; see NOTICE for attribution. The dataset's own listing terms also apply and have not been reviewed by the owner; review them before using or sharing TheLook data or screenshots.

The trust and FAQ page answers common questions about cost, data, credentials, and validation, with links to the supporting guides.

For contributors

Each edition has its own dependency lock, dbt project, profile example, and setup instructions. Work from that edition's directory and do not mix profiles or generated dbt artifacts between editions. See AGENTS.md for contribution conventions. Never commit credentials, private customer data, or local profile files.

editions/
  Bigquery/       BigQuery online retail dbt project and guide
  Snowflake/      Snowflake dbt project and setup/adaptation guides
docs/             Shared guides (staged into the docs site)
scripts/          Docs-site build and dependency-lock check
website/          MkDocs Material site config (built site is gitignored)
.github/workflows/  Edition-specific CI workflows

Documentation site

A MkDocs Material site packages these Markdown guides plus embedded dbt docs for each edition. After GitHub Pages is enabled, the published URL is https://htiwari-dev.github.io/de-starter-kits/.

Local preview (Python 3.11):

./scripts/build_docs_site.sh
cd website && .venv/bin/mkdocs serve

See Documentation site for CI, empty-catalog dbt docs, and the optional Cloud Run container path.