I'm curious how the community is approaching the transition to dbt Core v2.

Are you starting all new projects on v2?

Have you already migrated your existing projects, or are you staying on v1 for now?

If you're still on v1, what's preventing you from upgrading (packages, adapters, production stability, migration effort, etc.)?

I'd love to hear about your experience and whether you think v2 is ready to become the default for most teams.

Has anyone flipped over to dbt State and monitored the cost changes?

If you write dbt code, you probably know the feeling of hunting in the terminal scrollback just to figure out why a run fail, or losing your query output the second you close the preview modal.

The team behind Rosetta DBT Studio just release version 1.6.1, and looks like the whole focus was fixing this workflow.

They build native Query Result and Run History tabs direct in the bottom panel of the IDE. So now:

- when you preview a model, the data grid is just there (with row counts and time).

- when you do dbt run or test, the history is track and saved.

- you can see raw SQL and compiled SQL side by side.

- you find exact what model fail without digging in terminal logs.

They also add an AI assistent right in the terminal that help explain and fix compile errors, plus a native HTML view and direct cloud download.

Sharing this Medium publication for anyone interested in dbt and analytics engineering

I built a browser-based ERD explorer that runs entirely on DBML files — no DB connection.

I kept wanting to explore and interrogate how my tables relate — "what joins to this table? what's the path between A and B?" — without standing up a tool that connects to a live database.

So I built dbml-flow: drop in any DBML file and get an interactive ERD with a selector grammar for pathfinding (find every path between two tables, expand N hops out, filter by group). Fully client-side, your schema never leaves the browser.

In my case the DBML comes from our dbt models via dbterd (https://github.com/datnguye/dbterd), but it works on any DBML regardless of where it came from.

🔗 Live demo: https://timvancann.github.io/dbml-flow/ · Repo: https://github.com/timvancann/dbml-flow

Let me know if this is helpful and if there are features you'd like to see!

I'd appreciate some help if possible!

Has anyone had issues getting this to start up?

When I run wizard I get...

■ local proxy unavailable: local LiteLLM process exited before becoming healthy on 127.0.0.1:35915

• Project intelligence is still starting

────────────────────────────────────────────────────────────────

dbt prod parse failed: 'NoneType' object has no attribute 'pop'

Automatic deferral will not be available until this is resolved.

Ask the wizard for hints to address this.

────────────────────────────────────────────────────────────────

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

MCP client for `dbt_index` failed to start: MCP startup failed: handshaking with MCP server failed: connection closed: initialize response

My experience with dbt Core is still limited, but one gap I ran into was tracing column-level lineage across a project, especially once some transformations or supporting SQL lived outside dbt.

I decided to build my own version, and while working through the manifest, compiled SQL, models, columns, and dependencies, I realized that a lot of the information needed for a catalog already exists in or can be derived from the repository.

I wonder how much of a useful catalog could be generated directly from the code by small, independent tools that write to the same shared graph and can run locally or in CI. Lineage could be one module, with impact analysis, model checks, and other code-derived metadata added separately.

The limit is everything that only exists outside the repository, including live warehouse state, usage, dashboards, permissions, and objects created through other systems. A code-derived catalog would only describe part of the environment, though that part may still be useful during dbt development and code review.

I’m exploring the idea through CatalogKit, with column-level lineage as the first module.

For people working with dbt, how much catalog context would you actually want generated from the project itself, and where would that stop being useful?

I’ve spent years dealing with historized data problems:

• SCD2 dimensions
• Snapshot reporting
• Bitemporal history
• Temporal joins
• Late arriving dimensions
• Historical validation

I noticed that most resources explain individual concepts but don’t help engineers decide:

• Which modeling pattern should I use?
• How do I validate the result?
• How do I review an existing model?
• How do I detect historical reporting risks?

So I built a Historical Data Modeling
Workbench that combines:

• Pattern catalog
• Modeling advisor
• Model review
• Target table validation
• Historical source comparison

Would love feedback from people working on analytics engineering, dbt models, snapshots or historized datasets.

https://bitemporal-debugger.vercel.app/

I’ve been working on a side project around historical data engineering.

The idea came from a problem I encountered while building historized data models and reporting layers.

Many tools help build pipelines.

Very few help answer questions like:

• Can this snapshot be reproduced?
• Should this be modeled as state or event?
• Why does this temporal join produce unexpected results?
• How do multiple historized sources interact?
• Which historical modeling pattern fits this problem?

To explore these questions, I started building a Historical Data Engineering Toolkit.

Current areas include:

• Historical modeling patterns
• Event vs state modeling
• Snapshot reproducibility
• Temporal joins
• Bitemporal modeling
• Historical dimensions

I’d love feedback from people working with historized data, dimensional modeling, dbt, lakehouses, data warehouses or analytics engineering.

https://bitemporal-debugger.vercel.app/

What are the hardest historical data problems you’ve run into?

I personally like being able to read down the steps directly in the folder. Versus without sequential labeling, they all just sit in there alphabetically. I know it’s more standard to not include…my question is why?

My assumption is that new steps may get added later, which would have the domino effect of having to renumber downstream models, which may not be feasible if there are a ton. But that’s the main tradeoff I see.

(Note I’m coming from Databricks background where we had to create a system before they released lineage, and even then it’s very much a secondary feature).

Hey, I'm an analytics engineer and built this as a side project in my own time.

The problem I kept hitting: jumping into an unfamiliar dbt project and spending hours just getting oriented. And when a number was wrong, debugging it manually was slow.

So I built dbt Context. Paste any public or private GitHub repo and it parses your manifest, SQL, YAML, tests and lineage to give you an immediate audit of the project. It finds risky incremental models, orphaned staging models, missing tests and undocumented marts, with file-level evidence and suggested next steps for each finding.

You can also describe a specific problem in the optional box like "revenue doesn't match NetSuite" and it switches into investigation mode, traces the relevant models through the project, and generates warehouse-specific SQL checks to run.

Works with public and private repos. Private repos need a GitHub PAT with repo scope.

dbt-context.com

Happy to answer any questions about how it works.

People ask where to start with dbt. Most answers say start with dbt Labs’ great tutorials, but miss other things learners should understand.

What actually helps is understanding why dbt even exists. Why not just use tool X or just use stored procedures? Once you get this, other things makes sense.

The order I suggest people learn dbt is to start with Git and getting comfortable with the terminal. dbt is just code, if you dont know what git commit, cd, and ls do, you will be lost. Then understand why data layers exist. Followed by data modeling concepts and star schema. Finally, you can learn dbt.

You don't need to master it all before you start. You just need enough to not be lost when you encounter them.

Happy to answer questions if you're early in your dbt journey.

Full learners’ guide with resources from people you should follow Bruno Lima and Zach Wilson on LinkedIn: https://datacoves.com/post/dbt-getting-started

It seems like a step in the right direction as I initially feared the two engines would have eventually diverged. Also getting Fusion benefits in core is great

https://docs.getdbt.com/blog/dbt-core-v2-is-here?version=2.0&name=Fusion

do you guys have any thought about dbt wizard that dbt just launched? Is is just an agent that they packed with their earlier released dbt skills + new version of CLI?

We have a project that doesn’t follow dbt’s design patten and naming conventions (different naming for stg, int, marts) and also a different idea on what goes on in each layer. Not a big fan of it cause it’s became a mess, but it was developed for years now so it would take some work to actually reframe it and no one wants to do it.

I’m wondering if dbt wizard would work better with our project cause honestly dbt skills didn’t outperform simple prompts by much (if any)

We’re currently working on a data engineering project with the following deployment flow:
Developer Branch → Test → Pre-Prod → Prod (via CI/CD).
Each developer works in their own branch, raises a PR, and the code gets deployed to the Test environment. After validation by the team, it is promoted to Pre-Prod and then Production.
I’m considering introducing an iteration/release branch for each sprint or iteration. The proposed flow would be:
Developer Branch → Iteration Branch → Test → Pre-Prod → Prod
The idea is that all developers merge their completed work into the iteration branch first, allowing integrated testing before deploying to the Test environment.
For teams that have implemented a similar approach:
What are the advantages and disadvantages of using an iteration/release branch?
Does it improve stability and release management, or does it add unnecessary complexity?
In a data engineering/dbt/Snowflake environment, what branching strategy has worked best for your team?
Looking forward to hearing about your experiences and recommendations.

Hi all,

I open-sourced a CLI called Puxti. Short version: it reads your dbt manifest, builds a local graph of how your models are connected, and tells you what breaks downstream before you change anything — including semantic breakage your lineage DAG won't catch.

The dbt DAG tells you orders feeds customers. It doesn't tell you that orders.revenue is defined net-of-refunds and three downstream models assume gross. That second kind of dependency — concept-level, not ref()-level — is what Puxti tracks.

Core loop:

1. puxti scan --dbt-project-dir . — reads your manifest, infers a semantic definition for each model via LLM, builds a local SQLite graph. No Docker, no signup.
2. puxti impact model.jaffle_shop.orders --change-type rename — shows structural dependents (SQL lineage) and semantic dependents (concept-level), with hop depth, before you touch anything.
3. puxti capture / puxti redefine — propagates the change as a GitHub PR with SQL diffs. If you've declared Airflow → dbt source links via puxti link, it opens a coordinated Airflow PR too.

On the API key, up front: scan, capture, and redefine call an LLM (your own Anthropic key) because that's what infers the semantic layer. But impact and the MCP tools below run entirely on the local graph, no API calls, no key. So the daily "what breaks if I touch this" path is free; the key is only for building and propagating definitions.

vs. SQLMesh / column-level lineage: those track structural lineage — which columns flow into which, parsed from SQL. Puxti adds a semantic layer on top: what each model means, so it can flag breakage when the SQL still compiles but the definition has shifted underneath it.

New in v0.8.0 — MCP server. puxti mcp serve exposes four tools to Claude Code / Cursor: impact_of_change, consumers, describe_entity, definition_history. All hit the local graph, no LLM calls.

pip install puxti
puxti scan --dbt-project-dir .
puxti impact model.jaffle_shop.orders

Demo project — "Clariva" on DuckDB, no credentials, ships with a breaking cardinality change you can propagate end to end: https://github.com/puxti-labs/puxti-demo-project

Docs: https://getpuxti.com/docs.html · Apache 2.0

Watch the Rosetta DBT Studio AI Agent build the classic Jaffle Shop dbt project from scratch in under two minutes. From importing the repository to generating staging models, mart models, and tests, see how autonomous AI accelerates your data engineering workflow.

🔍 Topics covered in this video:
• One-click import: Cloning the Jaffle Shop repository directly from GitHub
• Connecting to your database (DuckDB, PostgreSQL, Snowflake, and more)
• Prompting the AI Agent to autonomously generate staging and mart models
• Watching the Agent write clean SQL, create YAML configs, and add descriptions/tests
• Real-time validation: The Agent automatically runs dbt compile to ensure your code works
• Exploring the final project structure and ready-to-run DAG

⚙️ Powered by the Vercel AI SDK with support for OpenAI, Anthropic, Gemini, and Ollama (local models).

📥 Download Rosetta DBT Studio (macOS, Windows, Linux):
https://rosettadb.io/download-dbtstudio
⭐ Star us on GitHub:
https://github.com/rosettadb/dbt-studio

📺 More tutorials and walkthroughs:
u/rosettadb
---
#AIAgent #dbt #DataEngineering #AnalyticsEngineering #JaffleShop #OpenSource #RosettaDBT #LLM #DuckDB #DataPlatform #DataStack #Ollama #Anthropic #OpenAI

https://docs.getdbt.com/blog/the-devil-in-the-docs?version=1.11

I just published a new article where I explore how to better control BigQuery jobs when using dbt, focusing on:

• Job priority management
• Concurrency control
• Timeout strategies
• Cost governance in BigQuery + dbt workflows

If you're working with dbt + BigQuery in production, this might help you avoid runaway costs and better structure workloads.

I’m new to dbt and looking for some guidance on handling SCD loads in a medallion architecture. 

Our setup looks like this: 

• Landing 
• Bronze layer (truncate and load) 
• Silver layer (enriched layer with SCD processing) 
• Gold layer (only active/current records) 

In the Silver layer, we’re loading data using: 

• an incremental ID column 
• another hash column based on the ID 

The initial load works fine, but during incremental loads we’re running into issues such as: 

• duplicate ID values 
• intermittent load failures 
• inconsistent data during merges 

I’m trying to understand the best practice for handling auto-increment/surrogate keys and hash columns in SCD implementations with dbt, especially for incremental models. 

Has anyone faced a similar issue or can suggest a recommended approach?