Databricks

Operator's take

Most data teams don't have a tooling problem — they have a silo problem. The engineers are building pipelines in one system, the analysts are querying in another, and the data scientists are training models somewhere else entirely. Databricks is the bet that you can collapse all of that into one platform and have everyone working on the same governed data, without forcing them all to use the same interface.

The lakehouse architecture is the central idea: you get the cost and flexibility of a data lake (store everything, pay for what you use) layered with the reliability and queryability of a data warehouse (ACID transactions, schema enforcement, fast SQL). The practical result is that you're not choosing between cheap-but-messy and structured-but-expensive. The Delta Lake layer handles reliability while collaborative notebooks let SQL analysts and Python engineers work alongside each other on the same data.

The AI layer has expanded significantly. Genie lets business users query enterprise data in plain language without SQL. Agent Bricks is the framework for building production AI agents grounded in your data. AI/BI handles natural-language dashboard creation. And Lakebase — a serverless Postgres database integrated with the lakehouse — rounds it out for teams building applications and agents that need a transactional database alongside their analytics.

This is firmly enterprise terrain. The learning curve is real, the pricing scales with compute (DBU-based, pay-as-you-go), and the AI and AutoML features are there but they're runway lights, not the destination. If your data team is fewer than five people and your analytics stack is basically one database and a BI tool, Databricks is overkill — Snowflake or even Supabase will serve you better at a fraction of the cost and operational overhead. Databricks earns its keep when fragmentation is the actual problem and you have the engineering resources to match.

What it's good at

• Lakehouse unification — combines data lake flexibility and data warehouse reliability in one platform, so engineering, analytics, and ML teams all work from the same governed data source.
• Collaborative notebooks — SQL, Python, and R in real-time shared notebooks with visualization; analysts don't need to be engineers to contribute.
• Delta Lake reliability — ACID transactions on big data workloads protect pipelines from failures and concurrent writes; you don't lose data when a job fails mid-run.
• AutoML and low-code ML — automated hyperparameter tuning and model selection lets less-technical users build ML models without deep data science expertise.
• Genie — conversational analytics interface; business users ask plain-language questions and get answers from enterprise data without writing SQL or filing a ticket with the data team.
• Agent Bricks — framework for building and deploying production AI agents grounded in your organization's data, with quality evaluation built in.
• AI/BI — natural language dashboard creation and conversational analytics for end users who need insights without BI-tool training.
• Lakebase — serverless Postgres database integrated with the lakehouse, built for applications and AI agents that need a transactional layer alongside analytics, with autoscaling to zero and database branching. Databricks acquired Neon, the serverless Postgres company, in May 2025 and launched Lakebase shortly after; Lakebase's own product page is careful to say it runs the open-source Postgres engine with a Postgres extension rather than positioning itself as a Neon rebrand.
• LakeFlow Connect — managed connectors (the Lakeflow product family also includes Lakeflow Jobs and Spark Declarative Pipelines) that pull data from external sources into the lakehouse without custom integration code.
• Governance at scale — built-in data lineage, access controls, and audit capabilities designed for regulated industries where you can't cut corners on who sees what.

What it's not

• Not for small teams — designed for organizations with data engineering resources; a team of two analysts without engineering support will spend most of their time managing the platform rather than using it.
• Not cheap — compute-based (DBU) pricing scales quickly as workloads grow; the free trial and Community Edition (Spark learning) exist, but serious workloads add up fast, especially at enterprise scale.
• Not a replacement for a standalone BI tool — Databricks now ships native AI/BI Dashboards (natural-language dashboard creation) and Genie for conversational analytics, so light BI needs are covered in-platform; deeper, polished end-user reporting is still where many orgs reach for Tableau or Power BI alongside it.
• Not the move if you're primarily warehouse-first — if your team's work is mostly SQL analytics with simple reporting needs, Snowflake or BigQuery give you the same query experience with far less operational overhead.