Iceberg Data Platform · October 2026

An open lakehouse you actually own.

Apache Iceberg as the table format, with every other building block open source, replaceable, and running on a single laptop.

Sander de Wildt
Apache Iceberg · Polaris · Keycloak
RustFS · marimo · DuckDB

Intro

Hi,

Knowledge about Apache Iceberg is not spreading as easily as it should. Before you can start, you need quite a few components: a catalog, object storage, identity, an engine and a place to work. Vendors make that part easy, and that is usually where vendor lock-in starts.

So I built a small platform that shows the other route: every building block open source, running on one laptop with a single command. Apache Iceberg is a very good choice for autonomy and portability.

Three things to land today

How Iceberg actually works, and why it is not a platform like Snowflake.
What a fully open source stack gives us: autonomy, portability and no lock-in.
A demo of one grid sensor data product: loaded by a team, given meaning by a coding agent, shared with another team, listed in a data catalog and asked about in a chat.

Intro · How I build

Five principles behind every choice

1 · Clean UXOne Design System. From sign-in to a notebook in a few clicks, one design language in both portals, and a Getting started page in each.

2 · API firstThe portal is just a client. Every button is a documented route with an OpenAPI schema, so a script can do everything a person can.

3 · AI nativeAgents are first-class users. Both portals serve MCP with the user's own identity, data products carry their meaning in a semantic model, and agent skills ship with the installer.

4 · Open sourceNo licences. Every building block is Apache-2.0 or MIT, and so is the platform. No vendor account and no trial clock.

5 · Loosely coupledOpen interfaces between the components. Iceberg REST, S3, OIDC and MCP, so you can replace one block without touching the others. Bigger capabilities come as extensions behind one contract, so the core stays small.

The one-liner

Apache Iceberg is doing for data what open APIs did for applications.

REST and OpenAPI made it possible for any service to talk to any application. Apache Iceberg does the same for structured data: one open table standard, so no single engine owns your data.

Sander de Wildt

How Apache Iceberg works

And why it is not a database platform like Snowflake.

01 · Iceberg

Iceberg is a specification, not a product you buy

Snowflake · Databricks · BigQuery

One vendor delivers storage, compute, catalog, security and the UI as a single product.
Your tables live in the vendor's format and in the vendor's account.
You rent the capability, so leaving means migrating the data and everything built around it.

Apache Iceberg

An open specification for how table data and metadata are laid out in object storage.
Data files are plain Parquet and metadata is JSON and Avro, so no server owns them.
You bring the catalog, the engines, the storage and the identity provider, and each one stays replaceable.

01 · Iceberg

Start with what you already know: PostgreSQL

Imagine a simple table: columns, rows and SQL queries.

Who does the job?	A regular PostgreSQL table	An Iceberg table in this platform
Run your SQL	The PostgreSQL server.	An engine such as DuckDB or Spark.
Find the table and check access	PostgreSQL's system catalogs and permissions.	Polaris: the table's name, permissions and current metadata location.
Describe the columns	PostgreSQL's system catalogs.	An Iceberg metadata file in object storage.
Store the rows	Files managed and read by PostgreSQL.	Parquet files in a bucket, read directly by the engine.
Scale storage and compute	One server: more data or more queries means a bigger server.	Separate: storage grows in the bucket, and each engine brings its own compute.

Compare responsibilities, not identical internals. The last row is the separation of storage and compute: the bucket grows on its own, and DuckDB on a laptop, Spark on a cluster or Snowflake each bring their own compute to the same files. PostgreSQL also stores data in files; applications normally access those files through its server. For an Iceberg query, the engine asks Polaris for the table metadata and access, then reads table files directly from storage. Polaris can read and write Iceberg metadata during catalog operations, but query rows do not flow through it. Even when Polaris uses PostgreSQL as its backing store, that database holds catalog records and permissions; the Iceberg schema and snapshot history live in object storage. Sources: PostgreSQL architecture, PostgreSQL system catalogs, Polaris 1.8.0 entities, Polaris 1.8.0 vended credentials, Polaris 1.8.0 JDBC metastore. Background reading: The Architecture of Apache Polaris, Under the Hood.

01 · Open source

Every building block is open source

Capability	Component	Version	License	Swappable for
Table format	Apache Iceberg (PyIceberg)	spec v2 and v3 · 0.12	Apache-2.0	The standard itself, engines come and go.
Catalog and permissions	Apache Polaris	1.8.0	Apache-2.0	Lakekeeper, Nessie, Gravitino, Unity Catalog OSS, AWS Glue
Identity (OIDC)	Keycloak	26.8.0	Apache-2.0	Entra ID, Okta, any OIDC provider
Object storage (S3 API)	RustFS	1.0.1	Apache-2.0	AWS S3, MinIO, Ceph, Cloudflare R2, on-prem
Catalog metadata store	PostgreSQL	18	PostgreSQL	Managed Postgres anywhere
Notebooks	marimo	0.25.1	Apache-2.0	Jupyter, VS Code, any Python client
Query engine	DuckDB with the iceberg extension, reads and writes	1.5.6	MIT	Trino, Spark, Flink, Polars, ClickHouse
Portals and API	FastAPI, Python 3.14	0.143	MIT	Built here, deliberately thin.
Semantic models	Apache Ossie (incubating), stored by Polaris	spec 0.2.0	Apache-2.0	Any tool that reads the open specification
Agent interface	Model Context Protocol, Python SDK	2.3	MIT	Any MCP client: Claude Code, IDEs, a custom agent
Conversational BI (extension)	Pydantic AI, CopilotKit, Chart.js, DuckDB	2.55 · 1.79 · 4.5 · 1.5.5	MIT	Any LLM provider: OpenAI, Anthropic, Gemini, Bedrock
dbt pipelines (extension)	dbt v2 OSS on DuckDB	2.0.8 · 1.5.5	Apache-2.0	Any dbt adapter for an Iceberg engine
Ingestion (extension)	dlt with its Iceberg destination	1.31	Apache-2.0	Airbyte, Meltano, any Iceberg REST writer
Data catalog (extension)	OpenMetadata, fed through its REST API	2.0.5	Apache-2.0	DataHub, Unity Catalog OSS, Apache Atlas

The platform

Teams, databases, AI agents and extensions, on one identity and with each user's own permissions throughout.

02 · Platform

Architecture

02 · Platform

Polaris RBAC per team: the high-level view

02 · Platform · Team workspace

Team administrators run their own databases

User portal Databases page with the create database form filled in

02 · Platform · AI agents

The catalog, for an AI agent

claude mcp add --transport http \
  --client-id iceberg-mcp --callback-port 3010 \
  iceberg-user http://localhost:3002/mcp

list_databases	Teams, the role in each, owned and shared databases
list_namespaces · list_tables	Only what the grants allow
describe_table · describe_view	Schema, partitioning, snapshots, branches, view SQL
list_semantic_models · describe_semantic_model	Datasets, joins, agreed metrics and AI context · a summary, searchable, for large models
query_semantic_model	Answers from the agreed metrics and joins, read with the user's own grants
list_dimension_values	Exact filter values, such as Example Dunmore
preview_rows	Up to 100 rows, at any snapshot
list · create · update · delete_share	Team administrators only · sharing a model adds its tables
create · rename · delete_database	Team administrators only · delete needs the exact name

The agent signs in through Keycloak as you: a public client with PKCE, no secret to store.
Every call runs with your Polaris grants, so the agent sees exactly what you see and nothing more.
The agent never writes SQL: it names metrics, dimensions and filters, and the portal builds the query from the semantic model, so a metric always uses the agreed definition.
A shared database stays read-only for the agent too.
Destructive tools are marked as such, and the agent is told to ask before using them.

02 · Platform · Extensions

Keep the core small: extensions come in through one Bridge

This is the loosely coupled principle one level up. An LLM chat brings a heavy dependency tree, a model provider and a fast release cycle; none of that belongs in the catalog and identity layer every team depends on. The core stays small: catalog, storage, identity, portals and notebooks. Sources: contracts/bridge/README.md and COMPATIBILITY.md in the core, and the AGENTS.md of the extension. The Bridge capabilities are user-context, automation-principals, automation-tokens, shared-data and listings. OpenMetadata uses only listings: the metadata of the shares a team lists in the data catalog, never a catalog token. An automation principal is a service account for one team in one environment; only that team's administrators create it, and it reaches the data shares the team received through their read role. Its catalog tokens last an hour. Conversational BI asks only for read tokens. dbt pipelines and Ingestion (dlt) write: each run is an isolated container that writes as the team's service account with a short-lived token, and Ingestion keeps pipeline secrets encrypted and write-only. The extension signs people in through its own Keycloak clients, whose tokens Polaris refuses, so a chat session can never be replayed against the catalog. Any producer can describe a table with the table-property conventions, which the catalog shows as Produced by.

02 · Platform · Extensions

Team administrators switch it on, the platform keeps oversight

Conversational BI model list for mila with grid-planning development and production not enabled yet, each with an Enable button

Admin portal Teams page with the extension service accounts of Conversational BI for outage-response and grid-planning, each with Revoke

02 · Platform

Security model, in one slide

The user portal, Polaris, the notebooks and MCP all work with that user's Keycloak token. No shared service account for data access.
Polaris vends short lived, table scoped S3 credentials; notebooks never see bucket root keys.
Each notebook session gets its own container and network: read-only root filesystem, dropped capabilities, CPU, memory and process limits, no Docker socket.

Revoking a membership stops the affected runtimes within 30 seconds.
A data share holds one read grant per table, view or semantic model. A receiving team never gets administration; an external secret is shown once, never stored. A listing in the data catalog publishes metadata only.
MCP accepts only tokens issued to the iceberg-mcp client, verified on every call: an agent never holds more than its user.
Extensions get no platform secrets or storage keys: a team's service account, enabled by its administrators and revocable by the platform administrator. dbt and Ingestion runs are isolated containers with a short-lived token.

From raw data to answers

A load forecast written in a notebook, given meaning by its contract, shared, listed in a data catalog and asked about in a chat.

03 · Demo · Notebook

A notebook checks the data product, then writes it

Example notebook 09 in the grid-sensors database: a query joins the congestion outlook to assets, feeders and substations and counts future congestion points per substation, with the resulting table

Example notebook 09: the contract's quality checks, run as SQL, all with zero violations, before the tables are written to Iceberg

The demo product is a synthetic medium-voltage load forecast for network architects: six substations, 48 feeders and 288 assets, a forecast per asset, year and direction from 2026 to 2040, and a precomputed congestion outlook that says when each asset becomes congested and how badly. It is example notebook 09, which every team gets in its notebook workspace. One contract file, grid_forecast.product.yaml, defines the product: the tables, a description for every column, keys, joins, metrics and quality checks. The notebook generates every row, derives the congestion outlook and runs the contract's SQL checks; only when all of them pass does it write the eight Iceberg tables, partitioned by year. The notebook's engine is just an example: any engine with an Iceberg writer works the same way. Polaris checks sander's identity and vends storage credentials scoped to each table. A rerun overwrites the rows of the same tables, so their history and shares survive.

03 · Demo · Semantic model

The meaning, stored beside the tables

Diagram tab of the mv_load_forecast semantic model: eight datasets drawn as tables with their keys, relationships as lines between the join columns

↓ AI context and metrics

User portal · semantic model overview with its AI context

User portal · the agreed metrics and relationships

The notebook stores the meaning as an Apache Ossie semantic model in Polaris, in the same namespace as the tables. It comes straight from the contract: every dataset points at its Iceberg table and names its key, every field has a description, a label for its concept and synonyms, the relationships state the joins, and the metrics carry the agreed SQL, such as congestion_points and peak_utilization_pct. The AI context holds what a person would tell a new colleague: the grain, the horizon, the missing-value policy, and never to add up peaks of different assets, because they do not occur at the same moment. The catalog reads the model with the user's own grants, like a table, and draws it. Ossie points a relationship from the many side to the one side; the platform only joins that way, so no row is ever counted twice.

03 · Demo · Data sharing

Share the model with another team, without a copy

User portal data share form with outage-response as recipient team, the mv_load_forecast semantic model selected and its eight tables added

noor's catalog of the shared grid-sensors database, marked shared and read-only: eight tables and one semantic model, owner named

03 · Demo · Data catalog

List it in the data catalog, so others find it

User portal data share form for mv-load-forecast-catalog: only Listed in the data catalog checked, with grid-planning@example.com as the contact for access requests and the mv_load_forecast model selected

OpenMetadata data product mv-load-forecast-catalog: the agreed metrics, the ER diagram of its eight tables with their primary and foreign keys, and the list of foreign keys

↓ a table and a metric up close

OpenMetadata · load_forecast with the model's descriptions, glossary terms and keys

OpenMetadata · the agreed congestion_points metric with its description and dimensions

A listing is the third audience of a data share, next to another team and an external party. It grants nothing: no principal role, no credential, no data. The platform publishes names, column schemas, descriptions and the semantic model through the Extension Bridge's listings capability, never view SQL, storage locations or snapshots. The OpenMetadata extension pushes that to OpenMetadata 2.0 through its REST API: OpenMetadata dropped its own Iceberg connector and never read semantic models, so pulling was not an option. Each listing becomes a data product with its schemas, tables and metrics as assets. The model's datasets and fields become glossary terms, their descriptions land on the tables and columns, its keys become table constraints and an ER diagram, and every metric becomes a Metric entity with its SQL, synonyms and dimensions. Field labels become classification tags. Unlisting, revoking or an expiry removes it within a minute. OpenMetadata is one extension; another data catalog can use the same contract. Someone who finds the product asks grid-planning for a data share.

03 · Demo · Conversational BI

noor asks the shared model a question

Conversational BI answer for noor on the shared mv_load_forecast model: a query card with congestion_points by SUBSTATION.name for offtake, the join path from congestion to substation, and a bar chart led by Example Dunmore with 38

↓ the SQL

Conversational BI · the SQL compiled from the model: the agreed metric and the joins from congestion to substation

03 · Demo · Conversational BI

How a question becomes a governed query

1 · The LLM picks namesanalyst.yaml · any providerMetrics, dimensions and filters from the selected model. It sees the model's context and at most 20 sample rows, never a credential.

2 · The model writes the SQLcompiled, not generatedThe agreed metric SQL, joined only along the model's relationships. An allowlist refuses subqueries, table functions and file readers.

3 · A disposable worker readsDuckDB · one process per queryWith the team's one-hour read token and vended, table-scoped credentials. Shared models need nothing extra: the team's read role reaches them.

4 · The answer shows its workCopilotKit · Chart.jsQuery card, definitions, join path and SQL. The chart loads its rows from the result by id, never from the LLM.

Wrap up

As far as I'm concerned

Iceberg complements platforms like Snowflake. Use it where openness pays off: data that several teams, engines or partners read. Not every table needs to be an Iceberg table.
Combining data products is where an integrated platform wins. Joining many data products in one place with good performance is easier when storage, compute and catalog come from one vendor.
For Iceberg tables, engines are replaceable. My advice is to treat them as workload decisions, not as platform commitments.
Meaning belongs next to the data. A semantic model is part of the data product: owned by the team, checked before it is published, and shared and governed like the tables.
Self-service only works with clear ownership. Teams create, own and share their databases, while the platform team owns the building blocks.
Agents use the same front door. They get the same identity and grants as the people they work for.
Keep the core small, grow through extensions. New capabilities such as Conversational BI, dbt pipelines, Ingestion and OpenMetadata come with their own release, behind one versioned contract.

Thanks

Reach out.

Want to run this yourself, or see a gap I missed? Let me know. We'll take it from there.

Read the documentation: install, use and extend the platform
Watch the 90s launch video: six open building blocks, one platform
Watch the 60s semantic model promo: Conversational BI

Regards, Sander
github.com/sanderdw/iceberg-data-platform
Apache-2.0