Apache Iceberg as the table format, with every other building block open source, replaceable, and running on a single laptop.
Knowledge about Apache Iceberg is not spreading as easily as it should. Before you can start, you need quite a few components: a catalog, object storage, identity, an engine and a place to work. Vendors make that part easy, and that is usually where vendor lock-in starts.
So I built a small platform that shows the other route: every building block open source, running on one laptop with a single command. As far as I'm concerned, Iceberg is a very good choice for autonomy and portability, not only for scale.
Read morehttps://medium.com/@sanderdw/making-apache-iceberg-feel-like-a-database-service-e89c5d0bed98
https://www.linkedin.com/posts/sanderdw_at-alliander-we-want-to-take-data-sharing-ugcPost-7505007789820256256-eoeA/
These principles decide the trade-offs. When a feature does not fit them, it does not go in.
$ curl -fsSL https://github.com/sanderdw/iceberg-data-platform/releases/latest/download/install.sh | sh
Downloading release v0.5.2...
Pulling images for both stacks (the notebook image may take a few minutes)...
Starting the user portal...
Iceberg Data Platform 0.5.2 is running.
1. Sign in to the administration portal
http://localhost:3000/#guide
Username platform-admin
Password •••••••••••• (temporary: you choose a new one)
2. Create a team and a user, then sign in as that user
http://localhost:3002/#guide
Or skip the setup: the demo-company skill below does it
Optional
Skills for coding agents such as Claude Code, Codex and GitHub Copilot:
quick-share Share the portals for a class or demo
demo-company Create a demo company, one account per participant
.env. It ends with who signs in first, and where.iceberg_connect.py for your own tools and the agent skills in the same folder..env and the data stay.
[mcp_servers.iceberg-user]
url = "http://localhost:3002/mcp"
scopes = ["openid", "profile", "offline_access"]
[mcp_servers.iceberg-user.oauth]
client_id = "iceberg-mcp"
claude mcp add --transport http --client-id iceberg-mcp \
--callback-port 3010 iceberg-user http://localhost:3002/mcp
Codex, GitHub Copilot, Claude Code or any other MCP client. One URL and one public client ID, no secret.
The agent opens Keycloak in your browser. From then on it works as you, with your grants.
“Which tables in smart-meter-readings changed this week, and what do their schemas look like?”
The portals print these recipes with your own address, for both the user and the administration endpoint.
> Set up a demo company for my class
How many participants? 12
Which company? Energy · Voltara Energy
Plan · 3 teams of 4 · 1 team admin and 3 writers each
create_team grid-planning, … 3
create_database production, development 6
create_user one per participant 12
# Voltara Energy demo accounts: 12 participants
| # | Username | Team | Role | Temporary password |
| 1 | noor-bakker | grid-planning | admin | •••••••••••• |
| … | | | | |
A fictional Energy, Webshop or Retail company for a class of up to 48. One account per participant, teams with their own databases, and a sign-in list to hand out. All through the administration MCP server.
Opens the portals to participants anywhere for a class or demo, over temporary public HTTPS addresses. No app to install, no certificate to import, and it knows how to undo it.
In .agents/skills of the installation folder, in the open Agent Skills format. Start your agent there and ask.
A skill carries the know-how, MCP does the work with your permissions, through the same API as the portal.
REST and OpenAPI made it possible for any service to talk to any application. Apache Iceberg does the same for structured data: one open table standard, so no single engine owns your data.
And why it is not a database platform like Snowflake.
The point of the comparison is ownership: which parts of the stack you control yourself, and which parts you rent.
Imagine a simple table: columns, rows and SQL queries.
| Who does the job? | A regular PostgreSQL table | An Iceberg table in this platform |
|---|---|---|
| Run your SQL | The PostgreSQL server. | An engine such as DuckDB or Spark. |
| Find the table and check access | PostgreSQL's system catalogs and permissions. | Polaris: the table's name, permissions and current metadata location. |
| Describe the columns | PostgreSQL's system catalogs. | An Iceberg metadata file in object storage. |
| Store the rows | Files managed and read by PostgreSQL. | Parquet files in a bucket, read directly by the engine. |
PostgreSQL brings these jobs together. With Iceberg, you choose the components separately.
The engine follows a chain of references. Each step answers one question.
A bucket can also contain old or unused files. The references define the table.

In PostgreSQL, you inspect columns with \d or information_schema. Here, they are recorded in JSON.
{
"schemas": [{
"schema-id": 0,
"fields": [{
"id": 2,
"name": "home_id",
"type": "string"
}]
}],
"current-snapshot-id": 4747982721264509729,
"snapshots": [{
"snapshot-id": 4747982721264509729,
"manifest-list":
"s3://db-…/metadata/snap-4747…avro"
}]
}
home_id is text. Its stable ID lets an engine recognize the column after a rename.
The snapshot ID selects a saved state of the table's data.
The manifest-list path leads to the file lists for that snapshot.

Imagine adding today's electricity readings. Readers should see the whole batch together.
The writer adds the new rows and file lists to storage. The current table still points to the previous version.
New table metadata is written. Polaris checks the commit and atomically switches the current metadata reference.
New queries can see the batch. Queries already reading an older snapshot keep a consistent view.
If another writer commits first, the change must be rechecked and retried or rejected.

| You want to… | A regular PostgreSQL table | An Iceberg table |
|---|---|---|
| Publish a batch safely | Commit a transaction; it can span several tables. | Publish one table version atomically through the catalog. |
| Rename a column | Change the column definition with ALTER TABLE. | Update metadata; the column ID stays the same and data files stay in place. |
| Read an earlier version | Usually needs history tables or a backup restore. | Choose an older snapshot, while its metadata and files are retained. |
| Use another engine | Other tools normally query through PostgreSQL. | Compatible engines read the same table files directly. |
Iceberg defines the table rules. Engines and catalogs implement them; feature support varies.
That separation is deliberate: because the building blocks are independent, you decide who owns each one.
Autonomy, portability, and the trade-offs that come with them.
| Capability | Component | Version | License | Swappable for |
|---|---|---|---|---|
| Table format | Apache Iceberg (PyIceberg) | spec v2 and v3 · 0.12 | Apache-2.0 | The standard itself, engines come and go. |
| Catalog and permissions | Apache Polaris | 1.7.0 | Apache-2.0 | Lakekeeper, Nessie, Gravitino, Unity Catalog OSS, AWS Glue |
| Identity (OIDC) | Keycloak | 26.7.4 | Apache-2.0 | Entra ID, Okta, any OIDC provider |
| Object storage (S3 API) | RustFS | 1.0.0 | Apache-2.0 | AWS S3, MinIO, Ceph, Cloudflare R2, on-prem |
| Catalog metadata store | PostgreSQL | 18 | PostgreSQL | Managed Postgres anywhere |
| Notebooks | marimo | 0.25.0 | Apache-2.0 | Jupyter, VS Code, any Python client |
| Query engine | DuckDB with the iceberg extension, reads and writes | 1.5.5 | MIT | Trino, Spark, Flink, Polars, ClickHouse |
| Portals and API | FastAPI, Python 3.14 | 0.141 | MIT | Built here, deliberately thin. |
| Agent interface | Model Context Protocol, Python SDK | 2.2 | MIT | Any MCP client: Claude Code, IDEs, your own agent |
The data is Parquet in a bucket you control. Moving means copying the bucket and pointing another catalog at it, without an export job or a negotiation about egress.
Iceberg REST is an open API, so Polaris today can be another REST catalog tomorrow. The table metadata already lives next to the data.
Whether a team runs local analysis with DuckDB, distributed queries on Trino, or Snowflake where that is justified, every engine points at the same dataset without duplication.
Vendor lock-in rarely happens overnight; it creeps in through the default choices. Open formats keep the platform flexible and save us a costly migration later.
curl -fsSL https://github.com/sanderdw/iceberg-data-platform/releases/latest/download/install.sh | shOpen source does not mean free of work. The ownership you gain comes with the responsibility that goes with it.
Teams, databases, notebooks, data shares and AI agents, on one identity and your own permissions throughout.
Ownership is explicit: a team owns a database and a database owns a bucket. That is the governance model.
| GET POST DELETE | /api/session | Auth state, sign in, sign out |
| GET | /api/overview | Health and every resource |
| GET POST PATCH DELETE | /api/teams[/{id}] | Teams |
| GET POST PATCH DELETE | /api/databases[/{id}[/name]] | Create, move, rename, delete |
| GET | /api/databases/{id}/connection | Iceberg REST settings |
| GET POST PATCH DELETE | /api/users[/{id}] | Users, a role per team |
| DELETE | /api/shares/{id} | Revoke a data share |
| GET POST | /api/identity/{accounts,users,links} | Keycloak accounts |
| POST | /api/users/{id}/identity[/…] | Link, retry, reset password |
| GET | /api/admin/explorer/{databases,contents} | Browse every catalog |
| GET | /api/{infrastructure,health} | Live metrics, liveness |
| MCP | /mcp | The same, as agent tools |
| GET POST DELETE | /api/session | Sign in, sign out |
| GET PATCH | /api/{workspace,team,environment} | My context, switch it |
| POST PATCH DELETE | /api/databases[/{id}] | Team administrators |
| GET | /api/{contents,details} | Tables, schema, snapshots |
| POST | /api/preview | 100 rows max |
| POST DELETE | /api/notebooks[/{id}] | Start, stop marimo |
| GET POST PATCH DELETE | /api/shares[/{id}[/rotate]] | Outgoing shares |
| GET | /api/{share-teams,received-shares} | Targets, received |
| ANY · WS | /workspaces/{id}/… | Notebook proxy |
| MCP | /mcp | Agent tools |
There is no private backdoor for the UI: every button is a fetch to one of these routes, and the OpenAPI schema is generated from the same code. One level down it is open APIs again: Iceberg REST on Polaris, S3 on RustFS and OIDC on Keycloak.







CREATE TABLE lakehouse.iceberg_v3.sensor_events (
event_id BIGINT,
device_id VARCHAR,
site VARCHAR,
location GEOMETRY,
measured_at TIMESTAMP_NS,
payload VARIANT,
synthetic BOOLEAN
) WITH ('format-version' = 3);
ALTER TABLE … ADD COLUMN firmware VARCHAR DEFAULT 'v1.0';
UPDATE … SET firmware = 'v2.0' WHERE device_id = 'SENSOR-001';
DELETE FROM … WHERE payload.kind::VARCHAR = 'fault';
firmware rewrites none of the 480 rows.
A team administrator picks tables and views of one database in the user portal and chooses the recipient: another team, an external party, or both. No platform administrator in the loop.
Access is granted per object: an explicit read grant for each selected table or view, rather than blanket permissions on the namespace or the catalog.
Members of the receiving team use their own accounts, in the portal and in notebooks. An external party uses any Iceberg REST client with its own client ID and secret.
Every recipient reads the same files, and the team can expire or revoke access at any time. There is still only one copy of the data.
from pyiceberg.catalog import load_catalog
catalog = load_catalog(
"shared", type="rest",
uri="https://catalog.example/api/catalog",
warehouse="db-…",
credential="<client-id>:<client-secret>",
scope="PRINCIPAL_ROLE:ALL",
**{"header.X-Iceberg-Access-Delegation":
"vended-credentials"},
)
table = catalog.load_table(
"synthetic.neighborhood_electricity")
table.scan().to_arrow() # 26,880 rows
catalog.list_namespaces() # Forbidden
catalog.list_tables("synthetic") # Forbidden
catalog.load_table("synthetic.other") # Forbidden
claude mcp add --transport http \
--client-id iceberg-mcp --callback-port 3010 \
iceberg-user http://localhost:3002/mcp
| list_databases | Your teams, your role in each, owned and shared databases |
| list_namespaces · list_tables | Browse what your grants allow |
| describe_table · describe_view | Schema, partitioning, snapshots, branches, view SQL |
| preview_rows | Up to 100 rows, at any snapshot |
| create · rename · delete_database | Team administrators only · delete needs the exact name |
Any MCP client works: Claude Code, an IDE, or your own agent. The protocol is open, just like Iceberg REST.
claude mcp add --transport http \
--client-id iceberg-mcp --callback-port 3010 \
iceberg-admin http://localhost:3000/mcp
| get_overview · browse_catalog | Health, every team, database, user and share |
| create · update · delete_team | Teams |
| create · rename · move · delete_database | Databases across all teams |
| create · link_user · update_user_access | Keycloak accounts and a role per team |
| list_shares · revoke_share | Oversight on every data share |
delete_database refuses without the database's exact name.“Create a team for the heat transition, give it a development database and add Noor as a writer” takes one sentence instead of three screens.
iceberg-mcp client, verified on every call: an agent never holds more than its user.This build is meant for local development. HTTPS, persistent sessions and hostile-tenant isolation are on the roadmap, and the boundaries are documented.
Direction and ownership.
Feel free to reach out if you want to run this yourself, or if you see a gap I missed. We'll take it from there.