Iceberg Data Platform · September 2026

An open lakehouse you actually own.

Apache Iceberg as the table format, with every other building block open source, replaceable, and running on a single laptop.

Sander de Wildt
Apache Iceberg · Polaris · Keycloak
RustFS · marimo · DuckDB
Intro

Hi all,

Knowledge about Apache Iceberg is not spreading as easily as it should. Before you can start, you need quite a few components: a catalog, object storage, identity, an engine and a place to work. Vendors make that part easy, and that is usually where vendor lock-in starts.

So I built a small platform that shows the other route: every building block open source, running on one laptop with a single command. As far as I'm concerned, Iceberg is a very good choice for autonomy and portability, not only for scale.

Three things to land today
  1. How Iceberg actually works, and why it is not a platform like Snowflake.
  2. What a fully open source stack gives us: autonomy, portability and no lock-in.
  3. A demo of teams, databases, notebooks, data sharing and AI agents on one identity.
Background: Making Apache Iceberg feel like a database service on Medium · the LinkedIn post on data sharing at Alliander
Intro · How I build

Five principles behind every choice

1 · Clean UXNo ticket, no manual. From sign-in to a notebook in a few clicks, one design language in both portals, and a Getting started page in each.
2 · API firstThe portal is just a client. Every button is a documented route with an OpenAPI schema, so a script can do everything a person can.
3 · AI nativeAgents are first-class users. Both portals serve MCP with the user's own identity, and agent skills ship with the installer for work that takes many steps.
4 · Open sourceNothing to rent. Every building block is Apache-2.0 or MIT, and so is the platform. No vendor account and no trial clock.
5 · Loosely coupledOpen interfaces between the blocks. Iceberg REST, S3, OIDC and MCP, so you can replace one block without touching the others.

These principles decide the trade-offs. When a feature does not fit them, it does not go in.

First use · Install

One command, and the platform runs

$ curl -fsSL https://github.com/sanderdw/iceberg-data-platform/releases/latest/download/install.sh | sh
Downloading release v0.5.2...
Pulling images for both stacks (the notebook image may take a few minutes)...
Starting the user portal...

Iceberg Data Platform 0.5.2 is running.

1. Sign in to the administration portal
   http://localhost:3000/#guide
   Username  platform-admin
   Password  ••••••••••••  (temporary: you choose a new one)

2. Create a team and a user, then sign in as that user
   http://localhost:3002/#guide
   Or skip the setup: the demo-company skill below does it

Optional
   Skills for coding agents such as Claude Code, Codex and GitHub Copilot:
     quick-share    Share the portals for a class or demo
     demo-company   Create a demo company, one account per participant
  • Linux, macOS and Windows. Only Docker with Compose is needed: no source checkout, Python or Node.js.
  • The installer verifies the bundle and writes random credentials to .env. It ends with who signs in first, and where.
  • It pulls pinned images and starts both stacks.
  • It puts iceberg_connect.py for your own tools and the agent skills in the same folder.
  • To update, run it again. .env and the data stay.
Windows PowerShell · irm https://github.com/sanderdw/iceberg-data-platform/releases/latest/download/install.ps1 | iex
First use · Getting started

Each portal explains its three ways in

User portal Getting started page with the notebook, your own tools and AI agent tracks Administration portal Getting started page with the portal, AI agent and API tracks
User portal · notebook, your own tools, or an AI agent over MCPAdministration · portal, AI agent or API · examples use your own address
↓ full screen
User portal · Getting started
Administration portal · Getting started
First use · MCP

Connect the agent you already use

Codex · ~/.codex/config.toml
[mcp_servers.iceberg-user]
url = "http://localhost:3002/mcp"
scopes = ["openid", "profile", "offline_access"]

[mcp_servers.iceberg-user.oauth]
client_id = "iceberg-mcp"
Claude Code
claude mcp add --transport http --client-id iceberg-mcp \
  --callback-port 3010 iceberg-user http://localhost:3002/mcp
01 · Register

Codex, GitHub Copilot, Claude Code or any other MCP client. One URL and one public client ID, no secret.

02 · Sign in

The agent opens Keycloak in your browser. From then on it works as you, with your grants.

03 · Ask

“Which tables in smart-meter-readings changed this week, and what do their schemas look like?”

The portals print these recipes with your own address, for both the user and the administration endpoint.

First use · Skills

Skills turn one sentence into a finished setup

> Set up a demo company for my class

  How many participants?   12
  Which company?           Energy · Voltara Energy

  Plan · 3 teams of 4 · 1 team admin and 3 writers each
  create_team      grid-planning, …          3
  create_database  production, development   6
  create_user      one per participant       12

  # Voltara Energy demo accounts: 12 participants
  | # | Username    | Team            | Role  | Temporary password |
  | 1 | noor-bakker | grid-planning   | admin | ••••••••••••       |
  | … |             |                 |       |                    |
demo-company

A fictional Energy, Webshop or Retail company for a class of up to 48. One account per participant, teams with their own databases, and a sign-in list to hand out. All through the administration MCP server.

quick-share

Opens the portals to participants anywhere for a class or demo, over temporary public HTTPS addresses. No app to install, no certificate to import, and it knows how to undo it.

Where they live

In .agents/skills of the installation folder, in the open Agent Skills format. Start your agent there and ask.

A skill carries the know-how, MCP does the work with your permissions, through the same API as the portal.

The one-liner

Apache Iceberg is doing for data what open APIs did for applications.

REST and OpenAPI made it possible for any service to talk to any application. Apache Iceberg does the same for structured data: one open table standard, so no single engine owns your data.

Sander de Wildt
01

How Apache Iceberg works

And why it is not a database platform like Snowflake.

01 · Iceberg

Iceberg is a specification, not a product you buy

Snowflake · Databricks · BigQuery
  • One vendor delivers storage, compute, catalog, security and the UI as a single product.
  • Your tables live in the vendor's format and in the vendor's account.
  • You rent the capability, so leaving means migrating the data and everything built around it.
Apache Iceberg
  • An open specification for how table data and metadata are laid out in object storage.
  • Data files are plain Parquet and metadata is JSON and Avro, so no server owns them.
  • You bring the catalog, the engines, the storage and the identity provider, and each one stays replaceable.

The point of the comparison is ownership: which parts of the stack you control yourself, and which parts you rent.

01 · Iceberg

Start with what you already know: PostgreSQL

Imagine a simple table: columns, rows and SQL queries.

Who does the job?A regular PostgreSQL tableAn Iceberg table in this platform
Run your SQLThe PostgreSQL server.An engine such as DuckDB or Spark.
Find the table
and check access
PostgreSQL's system catalogs and permissions.Polaris: the table's name, permissions and current metadata location.
Describe the columnsPostgreSQL's system catalogs.An Iceberg metadata file in object storage.
Store the rowsFiles managed and read by PostgreSQL.Parquet files in a bucket, read directly by the engine.

PostgreSQL brings these jobs together. With Iceberg, you choose the components separately.

01 · Iceberg
↓ see the files

From a table name to its rows

The engine follows a chain of references. Each step answers one question.

1 · CatalogPolarisWhere do I start? Checks access and points to the current metadata file.
2 · Table descriptionmetadata.json · in the bucketWhat does this table look like? Its columns and saved versions, called snapshots.
3 · Lists of filesmanifest list → manifests · in the bucketWhich files belong to this snapshot? File lists and statistics help the engine choose what to read.
4 · The rowsParquet files · in the bucketWhat are the values? The actual readings your query uses.

A bucket can also contain old or unused files. The references define the table.

01 · Iceberg

Here are the files behind our demo table

RustFS console listing JSON metadata versions and Avro file lists in the demo table's metadata folder
metadata/ holds the table descriptions and file lists · data/ beside it holds the Parquet rows · larger tables have many more files
01 · Iceberg
↓ see the real file

metadata.json is the table's description

In PostgreSQL, you inspect columns with \d or information_schema. Here, they are recorded in JSON.

Selected fields from the demo · other fields omitted and storage path shortened · ↓ full file in RustFS
01 · Iceberg

You can open that description in the storage console

RustFS preview of the demo table's metadata JSON, containing its schema and snapshot references
The JSON describes the table · the readings are in Parquet · Polaris records which metadata file is current
01 · Iceberg

How a new batch becomes visible

Imagine adding today's electricity readings. Readers should see the whole batch together.

1 · Prepare

Write new files

The writer adds the new rows and file lists to storage. The current table still points to the previous version.

2 · Commit

Publish the new version

New table metadata is written. Polaris checks the commit and atomically switches the current metadata reference.

3 · Read

See a complete snapshot

New queries can see the batch. Queries already reading an older snapshot keep a consistent view.

If another writer commits first, the change must be rechecked and retried or rejected.

01 · Iceberg

A delete adds one small file instead of rewriting data

RustFS console listing the Parquet data files and Puffin deletion vectors of the v3 table
Iceberg v3 · RustFS · sensor_events/data · the Parquet data files stay untouched · each update or delete adds a Puffin deletion vector of a few hundred bytes
01 · Iceberg

What this means when you use the table

You want to…A regular PostgreSQL tableAn Iceberg table
Publish a batch safelyCommit a transaction; it can span several tables.Publish one table version atomically through the catalog.
Rename a columnChange the column definition with ALTER TABLE.Update metadata; the column ID stays the same and data files stay in place.
Read an earlier versionUsually needs history tables or a backup restore.Choose an older snapshot, while its metadata and files are retained.
Use another engineOther tools normally query through PostgreSQL.Compatible engines read the same table files directly.

Iceberg defines the table rules. Engines and catalogs implement them; feature support varies.

01 · Iceberg

What Iceberg is not

  • It is not a query engine and does not execute SQL, so you pick DuckDB, Trino, Spark, Snowflake, or all of them.
  • It is not a storage system: it sits on the S3 compatible object storage you already have.
  • It is not a security layer by itself. Permissions and credential vending live in the catalog and in your identity provider.
  • It is not maintenance free. Compaction, snapshot expiry and orphan cleanup are yours to schedule.

That separation is deliberate: because the building blocks are independent, you decide who owns each one.

02

Fully open source

Autonomy, portability, and the trade-offs that come with them.

02 · Open source

Every building block is open source

CapabilityComponentVersionLicenseSwappable for
Table formatApache Iceberg (PyIceberg)spec v2 and v3 · 0.12Apache-2.0The standard itself, engines come and go.
Catalog and permissionsApache Polaris1.7.0Apache-2.0Lakekeeper, Nessie, Gravitino, Unity Catalog OSS, AWS Glue
Identity (OIDC)Keycloak26.7.4Apache-2.0Entra ID, Okta, any OIDC provider
Object storage (S3 API)RustFS1.0.0Apache-2.0AWS S3, MinIO, Ceph, Cloudflare R2, on-prem
Catalog metadata storePostgreSQL18PostgreSQLManaged Postgres anywhere
Notebooksmarimo0.25.0Apache-2.0Jupyter, VS Code, any Python client
Query engineDuckDB with the iceberg extension, reads and writes1.5.5MITTrino, Spark, Flink, Polars, ClickHouse
Portals and APIFastAPI, Python 3.140.141MITBuilt here, deliberately thin.
Agent interfaceModel Context Protocol, Python SDK2.2MITAny MCP client: Claude Code, IDEs, your own agent
The platform itself is Apache-2.0 and is an independent project, not an Apache Software Foundation project.
02 · Open source

What avoiding lock-in means in practice

Data

The data is Parquet in a bucket you control. Moving means copying the bucket and pointing another catalog at it, without an export job or a negotiation about egress.

Catalog

Iceberg REST is an open API, so Polaris today can be another REST catalog tomorrow. The table metadata already lives next to the data.

Engines

Whether a team runs local analysis with DuckDB, distributed queries on Trino, or Snowflake where that is justified, every engine points at the same dataset without duplication.

Vendor lock-in rarely happens overnight; it creeps in through the default choices. Open formats keep the platform flexible and save us a costly migration later.

02 · Open source

Autonomy and portability

Autonomy
  • It runs on a laptop, a VM, Kubernetes or a cloud account, from the same Compose definition.
  • A one command installer for Linux, macOS and Windows, without a vendor account or a trial clock.
  • You decide where the data lives and which network it ever touches.
  • The cost is compute and storage you already pay for.
Portability
  • Storage: anything with an S3 API.
  • Identity: any OIDC provider, with users and roles staying yours.
  • Catalog: any Iceberg REST implementation.
  • Compute: whichever engine fits the job, per team and per workload.
curl -fsSL https://github.com/sanderdw/iceberg-data-platform/releases/latest/download/install.sh | sh
02 · Open source

PS: the honest part

  • Operations are your responsibility: upgrades, backups of Postgres and the buckets, monitoring, HTTPS and secrets.
  • Table maintenance is yours as well. Compaction, snapshot expiration and orphan file cleanup need a schedule.
  • Security boundaries are explicit. This build is a local development platform with in-memory sessions on a trusted Docker host, so read the security model before running it anywhere else.
  • Fewer vendors means tighter control over the architecture, but we have to be realistic: it also pushes more integration decisions onto a small team.
  • Performance across data products is harder. Combining many Iceberg data products and querying them fast takes more work than inside one integrated platform.

Open source does not mean free of work. The ownership you gain comes with the responsibility that goes with it.

03

The platform

Teams, databases, notebooks, data shares and AI agents, on one identity and your own permissions throughout.

03 · Platform

Architecture

Administrator Team member AI agentMCP client Admin portal :3000FastAPI · teams, users, databases User portal :3002FastAPI · catalog, notebooks KeycloakOIDC · people and agents Apache PolarisIceberg REST catalog · grants PostgreSQL 18catalog metadata only RustFS (S3)one bucket per database marimo runtimecontainer per user session /mcp bucket and IAM provisioning (S3 API) token validation (JWKS) vended, table-scoped credentials
Two Compose projects: iceberg-platform (admin, identity, data services) and iceberg-workspaces (user gateway, notebooks) · both portals serve /mcp for AI agents
03 · Platform

The capability model

  • A team owns databases and has a stable ID, a unique name and a description.
  • A user signs in with Keycloak, belongs to one or more teams and has a role per team: read, read and write, or admin.
  • A database is an Iceberg catalog in Polaris plus a dedicated S3 bucket. Team administrators create, rename and delete their own.
  • Every database has an environment: development, acceptance or production.
  • A data share gives another team, an external party, or both read access to selected tables and views of one database.
  • Effective access is the union of the user's teams, each at its own role. Moving a database re-syncs grants and bucket policies.
  • A team share follows membership: join the receiving team and you can read it, leave and you cannot.
  • There is no second application database: Polaris is the source of truth for teams, users and catalogs.
  • Portal administration is a Keycloak role; administering a team does not grant it.

Ownership is explicit: a team owns a database and a database owns a bucket. That is the governance model.

03 · Platform

API first: the portals are just clients, and so is an agent

Admin API · :3000 · OpenAPI at /docs
GET POST DELETE/api/sessionAuth state, sign in, sign out
GET/api/overviewHealth and every resource
GET POST PATCH DELETE/api/teams[/{id}]Teams
GET POST PATCH DELETE/api/databases[/{id}[/name]]Create, move, rename, delete
GET/api/databases/{id}/connectionIceberg REST settings
GET POST PATCH DELETE/api/users[/{id}]Users, a role per team
DELETE/api/shares/{id}Revoke a data share
GET POST/api/identity/{accounts,users,links}Keycloak accounts
POST/api/users/{id}/identity[/…]Link, retry, reset password
GET/api/admin/explorer/{databases,contents}Browse every catalog
GET/api/{infrastructure,health}Live metrics, liveness
MCP/mcpThe same, as agent tools
User gateway · :3002 · OpenAPI at /docs
GET POST DELETE/api/sessionSign in, sign out
GET PATCH/api/{workspace,team,environment}My context, switch it
POST PATCH DELETE/api/databases[/{id}]Team administrators
GET/api/{contents,details}Tables, schema, snapshots
POST/api/preview100 rows max
POST DELETE/api/notebooks[/{id}]Start, stop marimo
GET POST PATCH DELETE/api/shares[/{id}[/rotate]]Outgoing shares
GET/api/{share-teams,received-shares}Targets, received
ANY · WS/workspaces/{id}/…Notebook proxy
MCP/mcpAgent tools

There is no private backdoor for the UI: every button is a fetch to one of these routes, and the OpenAPI schema is generated from the same code. One level down it is open APIs again: Iceberg REST on Polaris, S3 on RustFS and OIDC on Keycloak.

03 · Platform

Try every route in the browser

Swagger UI of the user workspace API
User portal :3002/docs · Swagger with your own session and team permissions · Try it out adds the write header for you
Demo · Sign in

One identity for everything

Keycloak sign-in page for the iceberg realm
Keycloak · realm iceberg · the admin portal, the user portal and Polaris all trust the same tokens
Demo · Administration

Databases: a catalog plus a bucket, per team

Admin portal databases page
Admin portal :3000 · six databases across three fictional grid operator teams, development and production
Demo · Administration

Teams and their members

Admin portal teams page Admin portal users page
Teams · a central place for ownership and membershipUsers · Keycloak accounts with a role in every team they belong to
↓ full screen
Admin portal · teams
Admin portal · users
Demo · Administration

Create a user: an account and a role per team

Create user dialog with username, name, email and a writer role for team outage-response
Admin portal · Create user · a new Keycloak account or link an existing one · pick the teams and a role in each · the temporary password is shown once
Demo · Administration

A role per team

Edit access dialog with a role select for every team of the user
Edit access · sander reads asset-management and writes grid-planning · each role reaches only the databases of that team
Demo · Administration

Creating a user creates a Keycloak account

Keycloak admin console listing the users of the iceberg realm
Keycloak admin console · accounts created by the portal · temporary password shown once · change forced at first sign-in
Demo · Administration

Catalog explorer and live infrastructure

Admin catalog explorer showing database, namespace and table Admin infrastructure page with service health
All catalogs, namespaces, tables and views · read only for the portal adminHealth, CPU, memory, bucket data · separate monitor service
↓ full screen
Admin portal · catalog explorer
Admin portal · infrastructure
Demo · Team workspace

The user portal: your teams, your databases

User portal welcome page User portal with active team, environment and database selected
User portal :3002 · sign in with the same Keycloak accountActive team with your role in it · environment · seven bundled example notebooks
↓ full screen
User portal · welcome
User portal · team, environment, database
Demo · Team workspace

Team administrators run their own databases

User portal Databases page with the create database form filled in
mila administers grid-planning · create, rename and delete with the name typed to confirm · an interrupted deletion can be resumed
Demo · Iceberg v3

Iceberg v3, written by DuckDB

CREATE TABLE lakehouse.iceberg_v3.sensor_events (
    event_id     BIGINT,
    device_id    VARCHAR,
    site         VARCHAR,
    location     GEOMETRY,
    measured_at  TIMESTAMP_NS,
    payload      VARIANT,
    synthetic    BOOLEAN
) WITH ('format-version' = 3);

ALTER TABLE … ADD COLUMN firmware VARCHAR DEFAULT 'v1.0';
UPDATE … SET firmware = 'v2.0' WHERE device_id = 'SENSOR-001';
DELETE FROM … WHERE payload.kind::VARCHAR = 'fault';
  • VARIANT holds a different structure per event, without a fixed schema.
  • TIMESTAMP_NS and GEOMETRY are native column types.
  • Column defaults mean that adding firmware rewrites none of the 480 rows.
  • Deletion vectors let updates and deletes mark row positions in a small Puffin file.
  • Row lineage keeps a stable ID per row, together with the commit that last changed it.
Plain SQL through the Polaris REST catalog, with the user's own permissions · GEOGRAPHY and UNKNOWN are not in DuckDB yet · PyIceberg does not read these v3 types yet
Demo · Catalog details

Inspect a table without opening a notebook

Table overview with current snapshot, format version and location Table schema tab with field IDs and types
Overview · snapshot, format version, location, statisticsSchema with Iceberg field IDs
↓ full screen
User portal · namespace with one Iceberg table
User portal · table overview
User portal · schema
User portal · the v3 table · format version 3, three delete files
Demo · Catalog details

Snapshots, branches and a preview at any snapshot

Snapshot history and branches of the table Preview of 100 rows at the selected snapshot
History and refs · main branch100 rows at a selected snapshot · DuckDB in a disposable process, with the user's own permissions
↓ full screen
User portal · snapshots and refs
User portal · preview at a snapshot
User portal · preview of the v3 table · geometry as text, VARIANT as JSON
Demo · Data sharing

Share with another team or someone outside, without a copy

01 · Team administrator

A team administrator picks tables and views of one database in the user portal and chooses the recipient: another team, an external party, or both. No platform administrator in the loop.

02 · Polaris

Access is granted per object: an explicit read grant for each selected table or view, rather than blanket permissions on the namespace or the catalog.

03 · Recipient

Members of the receiving team use their own accounts, in the portal and in notebooks. An external party uses any Iceberg REST client with its own client ID and secret.

Every recipient reads the same files, and the team can expire or revoke access at any time. There is still only one copy of the data.

Demo · Data sharing

Choose the recipient and the tables in one form

User portal data share form with another team selected as recipient and external sharing switched off User portal form for an external data share with a recipient, an expiry and a namespace tree with a selected table
mila administers grid-planning · the street load for outage-response, no credential neededThe same table for the municipality, with an expiry · credential shown once
↓ full screen
User portal · share with another team
User portal · share externally
User portal · credential for the external recipient · copy it or a ready DuckDB snippet · the secret in this capture has been replaced
Demo · Data sharing

What the receiving team sees

User portal Databases page of a team without databases, listing a database shared with it User portal catalog of a shared database, marked shared and read-only
noor in outage-response · no database of their own in development · one shared with the teamCatalog · only the shared objects · read-only, owner team named
↓ full screen
User portal · databases shared with your team
User portal · shared database in the catalog
marimo · notebook on the shared database · files stay in the receiving team's own workspace
Demo · Data sharing

Teams manage their own shares, the platform keeps oversight

User portal list of data shares with edit, secret and revoke actions Admin portal page listing every data share with its database, team, tables and expiry
User portal · one share per recipient · edit the selection, replace an external secret, revokeAdmin portal · every share across all teams · revoke only
↓ full screen
User portal · data shares of a database
Admin portal · data shares
Demo · Data sharing

What an external recipient can and cannot do

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "shared", type="rest",
    uri="https://catalog.example/api/catalog",
    warehouse="db-…",
    credential="<client-id>:<client-secret>",
    scope="PRINCIPAL_ROLE:ALL",
    **{"header.X-Iceberg-Access-Delegation":
       "vended-credentials"},
)

table = catalog.load_table(
    "synthetic.neighborhood_electricity")
table.scan().to_arrow()         # 26,880 rows

catalog.list_namespaces()       # Forbidden
catalog.list_tables("synthetic")  # Forbidden
catalog.load_table("synthetic.other")  # Forbidden
  • A share holds no list privilege, so the recipient loads only the shared objects, by full name.
  • Storage follows the grant: vended S3 credentials are read-only and stop at the prefix of the shared table.
  • A view is a definition rather than a filter, so the tables it reads are shared with it and readable in full.
  • Revoking is immediate for catalog tokens; storage credentials already handed out expire on their own.
  • Outside one machine this needs your own TLS proxy in front of Polaris and RustFS.
Every line on this slide is asserted against Polaris and RustFS by scripts/share_smoke.py.
Demo · AI agents

Your catalog, for an AI agent

claude mcp add --transport http \
  --client-id iceberg-mcp --callback-port 3010 \
  iceberg-user http://localhost:3002/mcp
list_databasesYour teams, your role in each, owned and shared databases
list_namespaces · list_tablesBrowse what your grants allow
describe_table · describe_viewSchema, partitioning, snapshots, branches, view SQL
preview_rowsUp to 100 rows, at any snapshot
create · rename · delete_databaseTeam administrators only · delete needs the exact name
  • The agent signs in through Keycloak as you: a public client with PKCE, no secret to store.
  • Every call runs with your Polaris grants, so the agent sees exactly what you see and nothing more.
  • A shared database stays read-only for the agent too.
  • Destructive tools are marked as such, and the agent is told to ask before using them.

Any MCP client works: Claude Code, an IDE, or your own agent. The protocol is open, just like Iceberg REST.

Demo · AI agents

And one for the platform administrator

claude mcp add --transport http \
  --client-id iceberg-mcp --callback-port 3010 \
  iceberg-admin http://localhost:3000/mcp
get_overview · browse_catalogHealth, every team, database, user and share
create · update · delete_teamTeams
create · rename · move · delete_databaseDatabases across all teams
create · link_user · update_user_accessKeycloak accounts and a role per team
list_shares · revoke_shareOversight on every data share
  • The platform-admin role is checked in the token on every call, not once per session.
  • Irreversible tools are marked as destructive; delete_database refuses without the database's exact name.
  • Connection details never include credentials.
  • A new user's one-time password lands in the agent transcript, so treat that transcript as a secret.

“Create a team for the heat transition, give it a development database and add Noor as a writer” takes one sentence instead of three screens.

03 · Platform

Security model, in one slide

  • The user portal, Polaris, the notebooks and MCP all work with that user's Keycloak token. No shared service account for data access.
  • Polaris vends short lived, table scoped S3 credentials; notebooks never see bucket root keys.
  • Each notebook session gets its own container and network: read-only root filesystem, dropped capabilities, CPU, memory and process limits, no Docker socket.
  • Revoking a membership stops the affected runtimes within 30 seconds.
  • A data share holds one read grant per table or view. A receiving team never gets administration; an external secret is shown once, never stored.
  • MCP accepts only tokens issued to the iceberg-mcp client, verified on every call: an agent never holds more than its user.

This build is meant for local development. HTTPS, persistent sessions and hostile-tenant isolation are on the roadmap, and the boundaries are documented.

04

What this means for us

Direction and ownership.

04 · Wrap up

As far as I'm concerned

  • Iceberg complements platforms like Snowflake. Use it where openness pays off: data that several teams, engines or partners read. Not every table needs to be an Iceberg table.
  • Combining data products is where an integrated platform wins. Joining many data products in one place with good performance is easier when storage, compute and catalog come from one vendor.
  • For Iceberg tables, engines are replaceable. My recommendation is to treat them as workload decisions and not as platform commitments.
  • Self-service only works with clear ownership. Teams create, own and share their databases, while the platform team owns the building blocks.
  • Agents use the same front door. They get the same identity and grants as the people they work for.
Thanks

Reach out.

Feel free to reach out if you want to run this yourself, or if you see a gap I missed. We'll take it from there.

Regards, Sander
github.com/sanderdw/iceberg-data-platform
Apache-2.0