boilstream

Sicherer entfernter Secrets-Speicher mit OPAQUE-PAKE-Authentifizierung, E-Mail-/Passwort-Registrierung, MFA-Anmeldung und automatischem Ducklake-Mounting

Maintainer: dforsber

Installation und Laden

INSTALL boilstream FROM community;
LOAD boilstream;

Beispiel

-- Setup: Install and load boilstream extension
INSTALL httpfs; LOAD httpfs;
INSTALL boilstream FROM community;
LOAD boilstream;
-- Option 1: Login with email/password/MFA (establishes OPAQUE session)
-- NOTE: Remember to clean up your ~/.duckdb/history
PRAGMA boilstream_login('https://api.example.com/user@example.com', 'password', '123456');
┌────────────────────────┬─────────────────────┬────────────────────┐
status │ expires_at │ ducklakes_attached │
├────────────────────────┼─────────────────────┼────────────────────┤
Session token obtained │ 2025-11-21 10:30:002
└────────────────────────┴─────────────────────┴────────────────────┘
-- Option 2: Authenticate with bootstrap token (from SSO/web UI)
PRAGMA boilstream_bootstrap_session('https://api.example.com/secrets:TOKEN');
┌────────────────────────┬─────────────────────┬────────────────────┐
status │ expires_at │ ducklakes_attached │
├────────────────────────┼─────────────────────┼────────────────────┤
Session token obtained │ 2025-11-21 10:30:002
└────────────────────────┴─────────────────────┴────────────────────┘
-- List all available ducklakes
FROM boilstream_ducklakes();
┌──────────────┬──────────────┬─────────────────┐
│ catalog_id │ catalog_name │ description
├──────────────┼──────────────┼─────────────────┤
│ 5a805647-... │ my_analytics │ Sales analytics │
└──────────────┴──────────────┴─────────────────┘
-- List all cached secrets with expiration
FROM boilstream_secrets();
┌─────────────────────┬──────────┬─────────────────────┐
nametype │ expires_at │
├─────────────────────┼──────────┼─────────────────────┤
│ my_analytics │ ducklake │ 2026-11-19 20:00:00
│ my_analytics_s3 │ s3 │ 2026-11-19 20:00:00
│ my_analytics_pg │ postgres │ 2026-11-19 20:00:00
└─────────────────────┴──────────┴─────────────────────┘
-- Create new ducklake
PRAGMA boilstream_create_ducklake('new_catalog', 'My new ducklake');
-- Query ducklake (automatically attached)
USE my_analytics;
SHOW TABLES;
-- Create persistent secret onto BoilStream server
CREATE PERSISTENT SECRET s3_minio IN boilstream (
TYPE S3,
KEY_ID 'minioadmin',
SECRET 'minioadmin',
REGION 'eu-west-1',
ENDPOINT 'localhost:9000',
USE_SSL false,
URL_STYLE 'path',
SCOPE 's3://ingestion-data/'
);

Über boilstream

Die Boilstream-Erweiterung stellt entfernten Secrets-Speicher für mandantenfähige DuckDB-Deployments bereit.

Funktionen:

  • 🔐 OPAQUE-PAKE-Authentifizierung - Passwortlose Authentifizierung (IETF RFC 9807)
  • 🔒 Ende-zu-Ende-Verschlüsselung - AES-256-GCM/ChaCha20-Poly1305 für Secrets und Antworten
  • 🔑 Kryptographische Schlüsselableitung - HKDF-SHA256 mit Domain Separation
  • 🛡️ Defense in Depth - HMAC-Prüfung vor der AEAD-Entschlüsselung
  • 🎯 Sitzungsschutz - Lock-Step-Sequenzprotokoll verhindert Hijacking
  • 🗄️ Ducklake-Unterstützung - Automatisches Anhängen mandantenfähiger Datenkataloge
  • 🌐 SSO-Integration - Bootstrap-Token-Austausch für Webumgebungen

Sicherheit:

  • Facebooks geprüfte OPAQUE-Implementierung (Rust + C++ FFI)
  • Schützt vor MITM-Angriffen, selbst wenn TLS kompromittiert ist
  • Einmalige Bootstrap-Tokens mit anonymer Authentifizierung
  • Sitzungswiederaufnahme mit verschlüsselten Refresh-Tokens
  • Umfassende Testvektoren und Sicherheitsspezifikation

API-Funktionen:

  • PRAGMA boilstream_register_user(url_with_email, password) - Neuen Benutzer mit MFA registrieren (zeigt QR-Code)
  • PRAGMA boilstream_verify_mfa(totp_code) - Registrierung abschließen und Backup-Codes erhalten
  • PRAGMA boilstream_login(url_with_email, password, mfa_code) - Anmeldung mit E-Mail/Passwort/MFA
  • PRAGMA boilstream_bootstrap_session(url_with_token) - Authentifizierung mit Bootstrap-Token
  • boilstream_ducklakes() - Verfügbare Ducklakes auflisten
  • boilstream_secrets() - Zwischengespeicherte Secrets mit Ablaufzeit auflisten
  • boilstream_buckets() - Verfügbare Storage-Buckets auflisten
  • PRAGMA boilstream_create_ducklake(name, description) - Neuen Ducklake anlegen
  • PRAGMA boilstream_help - Alle verfügbaren Befehle anzeigen

Voraussetzungen:

  • httpfs-Erweiterung (wird automatisch geladen)
  • postgres_scanner-Erweiterung (wird automatisch geladen)
  • ducklake-Erweiterung (wird automatisch geladen)
  • Kompatibler REST-API-Server (siehe API-Spezifikation)

Ausführliche Dokumentation, Sicherheitsspezifikation und Einrichtungsanleitung finden Sie im Erweiterungs-Repository.

Hinzugefügte Funktionen

function_name function_type description comment examples
boilstream_bootstrap_session pragma NULL NULL
boilstream_buckets table NULL NULL
boilstream_create_ducklake pragma NULL NULL
boilstream_ducklakes table NULL NULL
boilstream_help pragma NULL NULL
boilstream_login pragma NULL NULL
boilstream_quack_authn scalar NULL NULL
boilstream_quack_authz scalar NULL NULL
boilstream_quack_bind_session scalar NULL NULL
boilstream_quack_debug_session_count scalar NULL NULL
boilstream_register_user pragma NULL NULL
boilstream_secrets table NULL NULL
boilstream_verify_mfa pragma NULL NULL
ducklake_add_data_files table NULL NULL
ducklake_cleanup_old_files table NULL NULL
ducklake_commit table NULL NULL
ducklake_current_snapshot table NULL NULL
ducklake_delete_orphaned_files table NULL NULL
ducklake_expire_snapshots table NULL NULL
ducklake_flush_inlined_data table NULL NULL
ducklake_last_committed_snapshot table NULL NULL
ducklake_list_files table NULL NULL
ducklake_merge_adjacent_files table NULL NULL
ducklake_options table NULL NULL
ducklake_rewrite_data_files table NULL NULL
ducklake_scan table NULL NULL
ducklake_set_commit_message table NULL NULL
ducklake_set_option table NULL NULL
ducklake_settings table NULL NULL
ducklake_snapshots table NULL NULL
ducklake_table_changes table_macro NULL NULL
ducklake_table_deletions table NULL NULL
ducklake_table_info table NULL NULL
ducklake_table_insertions table NULL NULL
murmur3_32 scalar NULL NULL
pg_clear_cache table NULL NULL
postgres_attach table NULL NULL
postgres_configure_pool table NULL NULL
postgres_execute table NULL NULL
postgres_hstore_get scalar NULL NULL
postgres_hstore_to_json scalar NULL NULL
postgres_query table NULL NULL
postgres_scan table NULL NULL
postgres_scan_pushdown table NULL NULL
read_postgres_binary table NULL NULL

Überladene Funktionen

Diese Erweiterung fügt keine Funktionsüberladungen hinzu.

Hinzugefügte Typen

Diese Erweiterung fügt keine Typen hinzu.

Hinzugefügte Einstellungen

name description input_type scope aliases
ducklake_default_data_inlining_row_limit Default row limit for data inlining (0 disables inlining) UBIGINT GLOBAL []
ducklake_max_retry_count The maximum amount of retry attempts for a ducklake transaction UBIGINT GLOBAL []
ducklake_retry_backoff Backoff factor for exponentially increasing retry wait time DOUBLE GLOBAL []
ducklake_retry_wait_ms Time between retries UBIGINT GLOBAL []
ducklake_target_file_size Target file size for insertion and compaction VARCHAR GLOBAL []
ducklake_write_deletion_vectors [EXPERIMENTAL] Write Iceberg V3 deletion vectors (puffin) instead of positional delete files (parquet) BOOLEAN GLOBAL []
pg_array_as_varchar Read Postgres arrays as varchar - enables reading mixed dimensional arrays BOOLEAN GLOBAL []
pg_connection_cache Whether or not to use the connection pooling. This option is deprecated, instead to disable the connection pooling use “SET pg_pool_max_connections=0” for newly attached Postgres databases and “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, max_connections=0)” for already attached Postgres databases. BOOLEAN GLOBAL []
pg_connection_limit The maximum amount of concurrent Postgres connections. This option is deprecated, instead use “SET pg_pool_max_connections = 42” for newly attached Postgres databases and “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, max_connections=42)” for already attached Postgres databases. UBIGINT GLOBAL []
pg_debug_show_queries DEBUG SETTING: print all queries sent to Postgres to stdout BOOLEAN GLOBAL []
pg_experimental_filter_pushdown Whether or not to use filter pushdown BOOLEAN GLOBAL []
pg_idle_in_transaction_timeout_millis Postgres idle in transaction timeout in milliseconds to set on scan connections UINTEGER GLOBAL []
pg_null_byte_replacement When writing NULL bytes to Postgres, replace them with the given character VARCHAR GLOBAL []
pg_oauth_token OAuth bearer token for PostgreSQL OAUTHBEARER authentication. Takes priority over the PGOAUTHTOKEN environment variable VARCHAR GLOBAL []
pg_pages_per_task The amount of pages per task UBIGINT GLOBAL []
pg_pool_acquire_mode How to acquire connections from the pool: ‘force’ (always connect, ignore pool limit), ‘wait’ (block until available), ‘try’ (fail immediately if unavailable) (default: force) VARCHAR GLOBAL []
pg_pool_enable_reaper_thread Whether to enable the connection pool reaper thread, that periodically scans the pool to check the ‘max_lifetime_millis’ and ‘idle_timeout_millis’ and closes the connection which exceed the specified values. Either ‘max_lifetime_millis’ or ‘idle_timeout_millis’ must be set to a non-zero value for this option to be effective. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, enable_reaper_thread=TRUE)” BOOLEAN GLOBAL []
pg_pool_enable_thread_local_cache Whether to enable the connection caching in thread-local cache. Such connections are getting pinned to the threads and are not made available to other threads, while still taking the place in the pool. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, enable_thread_local_cache=FALSE)” BOOLEAN GLOBAL []
pg_pool_health_check_query The query that is used to check that the connection is healthy. Setting this option to an empty string disables the health check. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, health_check_query=SELECT 42)” VARCHAR GLOBAL []
pg_pool_idle_timeout_millis Maximum number of milliseconds the connection can be kept idle in the pool. This number is checked when the connection is taken from the pool. When the connection pool reaper thread is enabled (‘pg_pool_enable_reaper_thread’ option), then this number is checked in background periodically. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, idle_timeout_millis=300000)” UBIGINT GLOBAL []
pg_pool_max_connections Maximum number of connections that are allowed to be cached in a connection pool for each attached Postgres database. This number can be temporary exceeded when parallel scans are used. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, max_connections=42)” UBIGINT GLOBAL []
pg_pool_max_lifetime_millis Maximum number of milliseconds the connection can be kept open. This number is checked when the connection is taken from the pool and returned to the pool. When the connection pool reaper thread is enabled (‘pg_pool_enable_reaper_thread’ option), then this number is checked in background periodically. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, max_lifetime_millis=600000)” UBIGINT GLOBAL []
pg_pool_wait_timeout_millis Maximum number of milliseconds to wait when acquiring a connection from a pool where all available connections are already taken. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, wait_timeout_millis=60000)” UBIGINT GLOBAL []
pg_staleness_query Custom query used in place of the default table staleness query when pg_staleness_query_enabled resolves to true. Must contain a ${SCHEMA} placeholder and return at least 3 columns (identity, name, revision marker). Empty (default) uses the built-in pg_class/xmin query. VARCHAR GLOBAL []
pg_staleness_query_enabled Whether or not the table catalog cache checks Postgres for external DDL changes before serving a cache hit. Defaults to the opposite of pg_use_information_schema_introspection when not explicitly set (off for pg protocol compatible databases that may not support the underlying query, on for Postgres). BOOLEAN GLOBAL []
pg_statement_timeout_millis Postgres statement timeout in milliseconds to set on scan connections UINTEGER GLOBAL []
pg_use_binary_copy Whether or not to use BINARY copy to read data BOOLEAN GLOBAL []
pg_use_ctid_scan Whether or not to parallelize scanning using table ctids BOOLEAN GLOBAL []
pg_use_information_schema_introspection Use SQL-standard information_schema views for ATTACH-time schema discovery instead of pg_catalog.For compatibility with pg protocol compatible databases like spanner, cockroach, redshift, … BOOLEAN GLOBAL []
pg_use_text_protocol Whether or not to use TEXT protocol to read data. This is slower, but provides better compatibility with non-Postgres systems BOOLEAN GLOBAL []