boilstream
Sicherer entfernter Secrets-Speicher mit OPAQUE-PAKE-Authentifizierung, E-Mail-/Passwort-Registrierung, MFA-Anmeldung und automatischem Ducklake-Mounting
Maintainer: dforsber
Installation und Laden
INSTALL boilstream FROM community;LOAD boilstream;Beispiel
-- Setup: Install and load boilstream extensionINSTALL httpfs; LOAD httpfs;INSTALL boilstream FROM community;LOAD boilstream;
-- Option 1: Login with email/password/MFA (establishes OPAQUE session)-- NOTE: Remember to clean up your ~/.duckdb/historyPRAGMA boilstream_login('https://api.example.com/user@example.com', 'password', '123456');┌────────────────────────┬─────────────────────┬────────────────────┐│ status │ expires_at │ ducklakes_attached │├────────────────────────┼─────────────────────┼────────────────────┤│ Session token obtained │ 2025-11-21 10:30:00 │ 2 │└────────────────────────┴─────────────────────┴────────────────────┘
-- Option 2: Authenticate with bootstrap token (from SSO/web UI)PRAGMA boilstream_bootstrap_session('https://api.example.com/secrets:TOKEN');┌────────────────────────┬─────────────────────┬────────────────────┐│ status │ expires_at │ ducklakes_attached │├────────────────────────┼─────────────────────┼────────────────────┤│ Session token obtained │ 2025-11-21 10:30:00 │ 2 │└────────────────────────┴─────────────────────┴────────────────────┘
-- List all available ducklakesFROM boilstream_ducklakes();┌──────────────┬──────────────┬─────────────────┐│ catalog_id │ catalog_name │ description │├──────────────┼──────────────┼─────────────────┤│ 5a805647-... │ my_analytics │ Sales analytics │└──────────────┴──────────────┴─────────────────┘
-- List all cached secrets with expirationFROM boilstream_secrets();┌─────────────────────┬──────────┬─────────────────────┐│ name │ type │ expires_at │├─────────────────────┼──────────┼─────────────────────┤│ my_analytics │ ducklake │ 2026-11-19 20:00:00 ││ my_analytics_s3 │ s3 │ 2026-11-19 20:00:00 ││ my_analytics_pg │ postgres │ 2026-11-19 20:00:00 │└─────────────────────┴──────────┴─────────────────────┘
-- Create new ducklakePRAGMA boilstream_create_ducklake('new_catalog', 'My new ducklake');
-- Query ducklake (automatically attached)USE my_analytics;SHOW TABLES;
-- Create persistent secret onto BoilStream serverCREATE PERSISTENT SECRET s3_minio IN boilstream ( TYPE S3, KEY_ID 'minioadmin', SECRET 'minioadmin', REGION 'eu-west-1', ENDPOINT 'localhost:9000', USE_SSL false, URL_STYLE 'path', SCOPE 's3://ingestion-data/');Über boilstream
Die Boilstream-Erweiterung stellt entfernten Secrets-Speicher für mandantenfähige DuckDB-Deployments bereit.
Funktionen:
- 🔐 OPAQUE-PAKE-Authentifizierung - Passwortlose Authentifizierung (IETF RFC 9807)
- 🔒 Ende-zu-Ende-Verschlüsselung - AES-256-GCM/ChaCha20-Poly1305 für Secrets und Antworten
- 🔑 Kryptographische Schlüsselableitung - HKDF-SHA256 mit Domain Separation
- 🛡️ Defense in Depth - HMAC-Prüfung vor der AEAD-Entschlüsselung
- 🎯 Sitzungsschutz - Lock-Step-Sequenzprotokoll verhindert Hijacking
- 🗄️ Ducklake-Unterstützung - Automatisches Anhängen mandantenfähiger Datenkataloge
- 🌐 SSO-Integration - Bootstrap-Token-Austausch für Webumgebungen
Sicherheit:
- Facebooks geprüfte OPAQUE-Implementierung (Rust + C++ FFI)
- Schützt vor MITM-Angriffen, selbst wenn TLS kompromittiert ist
- Einmalige Bootstrap-Tokens mit anonymer Authentifizierung
- Sitzungswiederaufnahme mit verschlüsselten Refresh-Tokens
- Umfassende Testvektoren und Sicherheitsspezifikation
API-Funktionen:
PRAGMA boilstream_register_user(url_with_email, password)- Neuen Benutzer mit MFA registrieren (zeigt QR-Code)PRAGMA boilstream_verify_mfa(totp_code)- Registrierung abschließen und Backup-Codes erhaltenPRAGMA boilstream_login(url_with_email, password, mfa_code)- Anmeldung mit E-Mail/Passwort/MFAPRAGMA boilstream_bootstrap_session(url_with_token)- Authentifizierung mit Bootstrap-Tokenboilstream_ducklakes()- Verfügbare Ducklakes auflistenboilstream_secrets()- Zwischengespeicherte Secrets mit Ablaufzeit auflistenboilstream_buckets()- Verfügbare Storage-Buckets auflistenPRAGMA boilstream_create_ducklake(name, description)- Neuen Ducklake anlegenPRAGMA boilstream_help- Alle verfügbaren Befehle anzeigen
Voraussetzungen:
- httpfs-Erweiterung (wird automatisch geladen)
- postgres_scanner-Erweiterung (wird automatisch geladen)
- ducklake-Erweiterung (wird automatisch geladen)
- Kompatibler REST-API-Server (siehe API-Spezifikation)
Ausführliche Dokumentation, Sicherheitsspezifikation und Einrichtungsanleitung finden Sie im Erweiterungs-Repository.
Hinzugefügte Funktionen
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| boilstream_bootstrap_session | pragma | NULL | NULL | |
| boilstream_buckets | table | NULL | NULL | |
| boilstream_create_ducklake | pragma | NULL | NULL | |
| boilstream_ducklakes | table | NULL | NULL | |
| boilstream_help | pragma | NULL | NULL | |
| boilstream_login | pragma | NULL | NULL | |
| boilstream_quack_authn | scalar | NULL | NULL | |
| boilstream_quack_authz | scalar | NULL | NULL | |
| boilstream_quack_bind_session | scalar | NULL | NULL | |
| boilstream_quack_debug_session_count | scalar | NULL | NULL | |
| boilstream_register_user | pragma | NULL | NULL | |
| boilstream_secrets | table | NULL | NULL | |
| boilstream_verify_mfa | pragma | NULL | NULL | |
| ducklake_add_data_files | table | NULL | NULL | |
| ducklake_cleanup_old_files | table | NULL | NULL | |
| ducklake_commit | table | NULL | NULL | |
| ducklake_current_snapshot | table | NULL | NULL | |
| ducklake_delete_orphaned_files | table | NULL | NULL | |
| ducklake_expire_snapshots | table | NULL | NULL | |
| ducklake_flush_inlined_data | table | NULL | NULL | |
| ducklake_last_committed_snapshot | table | NULL | NULL | |
| ducklake_list_files | table | NULL | NULL | |
| ducklake_merge_adjacent_files | table | NULL | NULL | |
| ducklake_options | table | NULL | NULL | |
| ducklake_rewrite_data_files | table | NULL | NULL | |
| ducklake_scan | table | NULL | NULL | |
| ducklake_set_commit_message | table | NULL | NULL | |
| ducklake_set_option | table | NULL | NULL | |
| ducklake_settings | table | NULL | NULL | |
| ducklake_snapshots | table | NULL | NULL | |
| ducklake_table_changes | table_macro | NULL | NULL | |
| ducklake_table_deletions | table | NULL | NULL | |
| ducklake_table_info | table | NULL | NULL | |
| ducklake_table_insertions | table | NULL | NULL | |
| murmur3_32 | scalar | NULL | NULL | |
| pg_clear_cache | table | NULL | NULL | |
| postgres_attach | table | NULL | NULL | |
| postgres_configure_pool | table | NULL | NULL | |
| postgres_execute | table | NULL | NULL | |
| postgres_hstore_get | scalar | NULL | NULL | |
| postgres_hstore_to_json | scalar | NULL | NULL | |
| postgres_query | table | NULL | NULL | |
| postgres_scan | table | NULL | NULL | |
| postgres_scan_pushdown | table | NULL | NULL | |
| read_postgres_binary | table | NULL | NULL |
Überladene Funktionen
Diese Erweiterung fügt keine Funktionsüberladungen hinzu.
Hinzugefügte Typen
Diese Erweiterung fügt keine Typen hinzu.
Hinzugefügte Einstellungen
| name | description | input_type | scope | aliases |
|---|---|---|---|---|
| ducklake_default_data_inlining_row_limit | Default row limit for data inlining (0 disables inlining) | UBIGINT | GLOBAL | [] |
| ducklake_max_retry_count | The maximum amount of retry attempts for a ducklake transaction | UBIGINT | GLOBAL | [] |
| ducklake_retry_backoff | Backoff factor for exponentially increasing retry wait time | DOUBLE | GLOBAL | [] |
| ducklake_retry_wait_ms | Time between retries | UBIGINT | GLOBAL | [] |
| ducklake_target_file_size | Target file size for insertion and compaction | VARCHAR | GLOBAL | [] |
| ducklake_write_deletion_vectors | [EXPERIMENTAL] Write Iceberg V3 deletion vectors (puffin) instead of positional delete files (parquet) | BOOLEAN | GLOBAL | [] |
| pg_array_as_varchar | Read Postgres arrays as varchar - enables reading mixed dimensional arrays | BOOLEAN | GLOBAL | [] |
| pg_connection_cache | Whether or not to use the connection pooling. This option is deprecated, instead to disable the connection pooling use “SET pg_pool_max_connections=0” for newly attached Postgres databases and “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, max_connections=0)” for already attached Postgres databases. | BOOLEAN | GLOBAL | [] |
| pg_connection_limit | The maximum amount of concurrent Postgres connections. This option is deprecated, instead use “SET pg_pool_max_connections = 42” for newly attached Postgres databases and “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, max_connections=42)” for already attached Postgres databases. | UBIGINT | GLOBAL | [] |
| pg_debug_show_queries | DEBUG SETTING: print all queries sent to Postgres to stdout | BOOLEAN | GLOBAL | [] |
| pg_experimental_filter_pushdown | Whether or not to use filter pushdown | BOOLEAN | GLOBAL | [] |
| pg_idle_in_transaction_timeout_millis | Postgres idle in transaction timeout in milliseconds to set on scan connections | UINTEGER | GLOBAL | [] |
| pg_null_byte_replacement | When writing NULL bytes to Postgres, replace them with the given character | VARCHAR | GLOBAL | [] |
| pg_oauth_token | OAuth bearer token for PostgreSQL OAUTHBEARER authentication. Takes priority over the PGOAUTHTOKEN environment variable | VARCHAR | GLOBAL | [] |
| pg_pages_per_task | The amount of pages per task | UBIGINT | GLOBAL | [] |
| pg_pool_acquire_mode | How to acquire connections from the pool: ‘force’ (always connect, ignore pool limit), ‘wait’ (block until available), ‘try’ (fail immediately if unavailable) (default: force) | VARCHAR | GLOBAL | [] |
| pg_pool_enable_reaper_thread | Whether to enable the connection pool reaper thread, that periodically scans the pool to check the ‘max_lifetime_millis’ and ‘idle_timeout_millis’ and closes the connection which exceed the specified values. Either ‘max_lifetime_millis’ or ‘idle_timeout_millis’ must be set to a non-zero value for this option to be effective. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, enable_reaper_thread=TRUE)” | BOOLEAN | GLOBAL | [] |
| pg_pool_enable_thread_local_cache | Whether to enable the connection caching in thread-local cache. Such connections are getting pinned to the threads and are not made available to other threads, while still taking the place in the pool. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, enable_thread_local_cache=FALSE)” | BOOLEAN | GLOBAL | [] |
| pg_pool_health_check_query | The query that is used to check that the connection is healthy. Setting this option to an empty string disables the health check. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, health_check_query=SELECT 42)” | VARCHAR | GLOBAL | [] |
| pg_pool_idle_timeout_millis | Maximum number of milliseconds the connection can be kept idle in the pool. This number is checked when the connection is taken from the pool. When the connection pool reaper thread is enabled (‘pg_pool_enable_reaper_thread’ option), then this number is checked in background periodically. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, idle_timeout_millis=300000)” | UBIGINT | GLOBAL | [] |
| pg_pool_max_connections | Maximum number of connections that are allowed to be cached in a connection pool for each attached Postgres database. This number can be temporary exceeded when parallel scans are used. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, max_connections=42)” | UBIGINT | GLOBAL | [] |
| pg_pool_max_lifetime_millis | Maximum number of milliseconds the connection can be kept open. This number is checked when the connection is taken from the pool and returned to the pool. When the connection pool reaper thread is enabled (‘pg_pool_enable_reaper_thread’ option), then this number is checked in background periodically. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, max_lifetime_millis=600000)” | UBIGINT | GLOBAL | [] |
| pg_pool_wait_timeout_millis | Maximum number of milliseconds to wait when acquiring a connection from a pool where all available connections are already taken. This option only applies to newly attached Postgres databases, to configure a database that is already attached use “FROM postgres_configure_pool(catalog_name=‘my_attached_postgres_db’, wait_timeout_millis=60000)” | UBIGINT | GLOBAL | [] |
| pg_staleness_query | Custom query used in place of the default table staleness query when pg_staleness_query_enabled resolves to true. Must contain a ${SCHEMA} placeholder and return at least 3 columns (identity, name, revision marker). Empty (default) uses the built-in pg_class/xmin query. | VARCHAR | GLOBAL | [] |
| pg_staleness_query_enabled | Whether or not the table catalog cache checks Postgres for external DDL changes before serving a cache hit. Defaults to the opposite of pg_use_information_schema_introspection when not explicitly set (off for pg protocol compatible databases that may not support the underlying query, on for Postgres). | BOOLEAN | GLOBAL | [] |
| pg_statement_timeout_millis | Postgres statement timeout in milliseconds to set on scan connections | UINTEGER | GLOBAL | [] |
| pg_use_binary_copy | Whether or not to use BINARY copy to read data | BOOLEAN | GLOBAL | [] |
| pg_use_ctid_scan | Whether or not to parallelize scanning using table ctids | BOOLEAN | GLOBAL | [] |
| pg_use_information_schema_introspection | Use SQL-standard information_schema views for ATTACH-time schema discovery instead of pg_catalog.For compatibility with pg protocol compatible databases like spanner, cockroach, redshift, … | BOOLEAN | GLOBAL | [] |
| pg_use_text_protocol | Whether or not to use TEXT protocol to read data. This is slower, but provides better compatibility with non-Postgres systems | BOOLEAN | GLOBAL | [] |