Deutsch

huggingface

Downloads

186

GitHub-Sterne

0

Erweiterungs-Repository auf GitHub

Hugging-Face-Datensätze entdecken, abfragen, profilieren und deren Speicherbedarf schätzen

Maintainer: dentiny

Installation und Laden

INSTALL huggingface FROM community;
LOAD huggingface;

Beispiel

INSTALL huggingface FROM community;
LOAD huggingface;


FROM hf_files(
    'ibm/duorc',
    config = 'ParaphraseRC',
    split = 'train'
);

Über huggingface

Die Erweiterung huggingface arbeitet mit Parquet-Datensätzen, die auf dem Hugging Face Hub gehostet werden. Sie kann Parquet-Dateien entdecken, Datensätze mit Projektions- und Filter-Pushdown scannen, Parquet-Metadaten profilieren und den Speicherbedarf externer Blobs anhand einer begrenzten Stichprobe schätzen.

Die Erweiterung verwendet cache_httpfs für zwischengespeicherten Zugriff auf hf://-Pfade und unterstützt Hugging-Face-Anmeldedaten über den Secret Manager von DuckDB. Lance- Datensätze werden derzeit nicht unterstützt.

Hinzugefügte Funktionen

function_name function_type description comment examples
cache_httpfs_add_exclusion_regex scalar NULL NULL
cache_httpfs_cache_access_info_query table NULL NULL
cache_httpfs_cache_status_query table NULL NULL
cache_httpfs_cleanup_dead_temp scalar NULL NULL
cache_httpfs_clear_cache scalar NULL NULL
cache_httpfs_clear_cache_for_file scalar NULL NULL
cache_httpfs_clear_profile scalar NULL NULL
cache_httpfs_get_cache_config table NULL NULL
cache_httpfs_get_cache_filesystems table NULL NULL
cache_httpfs_get_cache_type table NULL NULL
cache_httpfs_get_data_cache_config table NULL NULL
cache_httpfs_get_file_handle_cache_config table NULL NULL
cache_httpfs_get_glob_cache_config table NULL NULL
cache_httpfs_get_metadata_cache_config table NULL NULL
cache_httpfs_get_ondisk_data_cache_size scalar NULL NULL
cache_httpfs_get_profile scalar NULL NULL
cache_httpfs_list_exclusion_regex table NULL NULL
cache_httpfs_list_registered_filesystems table NULL NULL
cache_httpfs_reset_exclusion_regex scalar NULL NULL
cache_httpfs_wrap_cache_filesystem scalar NULL NULL
hf_dataset_estimate table NULL NULL
hf_files table NULL NULL
hf_profile table NULL NULL
hf_scan table NULL NULL
huggingface_internal_blob_size scalar NULL NULL

Überladene Funktionen

Diese Erweiterung fügt keine Funktionsüberladungen hinzu.

Hinzugefügte Typen

Diese Erweiterung fügt keine Typen hinzu.

Hinzugefügte Einstellungen

name description input_type scope aliases
cache_httpfs_cache_block_size Block size for cache, applies to both in-memory cache filesystem and on-disk cache filesystem. It’s worth noting for on-disk filesystem, all existing cache files are invalidated after config update. UBIGINT GLOBAL []
cache_httpfs_cache_directories_config Advanced configuration for on-disk cache. It supports multiple directories, separated by semicolons (‘;’). Cache blocks will be evenly distributed under different directories deterministically.Between different runs, it’s expected to provide same cache directories, otherwise it’s not guaranteed cache files still exist and accessible.Overrides ‘cache_httpfs_cache_directory’ if set. VARCHAR GLOBAL []
cache_httpfs_cache_directory The disk cache directory that stores cached data VARCHAR GLOBAL []
cache_httpfs_clear_cache_on_write Whether to clear cache entries on write operations. When enabled, write operations will invalidate cached metadata, file handles, and glob entries for the modified file, which could be expensive.Disabling this can improve write performance when many cache entries exist, but may lead to stale cache reads. By default disabled. BOOLEAN GLOBAL []
cache_httpfs_disk_cache_reader_enable_memory_cache Whether enable process-wise read-through/write-through cache for disk cache reader. When enabled, local cache file will be accessed with direct IO. BOOLEAN GLOBAL []
cache_httpfs_disk_cache_reader_mem_cache_block_count Max number of cache blocks for the read-through/write-through cache for disk cache reader. UBIGINT GLOBAL []
cache_httpfs_disk_cache_reader_mem_cache_timeout_millisec Timeout in milliseconds for the read-through/write-through cache for disk cache reader. UBIGINT GLOBAL []
cache_httpfs_enable_cache_validation Whether to enable cache validation using version tag and last modification timestamp. When enabled, cache entries are validated against the current file version tag and modification timestamp to ensure cache consistency. By default disabled. BOOLEAN GLOBAL []
cache_httpfs_enable_file_handle_cache Whether file handle cache is enable for cache filesystem. By default enabled. BOOLEAN GLOBAL []
cache_httpfs_enable_glob_cache Whether glob cache is enable for cache filesystem. By default enabled. BOOLEAN GLOBAL []
cache_httpfs_enable_metadata_cache Whether metadata cache is enable for cache filesystem. By default enabled. BOOLEAN GLOBAL []
cache_httpfs_evict_policy Eviction policy for on-disk cache cache blocks. By default it’s creation timestamp based (‘creation_timestamp’), which deletes all cache blocks created earlier than threshold. Other supported policy include ‘lru_single_proc’ (LRU forsingle process access), which performs LRU-based eviction, mainly made single processusage. VARCHAR GLOBAL []
cache_httpfs_file_handle_cache_entry_size Max cache size for file handle cache. UBIGINT GLOBAL []
cache_httpfs_file_handle_cache_entry_timeout_millisec Cache entry timeout in milliseconds for file handle cache. UBIGINT GLOBAL []
cache_httpfs_glob_cache_entry_size Max cache size for glob cache. UBIGINT GLOBAL []
cache_httpfs_glob_cache_entry_timeout_millisec Cache entry timeout in milliseconds for glob cache. UBIGINT GLOBAL []
cache_httpfs_ignore_sigpipe Whether to ignore SIGPIPE for the extension. By default not ignored. Once ignored, it cannot be reverted. BOOLEAN GLOBAL []
cache_httpfs_in_mem_cache_block_timeout_millisec Data block cache entry timeout in milliseconds. UBIGINT GLOBAL []
cache_httpfs_in_mem_cache_storage Storage backend for in-memory data block cache. ‘extension’ (default) uses an extension-managed LRU bounded by cache_httpfs_max_in_mem_cache_block_count and cache_httpfs_in_mem_cache_block_timeout_millisec. ‘object_cache’ parks blocks in DuckDB’s per-instance ObjectCache. Must be set before any cache access for the change to take effect. VARCHAR GLOBAL []
cache_httpfs_max_fanout_subrequest Cached httpfs performs parallel request by splittng them into small request, with request size decided by config [cache_httpfs_cache_block_size]. The setting limits the maximum request to issue for a single filesystem read request. 0 means no limit, by default we set no limit. BIGINT GLOBAL []
cache_httpfs_max_in_mem_cache_block_count Max in-memory cache block count for in-memory caches for all cache filesystems, so users are able to configure the maximum memory consumption. It’s worth noting it should be set only once before all filesystem access, otherwise there’s no affect. UBIGINT GLOBAL []
cache_httpfs_metadata_cache_entry_size Max cache size for metadata LRU cache. UBIGINT GLOBAL []
cache_httpfs_metadata_cache_entry_timeout_millisec Cache entry timeout in milliseconds for metadata LRU cache. UBIGINT GLOBAL []
cache_httpfs_min_disk_bytes_for_cache Min number of bytes on disk for the cache filesystem to enable on-disk cache; if left bytes is less than the threshold, LRU based cache file eviction will be performed.By default, 5% disk space will be reserved for other usage. When min disk bytes specified with a positive value, the default value will be overriden. UBIGINT GLOBAL []
cache_httpfs_parallel_read_mode Thread model used for parallel cache read subrequests. ‘internal_thread_pool’ (default) spawns a private thread pool per read, keeping behaviour identical to the original implementation. ‘duckdb_task_scheduler’ submits tasks to DuckDB’s global TaskScheduler so that ‘SET threads = N’ caps total parallelism (including cache reads) in one place. VARCHAR GLOBAL []
cache_httpfs_profile_type Profiling type for cached filesystem. There are two options available: noop and temp. temp option stores the latest IO operation profiling result, which potentially suffers concurrent updates. VARCHAR GLOBAL []
cache_httpfs_type Type for cached filesystem. Currently there’re two types available, one is in_mem, another is on_disk. By default we use on-disk cache. Set to noop to disable, which behaves exactly same as httpfs extension. VARCHAR GLOBAL []