Initial Release
Load any public Hugging Face dataset into
InterSystems IRIS as a real SQL table — with one SQL statement.
SELECT IrisHF.API_ImportSimple('stanfordnlp/imdb');
{"status":"ok","dataset":"stanfordnlp/imdb","config":"plain_text","split":"train",
"table":"HF_stanfordnlp_imdb_plain_text_train","rowsAvailable":25000,
"rowsImported":25000,"columns":2,"truncatedCells":0,"failedRows":0,"elapsedSeconds":37}
The importer reads the dataset schema from the Hugging Face
datasets-server API, infers IRIS
column types, creates the table, and streams the rows in pages of 100. It is
written entirely in ObjectScript: no Python, no pip, no external libraries.
Hugging Face hosts hundreds of thousands of datasets, but getting one into IRIS
normally means writing a throw-away Python script, guessing the schema, and
handling pagination yourself. This project turns that into a single call that
works from SQL, from ObjectScript, or from a terminal.
https://datasets-server.huggingface.co. The free HFHF_TOKENgit clone https://github.com/foxchengyu/iris-hf-importer.git
cd iris-hf-importer
docker compose up -d --build
IRIS starts on http://localhost:52773/csp/sys/UtilHome.csp (_SYSTEM / SYS).
Run the demo import (falls back to a bundled sample CSV when the HF API is rate-limited):
docker compose exec iris iris session IRIS < docker/demo.script
With a working connection the demo imports 300 rows of stanfordnlp/imdb; when
the API returns HTTP 429 it imports the bundled demo/sample.csv instead.
SELECT IrisHF.API_ImportSimple('stanfordnlp/imdb')
Import src/cls with Management Portal → System Explorer → Classes →
Import, or from a terminal:
Do $SYSTEM.OBJ.LoadDir("/src/cls", "ck")
Run any of the entry points below.
The classes are installed in the namespace you import them into. The examples
useUSER.
-- one line, defaults for config/split, all rows SELECT IrisHF.API_ImportSimple('stanfordnlp/imdb');-- full control: dataset, target table, config, split, max rows SELECT IrisHF.API_ImportDataset('stanfordnlp/imdb', 'SQLUser.Imdb', 'plain_text', 'train', 1000);
-- what has been imported so far SELECT IrisHF.API_ListImports();
API_ImportSimple and API_ImportDataset return a JSON summary as a string, so
you can read the result in any SQL client.
Do ##class(IrisHF.CLI).Demo() ; 300 rows of imdb
Do ##class(IrisHF.CLI).Import("stanfordnlp/imdb") ; everything
Do ##class(IrisHF.CLI).Import("stanfordnlp/imdb", "SQLUser.MyTable", 500)
Or call the importer directly and inspect the returned object:
Set stats = ##class(IrisHF.Importer).Import("stanfordnlp/imdb", "", "", "", 100)
Write stats.%ToJSON()
SELECT TOP 5 * FROM HF_stanfordnlp_imdb_plain_text_train
Because it is a normal table, everything else works too: indices, SELECT
aggregations, BI tools, Embedded Python, Interoperability productions.
| Argument | Meaning | Default |
|---|---|---|
dataset |
Dataset id, owner/name |
required |
targetTable |
Target table, optionally Schema.Table |
derived from dataset |
config |
Dataset configuration | default, else the first one |
split |
Dataset split | train, else the first one |
maxRows |
Row limit, 0 = all |
0 |
The default table name is HF_<owner>_<name>_<config>_<split>, sanitised to a
valid IRIS identifier.
Warning: an import always drops and recreates the target table.
Re-running an import replaces the previous data, and pointingtargetTable
at an existing table overwrites it.
Set HF_TOKEN in the environment (or pass a token to IrisHF.Client). With
Docker:
HF_TOKEN=hf_xxx docker compose up -d --build
Outbound HTTPS must reach datasets-server.huggingface.co. Behind a corporate
proxy, set HF_PROXY to the proxy address; the client opens an HTTP CONNECT
tunnel and negotiates TLS through it:
HF_PROXY=http://proxy.example.com:8080 docker compose up -d --build
To use a mirror of the datasets-server API instead, set HF_ENDPOINT:
HF_ENDPOINT=https://datasets-server.internal.example.com docker compose up -d --build
Both are read from the environment by IrisHF.Client. In ObjectScript, set the
Proxy and Endpoint properties instead.
IRIS needs a client SSL configuration for every HTTPS call. Some images ship
without one, which makes requests fail with error #6159. The importer looks
for a configuration and creates IrisHFTLS on first use when none exists. The
created configuration is a client configuration that verifies the server
certificate against the operating system trust store
(CAFile=%OSCertificateStore, VerifyPeer=1). To use a different one, set the
SSLConfiguration property on IrisHF.Client. Nothing is created when a
configuration with that name already exists.
SQL / ObjectScript
│
▼
IrisHF.Importer ──► IrisHF.Client ──► https://datasets-server.huggingface.co
│ /info schema + splits
│ /rows pages of 100 decoded rows
▼
IrisHF.Schema ──► infer IRIS types, sanitise names
│
▼
IrisHF.Writer ──► CREATE TABLE + batched INSERT (200 rows / 200 KB per statement)
│
▼
IrisHF.Manifest ──► audit row (dataset, table, rows, status, timings)
Design notes:
features give the dtype and, forClassLabel, the label names. String columns are sized from a 100 rowINSERTs. If aVARCHAR, allVARCHAR columns are widened to LONGVARCHAR and the batch is retried. AfailedRows counts every rejected row and up to 20 sample errorerrors.429 and 5xx responses are retried with exponential backoff.IrisHF.Manifest.| Hugging Face | IRIS |
|---|---|
string (≤ 255 chars in sample) |
VARCHAR(255) |
string (longer) |
LONGVARCHAR |
int8/16/32/64, uint* |
BIGINT |
float16/32/64, decimal* |
DOUBLE |
bool |
%Library.Boolean (0/1) |
timestamp[...], date |
VARCHAR(64) / VARCHAR(32) |
ClassLabel |
VARCHAR(255) (label name) |
Sequence, Translation, Image, Audio, Video, Pdf |
LONGVARCHAR (JSON) |
VARCHAR columns inferred from the sample are widened to LONGVARCHAR
automatically when a later row does not fit, so no row is lost to an
undersized column.
/rows endpoint returns at most 100 rows per request./rows cells; the importer still countstruncated_cells the API reports per row in truncatedCells.src/cls/IrisHF/Utils.cls identifier + literal helpers
src/cls/IrisHF/Client.cls datasets-server REST client
src/cls/IrisHF/TLS.cls outbound TLS configuration helper
src/cls/IrisHF/Schema.cls feature -> IRIS type inference
src/cls/IrisHF/Writer.cls batched INSERT writer
src/cls/IrisHF/Importer.cls orchestration
src/cls/IrisHF/Manifest.cls audit trail
src/cls/IrisHF/API.cls SQL entry points
src/cls/IrisHF/CLI.cls terminal helpers
Rebuild after editing sources:
docker compose up -d --build
MIT — see https://github.com/foxchengyu/iris-hf-importer/blob/main/LICENSE.