iris-hf-importer  

Load any public Hugging Face dataset into
InterSystems IRIS as a real SQL table — with one SQL statement.

SELECT IrisHF.API_ImportSimple('stanfordnlp/imdb');

{"status":"ok","dataset":"stanfordnlp/imdb","config":"plain_text","split":"train",
 "table":"HF_stanfordnlp_imdb_plain_text_train","rowsAvailable":25000,
 "rowsImported":25000,"columns":2,"truncatedCells":0,"failedRows":0,"elapsedSeconds":37}

The importer reads the dataset schema from the Hugging Face
datasets-server API, infers IRIS
column types, creates the table, and streams the rows in pages of 100. It is
written entirely in ObjectScript: no Python, no pip, no external libraries.

Why

Hugging Face hosts hundreds of thousands of datasets, but getting one into IRIS
normally means writing a throw-away Python script, guessing the schema, and
handling pagination yourself. This project turns that into a single call that
works from SQL, from ObjectScript, or from a terminal.

Quick start

Docker (recommended)

git clone https://github.com/foxchengyu/iris-hf-importer.git
cd iris-hf-importer
docker compose up -d --build

IRIS starts on http://localhost:52773/csp/sys/UtilHome.csp (_SYSTEM / SYS).

Run the demo import:

docker compose exec iris iris session IRIS < docker/demo.script

Or open Management Portal → System Explorer → SQL and run:

SELECT IrisHF.API_ImportSimple('stanfordnlp/imdb')

Existing IRIS instance

1. Import src/cls with Management Portal → System Explorer → Classes →
   Import, or from a terminal:

   Do $SYSTEM.OBJ.LoadDir("/src/cls", "ck")
   

2. Run any of the entry points below.

The classes are installed in the namespace you import them into. The examples
use USER.

Usage

From SQL

-- one line, defaults for config/split, all rows
SELECT IrisHF.API_ImportSimple('stanfordnlp/imdb');
-- full control: dataset, target table, config, split, max rows
SELECT IrisHF.API_ImportDataset('stanfordnlp/imdb', 'SQLUser.Imdb', 'plain_text', 'train', 1000);
-- what has been imported so far
SELECT IrisHF.API_ListImports();

API_ImportSimple and API_ImportDataset return a JSON summary as a string, so
you can read the result in any SQL client.

From ObjectScript

Do ##class(IrisHF.CLI).Demo()                                  ; 300 rows of imdb
Do ##class(IrisHF.CLI).Import("stanfordnlp/imdb")              ; everything
Do ##class(IrisHF.CLI).Import("stanfordnlp/imdb", "SQLUser.MyTable", 500)

Or call the importer directly and inspect the returned object:

Set stats = ##class(IrisHF.Importer).Import("stanfordnlp/imdb", "", "", "", 100)
Write stats.%ToJSON()

Querying the result

SELECT TOP 5 * FROM HF_stanfordnlp_imdb_plain_text_train

Because it is a normal table, everything else works too: indices, SELECT
aggregations, BI tools, Embedded Python, Interoperability productions.

Options

The default table name is HF_<owner>_<name>_<config>_<split>, sanitised to a
valid IRIS identifier.

Private and gated datasets

Set HF_TOKEN in the environment (or pass a token to IrisHF.Client). With
Docker:

HF_TOKEN=hf_xxx docker compose up -d --build

Proxies and restricted networks

Outbound HTTPS must reach datasets-server.huggingface.co. Behind a corporate
proxy, set HF_PROXY to the proxy address; the client opens an HTTP CONNECT
tunnel and negotiates TLS through it:

HF_PROXY=http://proxy.example.com:8080 docker compose up -d --build

To use a mirror of the datasets-server API instead, set HF_ENDPOINT:

HF_ENDPOINT=https://datasets-server.internal.example.com docker compose up -d --build

Both are read from the environment by IrisHF.Client. In ObjectScript, set the
Proxy and Endpoint properties instead.

TLS configuration

IRIS needs a client SSL configuration for every HTTPS call. Some images ship
without one, which makes requests fail with error #6159. The importer looks
for a configuration and creates IrisHFTLS on first use when none exists. To
use a different one, set the SSLConfiguration property on IrisHF.Client.
Nothing is created when a configuration with that name already exists.

How it works

SQL / ObjectScript
        │
        ▼
 IrisHF.Importer ──► IrisHF.Client ──► https://datasets-server.huggingface.co
        │                                  /info   schema + splits
        │                                  /rows   pages of 100 decoded rows
        ▼
 IrisHF.Schema ──► infer IRIS types, sanitise names
        │
        ▼
 IrisHF.Writer ──► CREATE TABLE + batched INSERT (250 rows / 400 KB per statement)
        │
        ▼
 IrisHF.Manifest ──► audit row (dataset, table, rows, status, timings)

Design notes:

• No Python dependency. The datasets-server returns decoded JSON, so
  ObjectScript’s native JSON and HTTP support is enough. This keeps the image
  stock and the install a single class import.
• Schema inference. Hugging Face features give the dtype and, for
  ClassLabel, the label names. String columns are sized from a 100 row
  sample, and label indices are decoded to their human readable names.
• Batched, resilient writes. Rows are written in multi-row INSERTs. If a
  batch fails (for example a value longer than the inferred VARCHAR), the
  batch is retried row by row so one bad row cannot abort the import; failures
  are reported in the summary.
• Retries. 429 and 5xx responses are retried with exponential backoff.
• Audit trail. Every import is recorded in IrisHF.Manifest.

Type mapping

Limitations

• The datasets-server /rows endpoint returns at most 100 rows per request and
  may truncate very large cells; the summary reports how many cells were
  truncated.
• Only datasets that the datasets-server has indexed are supported. Datasets
  that fail server-side indexing return an HTTP 500 and are reported as errors.
• Images and audio are stored as JSON metadata, not as binaries.

Development

src/cls/IrisHF/Utils.cls      identifier + literal helpers
src/cls/IrisHF/Client.cls     datasets-server REST client
src/cls/IrisHF/TLS.cls        outbound TLS configuration helper
src/cls/IrisHF/Schema.cls     feature -> IRIS type inference
src/cls/IrisHF/Writer.cls     batched INSERT writer
src/cls/IrisHF/Importer.cls   orchestration
src/cls/IrisHF/Manifest.cls   audit trail
src/cls/IrisHF/API.cls        SQL entry points
src/cls/IrisHF/CLI.cls        terminal helpers

Rebuild after editing sources:

docker compose up -d --build

License

MIT — see https://github.com/foxchengyu/iris-hf-importer/blob/main/LICENSE.