Home Applications hf-split-ledger

hf-split-ledger

Community Project
This project is maintained by its author and is not officially supported by InterSystems. For technical support, please contact the project developer.
0
0 reviews
0
Awards
1
Views
0
IPM installs
0
Add to bundle
Details
Releases (1)
Reviews
Issues
Import Hugging Face dataset splits into InterSystems IRIS with ObjectScript, preserving source row numbers and JSON records.

What's new in this version

Initial Release

HF Split Ledger

English · 简体中文

Load a public Hugging Face Dataset Viewer split into InterSystems IRIS with one command. Each row keeps its zero-based source index and its JSON document, including nested fields. The source table identifies the dataset, configuration and split. This makes it possible to compare the imported rows with the original viewer page.

Run from the IRIS terminal

HFSplit.Importer is the ObjectScript implementation. It fetches Viewer pages over HTTPS, validates the JSON, resolves source identities and writes the rows in an IRIS transaction. Existing splits keep their original source keys.

  1. Copy or mount this repository where the IRIS instance can read it (for example, /app). Compile the class in the application namespace.
set sc=$system.OBJ.Load("/app/https://github.com/TerasaGUO/hf-split-ledger/blob/main/src/HFSplit.Importer.cls","ck")
do $system.Status.DisplayError(sc)
  1. Configure an IRIS client TLS profile with peer verification. On a clean Linux IRIS 2026.2 container an administrator can create the following profile in the %SYS namespace. Use the appropriate trusted CA file on other systems.
set p("Enabled")=1,p("Type")=0,p("VerifyPeer")=2
set p("CAFile")="/etc/ssl/certs/ca-certificates.crt"
set sc=##class(Security.SSLConfigs).Create("HFViewerTLS",.p)
do $system.Status.DisplayError(sc)
  1. Return to the application namespace. The command below returns 101. Omit the fourth argument to import the whole split. Pass an existing TLS profile name as the fifth argument if needed. The user needs permission to create and update the application tables in HFSplit. The IRIS host needs HTTPS access to Dataset Viewer.
write ##class(HFSplit.Importer).Import("cornell-movie-review-data/rotten_tomatoes","default","test",101)

Run from the shell

The compatible Python 3.10+ CLI passes its arguments to the ObjectScript class using the official driver. Set HF_VIEWER_TLS to select the outbound IRIS profile. For a remote database connection, set IRIS_SSL_CONFIG to your client TLS configuration. The CLI account needs permission to invoke the importer through the IRIS Native API in the application namespace.

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
export IRIS_DSN='localhost:1972/USER'
export IRIS_USER='your-import-user'
# Set IRIS_PASSWORD in your shell or secret manager.
python ledger.py cornell-movie-review-data/rotten_tomatoes default test --limit 101

Expected output for the sample is Committed 101 rows for cornell-movie-review-data/rotten_tomatoes / default / test. Omit --limit to refresh the entire split. Run this SQL in the same IRIS namespace to inspect the page boundary.

SELECT r.SourceRow, r.JsonBody
FROM HFSplit.Records r
JOIN HFSplit.Sources s ON s.SourceKey = r.SourceKey
WHERE s.DatasetName = 'cornell-movie-review-data/rotten_tomatoes'
  AND s.ConfigName = 'default' AND s.SplitName = 'test'
  AND r.SourceRow IN (99, 100)
ORDER BY r.SourceRow

Refresh behavior

Each dataset/config/split has its own key. A full refresh inserts or updates source rows and removes stale trailing rows in one transaction. A limited import updates only the requested prefix and preserves any previous suffix. A failed page or IRIS write rolls back the run. Use this tool for small and medium splits that fit in one IRIS transaction.

The source must expose complete JSON rows through the public Dataset Viewer API. Rows with truncated fields or inconsistent offsets are rejected. The importer checks the reported total and, when provided, the revision header while paging. It does not pin an immutable revision when the server omits that header. Network access to the viewer is required.

Verify

In a disposable test namespace, compile the deterministic page fixture. Use the same namespace for the verification commands below.

set sc=$system.OBJ.Load("/app/tests/HFSplitTest.Fixture.cls","ck")
do $system.Status.DisplayError(sc)
python -m unittest -v
python verify_live.py

The native tests cover page boundaries, preserved suffixes, empty splits and rollback on truncated rows or a revision change. The live script imports the complete public split twice and compares the total and two boundary rows with the source. Both runs on IRIS 2026.2 produced 1,066 rows.

Platform interfaces: Dataset Viewer rows and InterSystems Python DB-API.

Community Bounty idea: Load datasets from Hugging Face into IRIS.

Version
1.0.020 Nov, 2026
Category
Data Set
Works with
InterSystems IRIS
First published
20 Nov, 2026
Last edited
20 Nov, 2026