Interop Medic

For judges: two minutes

• Public link: https://<user>.github.io/<repo>/ (published when the contest opens). It is a
  static copy of the page that replays a recorded run.

• Local: docker compose up -d --build, then open http://localhost:52773/medic/index.html. The
  page needs no sign-in. The Management Portal uses _SYSTEM / SYS, or your IRIS_PASSWORD.

• What to click: pick Critical result, watch the trace build, then Approve. Then
  Run a batch of 5, and finally + Build your own to write a message of your own.

• The labels:

  • Live means the page is talking to the running IRIS production.
  • Replay of a real run means it is replaying a bundle (web/data/bundle.json) recorded with
    real models, as on the public site; Replay (mock models) is the same for a bundle recorded
    on the mocks.
  • Mock means the models are the offline mocks.
  • Downstream effects (resend, reroute and so on) are marked “(simulated)”: nothing downstream exists.

• Real models: Jev (fast tier) and Gemini via AI Hub (agent) need keys in .env and the AI Hub
  build 154 image (see below). Without them everything runs on the mock models, offline.

• Safety: the /medic web applications are unauthenticated, for a local demo only. Do not expose
  port 52773.

• The demo production runs Medic.Triage with DryRun off, so auto-applied decisions show up on
  the page; every effect is still simulated. To turn it back on, set the DryRun setting of
  Medic.Triage (Policy group) in the production configuration.

• Tests: node --test "web/tests/*.test.mjs" for the page, uv run python -m unittest eval/test_record.py for the recorder.

• Re-recording the public bundle:

  uv run python eval/record.py --batches 3 --out web/data/bundle.json
  

  If any case is skipped (for example on a quota error), the recorder leaves the bundle untouched,
  writes bundle.partial.json next to it and exits nonzero; --allow-partial overrides that. The committed bundle is a
  real Jev + Gemma run (local Gemma through AI Hub): add --fast and --agent to pick other
  models, and --timeout 900 for a local model on a CPU.

• Publishing: the Deploy web/ to GitHub Pages workflow (.github/workflows/pages.yml) is
  manual (workflow_dispatch) and runs the page tests before it deploys web/.

An agent that triages failed HL7 v2 and FHIR messages in InterSystems IRIS Interoperability.
For each errored or suspended message it decides one of resend, fix and resend,
reroute, quarantine or escalate, with a confidence, a severity, a rationale and a
proposed fix.

Status: this runs fully offline with mock providers. The fast tier can call Jev or Laya. The
AI Hub agent (Medic.AIHub.TriageAgent) only compiles on the AI Hub Early Access build (154). It
has run there on real Gemini and Gemma: it escalated a critical result using
ValidateHL7/CheckEndpoint and SearchPrecedents. The bundled recording is a real Jev + Gemma run of
16 scenarios and 15 random cases.

How it works

Medic.DemoFeed ──> Medic.Triage (BPL) ──> Medic.Agent ──> Medic.FastTier
 (case files)       policy + dry run       │  1. ask the fast tier (Jev, Laya or mock)
                         │                 │  2. confident & low severity? accept it
                         │                 └  3. otherwise run the tool-using agent
                         └─ held for a person ──> Medic.Review ──> Medic.Reviewers
                                                  (Workflow task in the IRIS Workflow inbox)

Every decision goes through Medic.Agent. The fast tier is the agent’s first step, not a bypass.
When it is confident and the case is low-stakes, the agent stops there; reroutes always go to the
full agent, because only its tools can name a target. Otherwise the agent runs with the fast
answer as a hint. Medic.Policy.AutoApply then decides whether the decision can be applied
without a person. DryRun is covered in the judges section above and in the table below.

Held decisions become Workflow tasks

When the policy holds a decision for a person, Medic.Triage also sends it to Medic.Review,
which opens an IRIS Workflow Engine task for the role Medic.Reviewers (the
EnsLib.Workflow.Operation item of that name; _SYSTEM, SuperUser and Admin are members).
The task shows the proposed action, confidence, severity, rationale and fix, with the choices
Approve, Escalate and Quarantine. Answer it in the Workflow inbox
(http://localhost:52773/csp/user/_DeepSee.UserPortal.Workflow.zen, or Management Portal >
Interoperability > Workflow > Workflow Inbox) and the answer goes through the same code as the
web page’s buttons: the trace gets a person step, the audit trail a row and the precedent store
the case. Whoever decides first wins; deciding on the page completes the task.

Every LLM call in the Visual Trace (Community idea DPI-I-689)

This entry implements DPI-I-689, Use InterSystems
Interoperability as a Traceability tool for GenAI LLM pipelines. Each model call is a message in
the production, so the Visual Trace of one failed message holds the whole pipeline: the fast
model’s call, the gate, the agent’s run with each tool call, the policy, and a person’s review.

To see it: triage a scenario on the web page (e.g. Critical result), then open Interoperability >
Message Viewer, pick the session’s newest Medic.Triage row and click Trace. Click the
Medic.Msg.Decision response of Medic.Agent (and of Medic.FastTier). Its body shows:

The same session also holds the Medic.Msg.ReviewRequest and the Workflow task when a person
has to look. Nothing leaves IRIS: prompts and patient data stay in the production, and the trace
leads straight to an action (approve, override, resend). The vector precedent store
(Medic.Data.Precedent, IRIS Vector Search) is part of the same trace: SearchPrecedents calls
appear in ToolCalls with the case they found.

Run it

You need Docker and uv. The Python scripts use only the standard
library; uv sync creates .venv with a matching Python.

uv sync
cp .env.example .env            # optional: your own IRIS password, fast-tier settings
uv run python dataset/generate.py --synthea dataset/synthea --cases-dir data/cases
docker compose up -d --build
uv run python dataset/feed.py --count 20 --rate 1

feed.py drips a shuffled mix of failed messages into the production. Use --only outage_critical_result,patient_mismatch to show the hard cases.

Open the Management Portal at http://localhost:52773/csp/user/EnsPortal.ProductionConfig.zen and
look at the Visual Trace for each message. If ports 52773 or 1972 are taken, set MEDIC_WEB_PORT
and MEDIC_SS_PORT.

Signing in. Use _SYSTEM with the IRIS_PASSWORD from .env, or SYS if none is set.

• No change-password prompt: _SYSTEM, SuperUser and Admin are never forced to change
  their password.
• When IRIS_PASSWORD is applied: the first time the container starts, and again only after you
  change it in .env. A password changed in the portal survives restarts.
• Recreated containers: recreating the container (docker compose down, or a rebuild) starts
  again from the image.

Readiness report. To check the production, providers, routing table, mounts and accounts:

docker exec interop-medic iris session IRIS -U USER "##class(Medic.Setup).Doctor()"

With AI Hub (Early Access build 154)

AI Hub is only in the Early Access Community build. Download it from the
EAP portal (“AI Hub” program),
then load it, retag it and build with it:

docker image load -i irishealth-community-2026.3.0AI.154.0-docker.tar.gz
docker images
docker tag  irishealth-community:2026.3.0AI.154.0
IRIS_IMAGE=irishealth-community:2026.3.0AI.154.0 docker compose up -d --build

Without it, the default community image runs everything with the mock providers.

Agent tier: the AI Hub agent

Medic.AIHub.TriageAgent is an AI Hub %AI.Agent. Its system prompt sets out the five actions
and the safety rules, and it must answer with one JSON decision. Its tools, in
Medic.AIHub.ToolSet, are all read-only:

• ValidateHL7: parses the message and checks its structure.
• GetRoutingRule and ClosestRoute: look up the routing table, including the closest match
  for a misspelt receiver.
• CheckEndpoint: reads a target’s queue depth.

Two policies sit on the agent’s tool manager:

• Medic.AIHub.ReadOnlyPolicy allows only those tools. The agent can read, but it never resends
  or edits anything. Applying a decision stays with Medic.Policy.AutoApply or a person.
• Medic.AIHub.ToolTrace records each tool call. The decision’s rationale lists the tools that
  ran, and token use becomes the decision’s cost.

Medic.Provider.AIHubAgent runs the agent once per message, with the fast tier’s verdict as a
hint, and checks the reply. A reply with no JSON or an unknown action is rejected. A reroute with
no target becomes an escalation. Without AI Hub, or on any agent error, the case escalates with
zero confidence rather than failing.

The AI Hub classes are in src-aihub/ and are compiled only when the image has %AI.Agent. The
default community image skips them, and the mock agent is used.

To use it, set IRIS_IMAGE to the build 154 image, then add to .env:

MEDIC_AGENT_PROVIDER=Medic.Provider.AIHubAgent
AIHUB_PROVIDER=openai
AIHUB_MODEL=
AIHUB_API_KEY=

Choosing the agent’s model

The agent’s model switch on the web page lists every model in MEDIC_AGENT_MODELS whose key is
configured, plus Mock and (with OLLAMA_BASE_URL) Ollama. Each entry is Label=provider:model;
the provider is an AI Hub provider name (gemini, openai, anthropic, xai, deepseek, … as
listed by ##class(%AI.Provider).%GetProviders()), and a blank model means the provider’s default.

MEDIC_AGENT_MODELS=Gemini=gemini:gemini-3.5-flash,GPT=openai:gpt-5-mini,Claude=anthropic:claude-sonnet-5-5,Grok=xai:grok-4
GEMINI_API_KEY=
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
XAI_API_KEY=

Keys stay on the server. For each provider the agent takes the key from IRIS Interoperability
credentials named Medic.<provider> (e.g. Medic.anthropic, key in the Password field) if they
exist, else <PROVIDER>_API_KEY, else the variable AI Hub names for that provider (GOOGLE_API_KEY
for gemini), else AIHUB_API_KEY when the provider is AIHUB_PROVIDER. Entries with no key are
not shown.

OpenRouter is one of AI Hub’s providers, so its models need only OPENROUTER_API_KEY and entries
such as Kimi=openrouter:moonshotai/kimi-k2 (the model is OpenRouter’s slug). Any other
OpenAI-compatible API works under a name AI Hub does not know, with <NAME>_BASE_URL and
<NAME>_API_KEY: for Inception, Mercury=inception:mercury-2.5,
INCEPTION_BASE_URL=https://api.inceptionlabs.ai/v1/ and INCEPTION_API_KEY. The container reads
every .env variable, so new keys need no change to docker-compose.yml.

Local Ollama models go in the same list with the provider ollama and need no key, only
OLLAMA_BASE_URL: Gemma=ollama:gemma4,Qwen=ollama:qwen3.8.

Models listed without a key still appear on the page, greyed out, with the variable to add.

Judges’ own keys

The page’s API keys button lets each visitor paste their own key for any hosted model the
server lists (Claude, ChatGPT, Grok, Gemini, Mercury, OpenRouter models, …). A greyed-out model
turns on once its provider has a key, and a visitor’s key replaces the server’s for their calls.

• The key stays in that browser tab (sessionStorage); closing the tab forgets it.
• Each triage sends only the keys its chosen models need. The server keeps them in
  ^IRIS.Temp.MedicKeys (IRISTEMP: never journaled, cleared on restart) under a one-time token for
  that request only, and drops them when it ends. Messages carry the token, never the key, and any
  key a provider echoes in an error is replaced with *** before the decision is stored.
• Each tab also picks its own models, sent with every triage, so judges sharing one server never
  switch models for each other. POST /models still sets the server’s default.
• The browser can choose a model and send a key, never a URL: endpoints come from the server’s
  configuration only.

On a shared server (e.g. Azure), serve the page over HTTPS only (Container Apps or App Service
terminate TLS for you), or keys cross the network in clear text. The API has no login, so anyone
with the URL can also spend the keys in the server’s .env: on a public host, leave hosted keys out
of .env and let judges bring their own, or put the app behind authentication.

The fast tier can also run a fast chat model, such as Inception’s Mercury, through
Medic.Provider.ChatFast: one JSON-mode call to an OpenAI-compatible chat/completions endpoint
asks the same three questions as System One, with the same wrong-patient guardrail. List them in
MEDIC_FAST_MODELS, in the same format and with the same keys:

MEDIC_FAST_MODELS=Mercury 2.5=inception:mercury-2.5

The switch sends only the label; it sets LLMProvider, LLMModel, LLMBaseURL and LLMCredentials
(the credentials name, never the key) on Medic.Agent. Without MEDIC_AGENT_MODELS the switch
offers the one AIHUB_PROVIDER / AIHUB_MODEL model. Cost tracking uses one
AIHUB_PRICE_* pair for all of them.

Fast tier: mock, Jev or Laya

Jev (TypeSafe) and Laya (open weights, Apache 2.0) use the same POST /v1/systemone API, so one
provider, Medic.Provider.SystemOne, handles both. It sends one request per message with three
typed questions:

• the action (a choice),
• the severity (a score), and
• whether acting could attach data to the wrong patient (a yes/no probability).

If that wrong-patient probability is 0.7 or higher, the decision is forced to escalate. If the
server is down or replies badly, the case goes to the agent instead of failing.

Put the settings in .env (copied from .env.example, git-ignored):

SYSTEMONE_MODEL picks a Laya checkpoint (english, multilingual, typed-decisions). The same
values can be set on Medic.FastTier in the Management Portal, with the key stored as
Interoperability credentials; portal settings take precedence over env vars.

Run Laya locally with pip install "laya[serve]" then laya-serve. See the upstream docs for GPU
and Docker options.

To check the plumbing without either model, run uv run python eval/fake_systemone.py and point
SYSTEMONE_BASE_URL at http://host.docker.internal:8765.

Confidence from different models is not comparable, so re-run the eval after switching. The
auto-apply threshold is fitted per model.

Local models with Ollama

Ollama serves both tiers on your own machine, with no API keys:

• Fast tier: a decision model such as nimble-v3 answers Ollama’s POST /v1/systemone, the same API as Jev, so Medic.Provider.SystemOne uses it unchanged.
• Agent: any tool-calling chat model, such as gemma4, runs through AI Hub’s openai provider pointed at Ollama’s OpenAI-compatible /v1.

OLLAMA_BASE_URL=http://host.docker.internal:11434
OLLAMA_FAST_MODEL=nimble-v3      # default
OLLAMA_AGENT_MODEL=gemma4        # optional (agent); qwen3.8 (27B) needs a GPU: on a CPU it outlasts AI Hub's request timeout

The fast tier’s model switch then offers the model by name (“Nimble v3”); the agent’s offers its
model only when OLLAMA_AGENT_MODEL is set. On a CPU-only machine a call can take minutes. The fast tier waits up to OLLAMA_TIMEOUT seconds (default 600), and the image raises the web gateway’s response timeout to 900 s. The same values can be set by hand on Medic.Agent (LLMProvider openai, LLMModel, LLMBaseURL http://host.docker.internal:11434/v1/). Hosted providers that need a key (openai, anthropic, xai, …) go through MEDIC_AGENT_MODELS (see “Choosing the agent’s model”) or AIHUB_PROVIDER, AIHUB_MODEL and AIHUB_API_KEY.

Web page and REST API

The image serves a static page at http://localhost:52773/medic/index.html (this IRIS build has no default-page setting, so the bare /medic/ is a 404) and a same-origin REST API at /medic/api (health, scenarios, random cases, triage, approve/override, model switch). Both web applications are unauthenticated with role %All: this is a local demo, do not expose the container to a network. LAYA_BASE_URL (optional) adds a “Laya” choice to the fast-tier model switch. API tests: uv run python -m unittest tests/test_api.py -v.

Eval

uv run python eval/run.py

This sends all 450 cases through the running production twice: once through the full pipeline
and once to the fast tier alone, as a comparison. Results go to data/results/<tag>/, where the
tag defaults to the fast-tier provider (jevmock, systemone, …) or comes from --tag laya-local.
That way runs with different models sit side by side. The report shows accuracy per corruption type, which tier decided,
latency and estimated cost, and an auto-apply threshold with a guarantee:

• The threshold is chosen on the calibration split by Learn-then-Test with a Bonferroni
  correction over a fixed grid, using a Clopper-Pearson bound.
• With probability at least 1 − δ (default 90%), auto-applied decisions are wrong at most α of
  the time.
• The threshold is then checked on the held-out test split.
• Escalations are never auto-applied.

With the mock providers the pipeline scores 100% and the mock fast tier alone 64.7%, because the
mock fast tier can’t see context: critical results during outages, ambiguous dates, and receivers
that don’t resolve to a route. These numbers only show the harness works, since the mocks were
written knowing the labels. Real numbers come from re-running with Jev or Laya and the AI Hub agent.

Tests:

uv run python -m unittest eval/test_score.py dataset/test_generate.py
docker exec -i interop-medic iris session IRIS -U USER <<< 'set ^UnitTestRoot="/tmp" do ##class(%UnitTest.Manager).DebugRunTestCase("","Medic.Test.SystemOne","","") do ##class(%UnitTest.Manager).DebugRunTestCase("","Medic.Test.AIHubAgent","","") do ##class(%UnitTest.Manager).DebugRunTestCase("","Medic.Test.Models","","") do ##class(%UnitTest.Manager).DebugRunTestCase("","Medic.Test.Review","","")'

Dataset

dataset/generate.py writes 16 corruption types × N synthetic cases (reroutes weighted 3×), each
with a known correct action. With --synthea dataset/synthea, patients and lab results come from
Synthea, so names, ages and values look realistic.

• Patients: 300 Synthea patients.
• Lab results: their real potassium, sodium, glucose, haemoglobin, calcium, urea and chloride
  results.
• Critical results: Synthea values are almost all in range, so critical results are derived by
  pushing a real result past its critical limit.
• The committed sample: dataset/synthea is about 220 KB and keeps only ID, birth date, name and
  sex, plus lab results. To rebuild it from a full Synthea CSV export, run
  uv run python dataset/synthea_sample.py --src <export>/csv.
• Without --synthea: the generator uses its own made-up patients. Some cases share an error message but need different actions depending on context. For
  example, a timeout is normally a resend, but a critical potassium result stuck behind a 4-hour
  outage needs a person. Cases are split by hash into calibration (60%) and test (40%).

uv run python -m unittest dataset/test_generate.py

All data is synthetic. Don’t add real messages.

License

MIT, see https://github.com/Fruitkwan/interop-medic-ai-hub/blob/main/LICENSE. The AI Hub Early Access image and model weights are not part of this repository.