DataPipe an interoperability framework to ingest data in InterSystems IRIS in a flexible way.
You can try out the included quickstart container to have a look at DataPipe.
docker compose -f docker-compose.quickstart.yml up -d
DataPipe uses interoperability framework.
Check DataPipe.Test.Production interoperability production loggin-in as superuser
/ SYS
.
Now, you only need to generate some data.
Now you can generate some sample data that will be processed in your pipes:
Open an WebTerminal interactive session (terminal)
Generate data
do ##class(DataPipe.Test.Helper).QuickStart()
DataPipe includes an UI that allows you to manage the data that has been processed.
Follow these steps to run datapipeUI.
git clone https://github.com/intersystems-ib/iris-datapipeUI
cd iris-datapipeUI
docker compose up -d
dpadmin
/ demo
When processing data you need to follow these steps:
You need to define a model for the data you want to ingest.
A model is a class that extends from DataPipe.Model.cls where you must implement some methods.
In your model you will implement:
Have a look at these examples:
You need to implement an Ingestion process using DataPipe.Ingestion.BP.IngestionManagerContext as context.
This process will receive an input and will implement:
Input > InboxAttributes
:
Input > Model
:
You can find examples here:
Rest of the components are provided by DataPipe you need to add are provided by DataPipe.
These components will call the different methods you have in your model.
Usually you will add:
Have a look at a full example in DataPipe.Test.Production.cls
DPIPE
)zpm "install iris-datapipe"
DataPipe uses different security resources you can assign to InterSystems IRIS user account:
DP_ADMIN
- DataPipe AdministratorDP_MENU_DASHBOARD
- Access to Dashboard menu option in UIDP_MENU_SEARCH
- Access to Search menu option in UIYou can use the following as an EXAMPLE to set up the users in your system:
Create resources:
zn "%SYS"
write ##class(Security.Resources).Create("DP_ADMIN","DataPipe Admin Privilege")
write ##class(Security.Resources).Create("DP_MENU_DASHBOARD","DataPipe UI Dashboard Menu Access")
write ##class(Security.Resources).Create("DP_MENU_SEARCH","DataPipe UI Search Menu Access")
Create a DataPipe_Admin
role:
write ##class(Security.Roles).Create("DataPipe_Admin","DataPipe Administrator","DP_ADMIN:RWU,DP_MENU_DASHBOARD:RWU,DP_MENU_SEARCH:RWU,%DB_USER:RW,%DB_IRISSYS:R")
Grant access to tables and views to DataPipe_Admin
role:
GRANT INSERT,SELECT,UPDATE ON DataPipe_Data.Pipe, DataPipe_Data.Preference TO DataPipe_Admin
GRANT SELECT ON DataPipe_Data.VInbox, DataPipe_Data.VIngestion, DataPipe_Data.VStaging, DataPipe_Data.VOper TO DataPipe_Admin
Create a new user that belongs to DataPipe_Admin
role:
write ##class(Security.Users).Create("dpadmin","DataPipe_Admin","demo")
You can also check out the DataPipe_Admin role definition in InterSystems IRIS.
You can install DataPipe in one namespace and use it in different namespaces using the same UI.
To do so, you need to:
The following example uses the quickstart container to configure Datapipe also in USER
namespace:
DPIPE
namespace.DPIPE
to USER
:zn "%SYS"
// package mapping
set props("Database")="DPIPE-DATA"
write ##class(Config.MapPackages).Create("USER", "DataPipe", .props)
write ##class(Config.MapPackages).Create("USER", "Form", .props)// global mapping
write ##class(Config.MapGlobals).Create("USER", "DataPipe*", .props)
write ##class(Config.MapGlobals).Create("USER", "Form*", .props)
// routine mapping
write ##class(Config.MapRoutines).Create("USER", "Form*", .props)
USER
namespace:GRANT INSERT,SELECT,UPDATE ON DataPipe_Data.Pipe, DataPipe_Data.Preference TO DataPipe_Admin
GRANT SELECT ON DataPipe_Data.VInbox, DataPipe_Data.VIngestion, DataPipe_Data.VStaging, DataPipe_Data.VOper TO DataPipe_Admin
USER
which already have DataPipe components:zn "USER"
do $system.OBJ.Load("/app/install/SampleProduction.cls", "ck")
Start the production
Send a test message
zn "USER"
write ##class(DataPipe.Test.REST.Helper).SendHTTPRequests(1, "/test/user/api/message")
When enabling datapipeUI, you must consider the following:
You must consider CORS restrictions.
You can see a basic example here that allows any incoming connection (this is only recommended for testing).
Make sure that CSPSystem
user in InterSystems IRIS have read permission on the resource of the database where you have installed DataPipe.
See CONTRIB.md Testing Upgrade from DataPipe 0.0.2 to DataPipe 2.x section