csvgen-pyprod – a production for creating IRIS tables from CSV files

This project implements a csvgen interoperability production with InterSystems PyProd. It watches a directory for new CSV files (InboundAdapter + Business Service), passes them to a router (Business Process), which first sends a synchronous request to the ToCheckDB host (Business Operation) to check whether a table with the same name as the file already exists. The result is passed back to the router, which may then send a create-table request to ToDB (Business Operation), and finally sends a request to ToDB to insert the data.

This is a basic example of using PyProd to create productions in InterSystems, and was implemented as part of the InterSystems Developer Community Idea to Application program.

Get started

Clone the repo:

git clone https://github.com/gabriel-ing/pyprod-csvgen.git
cd pyprod-csvgen

Start up the Docker container:

docker-compose up --build -d

The components and production files are loaded during the Docker build, so you just need to start the production. This can be done through the management portal at:

http://localhost:62773/csp/ensemble/EnsPortal.ProductionConfig.zen?$NAMESPACE=ENSEMBLE&PRODUCTION=CsvgenPyprod.Prod

Or with the controls.py file:

docker-compose exec -it iris bash 
cd /src/csvgen-pyprod
python3 controls.py start CsvgenPyprod.Prod

After this, you can test the production by placing any CSV file into the ./IN directory. After a few seconds you should see it move to WORKING, then when complete, to OUT. Take a look at the production portal and messages viewer to see the message trace.

[!WARNING]
This interoperability production is meant as a demo application to show the use of PyProd. It is not designed for production usage.

Manual Loading

The production is contained in two Python files, components.py and production.py. The components file contains the definition of the business hosts (Services, Processes, Operations), the Inbound Adapter, and the message class. The production file contains the production definition. Because CsvgenPyprod.Prod is defined as a Production class, its settings should not be adjusted in the UI. You can alternatively define productions in the UI using the host components from the components file.

If you are doing this outside the docker-compose build in this repo, ensure PyProd is set up properly — instructions below.

To register the components and production in IRIS, open a shell in the container:

docker-compose exec -it iris bash

Then run:

intersystems_pyprod src/csvgen_pyprod/components.py
intersystems_pyprod src/csvgen_pyprod/production.py

If you make any changes to the code while the production is running, the production will need to restart before the changes are reflected!

You can start the production with the handy CLI:

python3 src/csvgen_pyprod/controls.py start CSVGenPyProd.Prod

To see the production in action, throw any random

IPM Install

This production has been packaged with IPM and is installable with a single command:

zpm "install csvgen-pyprod -v"

This, by default, will install this production in /usr/irissys/lib/csvgen-pyprod, and will create directories for the file watching (an IN folder, an OUT folder and a WORKING folder) here inside a directory called Data.

To change the location of the file processing directory, you can change this with an install argument:

zpm "install csvgen-pyprod -v -DfileWatcherRoot 'home/irisowner/dev'

By default, this will not set any of the required environmental variables required for PyProd to work, and will therefore fail on installation if you have not set the environment as detailed in the PyProd Setup section below. However, if you would like to automatically set these variables up, you can add a -DenvSetup 1 flag.

[!Warning]
The command below will change environmental variables, including setting default IRISUSERNAME and IRISPASSWORD. Use with caution

zpm "instal csvgen-pyprod -v -DenvSetup 1"

Unfortunately, despite my best efforts I haven’t managed to get the production to start automatically without the install erroring, so to start the production run:

do ##class(Setup.Production).StartProduction()

Or:

python3 -c "from intersystems_pyprod import director; director.start_production("CSVGenPyProd.Prod")

Please note, if you get an IrisSecureStart failed: IRIS_ACCESSDENIED (-15), this means Callin is not enabled, which is required for PyProd. This can be fixed with:

    zn "%SYS"
; enabling callin for Embedded Python
do ##class(Security.Services).Get("%Service_CallIn",.prop)
set prop("Enabled")=1
set prop("AutheEnabled")=48
do ##class(Security.Services).Modify("%Service_CallIn",.prop)

PyProd Setup

Set the following environment variables so Python can locate the IRIS libraries:

export IRISINSTALLDIR="/usr/irissys"
export LD_LIBRARY_PATH="$IRISINSTALLDIR/bin:$LD_LIBRARY_PATH"
export IRISUSERNAME="SuperUser"
export IRISPASSWORD="SYS"
export IRISNAMESPACE="ENSEMBLE"
export PYTHONPATH="$IRISINSTALLDIR/mgr/python"
export PATH="/usr/irissys/mgr/python/bin:$PATH"

Then install the PyProd package into the IRIS Python environment:

python3 -m pip install intersystems_pyprod --target /usr/irissys/mgr/python --update