Home Applications iris-parquet

iris-parquet

This application is not supported by InterSystems Corporation. Please be notified that you use it at your own risk.
4.5
1 reviews
1
Awards
198
Views
2
IPM installs
0
0
Details
Releases
Reviews
Awards
Issues
Pull requests
Videos
Articles
Dependencies
This app has a demo View
Read and write files and datasets between InterSystems IRIS and Parquet

What's new in this version

Initial Release

Gitter
Quality Gate Status
Reliability Rating

License: MIT

Iris-Parquet

This is a tool to generate parquet files from IRIS data or load parquet data on IRIS data.

Description

The Iris-Parquet allows you:

  • Generate parquet files from IRIS SQL instruction
  • Generate JSON from Parquet file to allow you save it on IRIS SQL tables or JSON Documents

Prerequisites

  • HADOOP_HOME configured to Hadoop folder

Installation with Docker

Clone/git pull the repo into any local directory

$ git clone https://github.com/yurimarx/iris-parquet.git

Open the terminal in this directory and call the command to build and run InterSystems IRIS in container:
Note: Users running containers on a Linux CLI, should use “docker compose” instead of “docker-compose”
See Install the Compose plugin

$ docker-compose build
$ docker-compose up -d

Installation with ZPM

USER> zpm install iris-parquet

Install hadoop files and set ENV variable to HADOOP_HOME:

wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz && \
    tar -xzf hadoop-3.3.6.tar.gz && \
    echo "export HADOOP_HOME=/<unzipped folder>/hadoop-3.3.6"

Testing using Swagger-ui

  1. Go to http://:/swagger-ui/index.html
  2. On field Explore insert http://:/parquet-api/_spec
  3. For online sample it is:

Testing using Postman

  1. Open the file IRISParquet.postman_collection.json (or download from iris parquet postman)
  2. Set the variables server (iris webserver host) and port (iris webserver port) on Variables tab of the collection
  3. Run the method /generate-persons one or more to generate sample person fake data
  4. Run the method /sql2parquet with this query on body: select * from dc_irisparquet.SamplePerson
  5. Download the parquet file on the link Download file
  6. Run the method /parquet2json to the parquet file generated on the past step and the results
  7. You can also open the Parquet file on VSCode (install the parquet-viewer extension to see the parquet content from VSCode - https://marketplace.visualstudio.com/items?itemName=dvirtz.parquet-viewer)
Read more
Made with
Install
zpm install iris-parquet download archive
Version
1.0.002 Dec, 2023
ObjectScript quality test
Category
Solutions
Works with
InterSystems IRISInterSystems IRIS for HealthHealthShare
First published
19 Nov, 2023