Summary

12 DataSets containing 43 tables based on datasets from data.world and kaggle.com
Application to create tables dynamically based on the files in folder and Load data by using all-new LOAD DATA InterSystems IRIS 2021.2 Preview functionality.
DataSet can be installed or removed by terminal or by using web application. Data can be view and copied by using Web application.

Application Layout

Online Demo

https://medicaldatasets.demo.community.intersystems.com/csp/datasets/index.csp by using SuperUser | SYS

Datasets List

1 - Synthetic Medical Data : Synthetic medical record data for Introduction to Biomedical Data Science. LICENCE:Public Domain
2 - Health Care Analytics - 1 : Part of Janatahack Hackathon in Analytics Vidhya LICENCE:Public Domain
3 - Global Suicide Data : World Health Organization (WHO) dataset LICENCE:Public Domain
4 - COVID-19 Polls : Survey data to understand how Americans view the coronavirus pandemic and the president’s response LICENCE:CC-BY
5 - Cancer Rates by U.S. State : The rates are the numbers out of 100,000 people who developed or died from cancer each year. LICENCE:Public Domain
6 - Opioid Overdose Deaths : Opioid addiction and death rates in the U.S. and abroad have reached “epidemic” levels. The CDC’s data reflects the incredible spike in overdoses caused by drugs containing opioids. LICENCE:Public Domain
7 - Heart Disease Prediction : This data set came from the University of California Irvine data repository and is used to predict heart disease LICENCE:Public Domain
8 - Yellowpages medical services dataset : Yellowpages dataset with more than 400K+ records from Crawl Feeds. LICENCE:Public Domain
9 - Hospital ratings : This are the official datasets used on the Medicare.gov Hospital Compare Website provided by the Centers for Medicare & Medicaid Services. These data allow you to compare the quality of care at over 4,000 Medicare-certified hospitals across the country. LICENCE:Public Domain
10 -Heart Failure Prediction : 12 clinical features por predicting death events. LICENCE:Attribution 4.0 International (CC BY 4.0)
11 -Diabetes Dataset : This dataset is originally from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective is to predict based on diagnostic measurements whether a patient has diabetes. LICENCE:Public Domain
12 -Chronic Kidney Disease : This dataset is originally from UCI Machine Learning Repository. The objective of the dataset is to diagnostically predict whether a patient is having chronic kidney disease or not, based on certain diagnostic measurements included in the dataset. LICENCE:Public Domain

Features

Dynamically creation of tables based on the CSV file data.
Import dataset by using LOAD DATA functionality by Terminal or by web application.
Remove dataset programmatically by terminal or by Web Application.
Dynamically view Imported Data from Web Application.
Functionality to Import data into CSV, Excel or PDF Format.
Responsive web application to view the status of Dataset.
All of the datasets can be installed or removed by single command.
Usage of %SQL_Diag.Result and %SQL_Diag.Message tables with LOAD DATA

Recommendation

Read related documentations LOAD DATA (SQL).

How to Run

To start coding with this repo, you do the following:

Clone/git pull the repo into any local directory

git clone https://github.com/mwaseem75/DataSetProj.git

Open the terminal in this directory and run:

docker-compose build

Run the IRIS container with your project:

docker-compose up -d

Installation with ZPM

zpm "install dataset-medical"

Repo Contents

Dockerfile, docker-compose.yml, and Installer.cls to create container
iris.script, contains script to execute during container initialization
/src with source files
/.vscode/settings.json for automatic server connections when opened in VS Code.

Requirements:

Getting Started

Importing DataSet from Terminal

Connect to IRIS Terminal

docker-compose exec iris iris session iris

Change Namespace to DATASETS

ZN "DATASETS"

Run below command to import Synthetic Medical Data dataset by passing dataset ID(1)

do ##class(dc.data.medical.utility).ImportDS(1)

11 Tables are created, 83,341 Records Loaded successfully, 140 errors Reported

Use of %SQL_Diag.Result and %SQL_Diag.Message to check errors

As you noticed previous LOAD DATA returns 140 errors. To check the error details we will use %SQL_Diag.Result and %SQL_Diag.Message tables. We need diag ID to check detail which in our case is 5 (Last column in previous snapshot)

We will use Management Portal SQL to check the details

Open Management Portal SQL and type below SQL command to check details in %SQL_Diag.Result table against Diag ID

SELECT * FROM %SQL_Diag.Result WHERE ID = 5

and to check the details of 140 errors type below SQL command

SELECT * FROM %SQL_Diag.Message WHERE severity = 'error' and diagResult = 5

System is displaying details of 140 errors. Please note that we are passing same %SQL_Diag.Result ID to diagResulg column

Importing DataSet from Web Application

Navigate to http://localhost:52773/csp/datasets/index.csp and press Install DataSet button

Dataset is imported successfully

View Data from Web Application

Navigate to http://localhost:52773/csp/datasets/index.csp and click any table from sidebar

Export Data from Web Application

Navigate to View Datasets and select particular dataset, click Excel,CSV or PDF button to export data

Removing DataSet from Terminal

Run below command to Remove Synthetic Medical Data dataset by passing dataset ID(1)

do ##class(dc.data.medical.utility).RemoveDS(1)

Removing DataSet from Web Application

Navigate to http://localhost:52773/csp/datasets/index.csp and press Remove DataSet button

Dataset is removed successfully

Importing all DataSets at once

Run below command to Import all datasets by passing ID 999

do ##class(dc.data.medical.utility).ImportDS(999)

Removing all DataSets at once

Run below command to Remove all datasets by passing ID 999

do ##class(dc.data.medical.utility).RemoveDS(999)

Check Status of DataSet

Run below command to check the status of particular DataSet by passing DataSet ID,

do ##class(dc.data.medical.utility).GetStatusDS(1)

Run below command to check the status of all Datasets

do ##class(dc.data.medical.utility).GetStatusDS(999)

Other information

Template used in web application is from Bootstrap 4 Admin Dashboard and it is free to use to develop non-commercial applications.

Thanks

Medical Datasets Works

Experts nomination

InterSystems Datasets Contest

Community nomination

InterSystems Datasets Contest