AERSHED Platform

Appearance

Emission Import ​

This section outlines instructions on how to import data into the platform using helper scripts and the input format for the data.

GitHub Repository ​

Location of the scripts: github.com/aerscape/data-pipelines

Getting Started ​

Clone the repository ​

bash
git clone git@github.com:/aerscape/data-pipelines

Install dependencies ​

Install the requests library via pip:

bash
pip install requests

Getting the latest code ​

Before running, make sure to have the latest code and cd into the dataimport-scripts folder:

bash
git pull
cd dataimport-scripts

Quick Reference ​

Before running any script:

bash
cd {PROJECTS_ROOT}/data-pipelines/
git pull
cd dataimport-scripts

Common command examples:

bash
# Emissions Bridger
python -m importers.emissions.bridger --environment dev --api_key {API_KEY_VALUE} --data_path {PATH_TO_FOLDER} --data_source SELF_REPORTED

# Emissions Generic (Any data provider)
python -m importers.emissions.generic --environment staging --api_key {API_KEY_VALUE} --data_path {PATH_TO_FOLDER} --data_provider "Existing Data Provider" --contains_no_detects 1 --data_source SELF_REPORTED

# Aerial Images
python -m importers.infrastructure.aerial_images --environment prod --api_key {API_KEY_VALUE} --data_path {PATH_TO_FOLDER}

# Pipelines
python -m importers.infrastructure.pipelines --environment prod --api_key {API_KEY_VALUE} --data_path {PATH_TO_FOLDER} --owner {OWNER_NAME}

Available Import Scripts ​

Emission Imports ​

Infrastructure Imports ​

Common Script Arguments ​

The following arguments are required for each script:

  • --environment - Environment to import to: local, dev, prod-testing, or prod
  • --api_key - Retrieved from the platform data_import admin view
  • --data_path - Full path to the folder containing data import files

Preparing the Data ​

Each script expects a path to a folder containing a data package. The package must contain:

  • Exactly one CSV file formatted to the script specification. If there is no CSV file found or if there are multiple CSV files found, the script will fail.
  • Referenced files: If a TIFF image or other file is referenced in the CSV file, it must be present in the data package, otherwise the script will fail.

Important: CSV files cannot have byte order mark (Excel adds by default). Use :set nobomb in vim to remove.

Error Handling ​

The import process follows an all-or-nothing rule - it'll try to process the operations sequentially, and if anything fails, it rolls back the entire process.

The messages are all being stored in the database. For common errors and solutions, see the Troubleshooting guide.