AERSHED Platform

Appearance

Emission Import ​

This section outlines instructions on how to import data into the platform using helper scripts and the input format for the data.

GitHub Repository ​

Location of the scripts: github.com/aerscape/data-pipelines

Getting Started ​

Clone the repository ​

bash
git clone git@github.com:/aerscape/data-pipelines

Install dependencies ​

Install the requests library via pip:

bash
pip install requests

Getting the latest code ​

Before running, make sure to have the latest code and cd into the dataimport-scripts folder:

bash
git pull
cd dataimport-scripts

Quick Reference ​

Before running any script:

bash
cd {PROJECTS_ROOT}/data-pipelines/
git pull
cd dataimport-scripts

Common command examples:

bash
# Emissions Bridger
python -m importers.emissions.bridger --environment dev --api_key {API_KEY_VALUE} --data_path {PATH_TO_FOLDER} --data_source SELF_REPORTED

# Emissions Generic (Any data provider)
python -m importers.emissions.generic --environment staging --api_key {API_KEY_VALUE} --data_path {PATH_TO_FOLDER} --data_provider "Existing Data Provider" --contains_no_detects 1 --data_source SELF_REPORTED

# Aerial Images
python -m importers.infrastructure.aerial_images --environment prod --api_key {API_KEY_VALUE} --data_path {PATH_TO_FOLDER}

# Pipelines
python -m importers.infrastructure.pipelines --environment prod --api_key {API_KEY_VALUE} --data_path {PATH_TO_FOLDER} --owner {OWNER_NAME}

# ChampionX surveys
python -m importers.extra_data.championx --environment staging --api_key {API_KEY_VALUE} --data_path {PATH_TO_FOLDER}

Available Import Scripts ​

Emission Imports ​

Survey / Extra Data Imports ​

  • ChampionX - ChampionX flight survey data (DataSurvey)

Infrastructure Imports ​

Common Script Arguments ​

The following arguments are required for each script:

  • --environment - Environment to import to: local, dev, prod-testing, or prod
  • --api_key - Retrieved from the platform data_import admin view
  • --data_path - Full path to the folder containing data import files

Preparing the Data ​

Each script expects a path to a folder containing a data package. The package must contain:

  • Exactly one CSV file formatted to the script specification. If there is no CSV file found or if there are multiple CSV files found, the script will fail.
  • Referenced files: If a TIFF image or other file is referenced in the CSV file, it must be present in the data package, otherwise the script will fail.

Exception: The ChampionX importer takes a multi-CSV folder (a surveys.csv manifest plus one point CSV per survey), not a single CSV.

Important: CSV files cannot have byte order mark (Excel adds by default). Use :set nobomb in vim to remove.

Updating Existing Records ​

Imports can update detections that are already in the platform, not just create new ones. A row is treated as an update when it carries a record_id column matching a detection already stored for the same data provider.

Identity is record_id scoped by owner:

  • If the row resolves to an owner (an owner column, or the import's --owner), it can only update a point belonging to that same owner. The same record_id under a different owner is a different detection and is imported as a new point.
  • If the row resolves to no owner, it matches on record_id alone.

When a row updates an existing point:

  • The point is updated in place, a history entry is written with the previous values, and its version is bumped (ORIGINAL → V2 → V3 …).
  • The point stays in its original data batch — it is not moved into the new import's batch.
  • The data field is replaced wholesale, so an update row must carry all of the extra columns the original import supplied, or they are lost.
  • Plume images are not replaced by default. Pass --overwrite-plumes-on-update to replace them.

Note: matching against infrastructure is not re-run for points that already have an emission record. An update that changes latitude/longitude will not move an existing match to a different site.

Error Handling ​

The import process follows an all-or-nothing rule - it'll try to process the operations sequentially, and if anything fails, it rolls back the entire process.

The messages are all being stored in the database. For common errors and solutions, see the Troubleshooting guide.