Celery Scheduled Tasks Documentation
This document describes the periodic tasks configured in celery.py.
Important: Changes to Waffle switches or Django settings that affect task scheduling require a container restart to take effect. The periodic task configuration is evaluated once when the Celery worker starts.
Task Queues
The following queues are configured:
| Queue | Purpose |
|---|---|
email | Email-related tasks |
longrunning | Default queue for long-running tasks |
dataimport | Data import tasks |
dataexport | Data export tasks |
Waffle Switches
The following Waffle switches control task scheduling:
| Switch Name | Description |
|---|---|
enable_periodic_tasks | Master switch that enables all periodic tasks (except queue health checks) |
epa_sep_scraped | Enables EPA SEP scraped data import task |
workflow-automation-enabled | Enables the workflow manager task |
Scheduled Tasks
Always Active (No Switch Required)
These tasks run regardless of the enable_periodic_tasks switch:
Queue Health Checks
| Task | Interval | Queue | Condition |
|---|---|---|---|
queue_health_check_task | Every 2 minutes | email | Requires WORKER_EMAIL_QUEUE_TEST_URL setting |
queue_health_check_task | Every 30 minutes | longrunning | Requires WORKER_LONGRUNNING_QUEUE_TEST_URL setting |
Requires enable_periodic_tasks Switch
All tasks below require the enable_periodic_tasks Waffle switch to be active.
Virus Scanning
| Task | Interval | Source |
|---|---|---|
check_virus_scan_complete | Every 2 minutes | data_uploads.tasks |
check_downloadable_virus_scan_complete | Every 30 seconds | data_downloads.tasks |
Image Processing
| Task | Interval | Source |
|---|---|---|
check_images_and_process | Every hour | rasters.tasks |
Backups
| Task | Interval | Source |
|---|---|---|
backup_task | Weekly (every 7 days) | backups.tasks |
Notifications
| Task | Interval | Source |
|---|---|---|
send_pending_notifications | Every 60 seconds | notifications.tasks |
Data Import Tasks (Daily at 8:00 AM)
| Task | Schedule | Condition | Source |
|---|---|---|---|
import_emit_data | 8:00 AM daily | Requires AWS_NASA_EMIT_STORAGE_BUCKET_NAME setting | data_import.tasks |
import_edf_methaneair_data | 8:00 AM daily | Requires DATA_IMPORT_S3_ACCESS_KEY_ID setting | data_import.tasks |
import_unep_data | 8:00 AM daily | Always (when periodic tasks enabled) | data_import.tasks |
import_carbonmapper_data | 8:00 AM daily | Always (when periodic tasks enabled) | data_import.tasks |
import_californiasmp_data | 8:00 AM daily | Always (when periodic tasks enabled) | data_import.tasks |
import_epa_sep_scraped_data | 8:00 AM daily | Requires epa_sep_scraped switch | data_import.tasks |
import_carbonmapper_scenes | 8:15 AM daily | Always (when periodic tasks enabled) | data_import.tasks |
Non-Detect Calculations
| Task | Schedule | Source |
|---|---|---|
recalculate_non_detects | 8:30 AM daily (modified only) | nondetects.tasks |
recalculate_non_detects (full) | 6:30 AM every 2 days | nondetects.tasks |
EPA SEP
| Task | Schedule | Source |
|---|---|---|
send_epa_reminders | 12:00 PM daily | epa_sep.tasks |
User Management
| Task | Schedule | Source |
|---|---|---|
remove_old_temporary_users | Every 2 hours | custom_auth.tasks |
Infrastructure
| Task | Schedule | Source |
|---|---|---|
try_matching_orphan_images | Midnight daily | infrastructure.tasks |
Auditing
| Task | Schedule | Source |
|---|---|---|
audit_emission_matching_runs | Every 15 minutes | emission_matching.tasks |
audit_data_import_runs | Every 15 minutes | data_import.tasks |
Workflows
| Task | Schedule | Condition | Source |
|---|---|---|---|
run_workflow_manager | Every 5 minutes | Requires workflow-automation-enabled switch | workflows.tasks |
Sensor Data
| Task | Schedule | Source |
|---|---|---|
remove_old_sensor_data_requests | 12:30 AM daily | sensor_data.tasks |
Conditional Schedule Summary
Settings-Based Conditions
| Setting | Tasks Affected |
|---|---|
WORKER_EMAIL_QUEUE_TEST_URL | Email queue health check |
WORKER_LONGRUNNING_QUEUE_TEST_URL | Longrunning queue health check |
AWS_NASA_EMIT_STORAGE_BUCKET_NAME | EMIT data import |
DATA_IMPORT_S3_ACCESS_KEY_ID | EDF MethaneAir data import |
Switch-Based Conditions
| Switch | Tasks Affected |
|---|---|
enable_periodic_tasks | All tasks except queue health checks |
epa_sep_scraped | EPA SEP scraped data import |
workflow-automation-enabled | Workflow manager |
Applying Changes
To apply changes to switches or settings that affect scheduling:
- Update the Waffle switch in Django admin or the setting in target environment
- Restart the Celery worker container for changes to take effect
The setup_periodic_tasks function runs once during worker initialization via the @app.on_after_finalize.connect decorator, so runtime changes to switches or settings will not affect the schedule until the worker is restarted.