# Background jobs — design notes - **Feature:** background jobs (cross-cutting, shared across repos) - **Last updated:** 2026-10-22 - **Related:** [tasks & todos](background-jobs-tasks-and-todos.md) · genotypes upload docs in `gn-uploader/WIP/` - **Status:** living design. The unification work below is a **major, not yet committed** undertaking. `gn_libs.jobs` is intended to become the single, shared background-jobs implementation for the GeneNetwork repositories. Today there are several independent implementations; this note records the target design, the current state, and the open questions. The migration checklist lives in [tasks & todos](background-jobs-tasks-and-todos.md). ## Current state: several implementations | Repo | Storage | Create/queue | Runner/launcher | Notes | |---|---|---|---|---| | gn-libs | SQLite | `gn_libs/jobs/jobs.py` (`initialise_job`, `launch_job`) | `gn_libs/jobs/launcher.py` | Intended shared implementation; migrations via `init_app`. | | gn-uploader | Redis (legacy) **and** SQLite | legacy `uploader/jobs.py`; new `gn_libs.jobs` | legacy `scripts/worker.py`; new `gn_libs.jobs.launcher` | Mid-migration; both coexist. | | genenetwork2 | Redis (`gn2:jobs`) | `gn2/jobs/jobs.py` (`queue`, `run`) | `gn2/scripts/run_external.py` | Redis hash per job; debug blueprint. | | genenetwork3 | Redis (`GN3::JOBS` + command queue) | `gn3/commands.py` (`queue_cmd`, `run_async_cmd`); `gn3/jobs.py` (`create_job`) | `sheepdog/worker.py` | `gn3/jobs.py` appears unused — verify. | | genenetwork4 | Redis (mirror of gn3) | same files as gn3 | `sheepdog/worker.py` | Essentially duplicated from gn3. | | gn-auth | Redis | `gn_auth/jobs.py` (`create_job`); `gn_auth/commands.py` (`queue_cmd`, `run_async_cmd`) | `gn_auth/scripts/worker.py` | Similar to gn3 but diverged. | Consequences: duplicated status models, two storage backends (Redis vs SQLite), inconsistent metadata/output handling, and UI/status code repeated per repo. ## Target design - **One implementation in `gn-libs`** for: job creation, storage, metadata, status, std-stream capture, listing/expiry, and launching/running. - **Repos supply job _definitions_ and _handlers_**, not their own job engines. - A **single runner/launcher** contract, so every repo launches jobs the same way (no per-repo `scripts/worker.py` / `run_external.py` copies). - Consistent status vocabulary and metadata keys across repos. - Backwards-compatible handling of existing jobs/external IDs where feasible. - **Move away from Redis:** the Redis-backed implementations are to be adapted to the SQLite implementation for running jobs; `gn_libs.jobs` is the target. ### Decisions - **Storage backend: SQLite.** We are moving away from Redis. `gn_libs.jobs` (SQLite) is the target implementation; the existing Redis-backed implementations must be adapted to use it for running jobs. No new Redis-backed job code. ### Open questions - **Process model:** subprocess-per-job (current) vs a resident worker/queue. - **Status vocabulary:** unify names (`queued`/`running`/`completed`/`error`/ `stopped` vs `pending`/`success`/...). - **External IDs:** how to preserve `jobs_by_external_id` semantics. - **Migration path:** per-repo, behind the existing APIs, to avoid a big-bang change. ## Multi-task jobs (design consideration) Today a background job is effectively **one script == one job**. Some features (e.g. genotype uploads) are one logical operation made of several dependent stages. Splitting them into unrelated single-script jobs loses the relationship between steps and makes gating/status awkward. ### Option A — keep "one script == one job" - Each stage is its own job; the client orchestrates the sequence. - ✅ No change to the job model; smallest near-term risk. - ❌ Several unrelated jobs for one operation. - ❌ Readiness/success gating duplicated in clients. - ❌ Coherent per-operation status is hard. ### Option B — one job containing multiple tasks - A job owns tasks; each task has a status; the job tracks overall progress. - ✅ Natural for multi-stage operations. - ✅ Central readiness/gating and per-task success. - ✅ Better user-visible status (e.g. "collect info done, awaiting input"). - ❌ Rework of storage schema, runner, handlers and UI. - ❌ Scope creep. ### Recommendation Do not couple this to any single feature. Treat it as part of the shared-jobs effort, and in the meantime allow Option A (or a thin orchestration layer) so features are not blocked. If adopted, a task model would need: task statuses (`pending`/`ready`/`running`/`success`/`error`/blocked), dependency gating, per-task outputs passed to later tasks, derived overall status, and retry/"fix and resume" (lower priority). ## Inventory of launch/tracking points The detailed, migration-oriented checklist (with file paths) lives in [tasks & todos](background-jobs-tasks-and-todos.md#migration-inventory-do-not-miss-any-spot). Keeping it there means the refactor has one checklist of places not to miss. Broadly, the points are: - **gn-uploader:** `uploader/jobs.py` + `scripts/worker.py` (legacy) and `uploader/background_jobs.py` + `gn_libs.jobs` (new); launch sites in `uploader/population/rqtl2.py`, `uploader/phenotypes/views.py`, `uploader/expression_data/{views,dbinsert}.py`, `uploader/samples/views.py`; worker-side status imports in several `scripts/`. - **genenetwork2:** `gn2/jobs/jobs.py`, `gn2/scripts/run_external.py`, launch at `gn2/wqflask/views.py`, tracking UI in `gn2/wqflask/api/jobs.py` and `gn2/wqflask/templates/jobs/`. - **genenetwork3/4:** `gn3/commands.py` + `sheepdog/worker.py`; launch sites in `gn3/api/gemma.py` and `gn3/api/correlation.py`; legacy `gn3/jobs.py`. - **gn-auth:** `gn_auth/jobs.py`, `gn_auth/commands.py`, `gn_auth/scripts/worker.py`; launch at `gn_auth/auth/authorisation/data/views.py`; `gn_auth/scripts/search_phenotypes.py`. ## Related reading - `gn-uploader/docs/dev/background_jobs.md` — earlier notes on per-user jobs, multiprocessing within jobs, and fetching job status.