1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
|
# Background jobs — design notes
- **Feature:** background jobs (cross-cutting, shared across repos)
- **Last updated:** 2026-10-22
- **Related:** [tasks & todos](background-jobs-tasks-and-todos.md) ·
genotypes upload docs in `gn-uploader/WIP/`
- **Status:** living design. The unification work below is a **major, not yet
committed** undertaking.
`gn_libs.jobs` is intended to become the single, shared background-jobs
implementation for the GeneNetwork repositories. Today there are several
independent implementations; this note records the target design, the current
state, and the open questions. The migration checklist lives in
[tasks & todos](background-jobs-tasks-and-todos.md).
## Current state: several implementations
| Repo | Storage | Create/queue | Runner/launcher | Notes |
|---|---|---|---|---|
| gn-libs | SQLite | `gn_libs/jobs/jobs.py` (`initialise_job`, `launch_job`) | `gn_libs/jobs/launcher.py` | Intended shared implementation; migrations via `init_app`. |
| gn-uploader | Redis (legacy) **and** SQLite | legacy `uploader/jobs.py`; new `gn_libs.jobs` | legacy `scripts/worker.py`; new `gn_libs.jobs.launcher` | Mid-migration; both coexist. |
| genenetwork2 | Redis (`gn2:jobs`) | `gn2/jobs/jobs.py` (`queue`, `run`) | `gn2/scripts/run_external.py` | Redis hash per job; debug blueprint. |
| genenetwork3 | Redis (`GN3::JOBS` + command queue) | `gn3/commands.py` (`queue_cmd`, `run_async_cmd`); `gn3/jobs.py` (`create_job`) | `sheepdog/worker.py` | `gn3/jobs.py` appears unused — verify. |
| genenetwork4 | Redis (mirror of gn3) | same files as gn3 | `sheepdog/worker.py` | Essentially duplicated from gn3. |
| gn-auth | Redis | `gn_auth/jobs.py` (`create_job`); `gn_auth/commands.py` (`queue_cmd`, `run_async_cmd`) | `gn_auth/scripts/worker.py` | Similar to gn3 but diverged. |
Consequences: duplicated status models, two storage backends (Redis vs SQLite),
inconsistent metadata/output handling, and UI/status code repeated per repo.
## Target design
- **One implementation in `gn-libs`** for: job creation, storage, metadata,
status, std-stream capture, listing/expiry, and launching/running.
- **Repos supply job _definitions_ and _handlers_**, not their own job engines.
- A **single runner/launcher** contract, so every repo launches jobs the same
way (no per-repo `scripts/worker.py` / `run_external.py` copies).
- Consistent status vocabulary and metadata keys across repos.
- Backwards-compatible handling of existing jobs/external IDs where feasible.
- **Move away from Redis:** the Redis-backed implementations are to be adapted
to the SQLite implementation for running jobs; `gn_libs.jobs` is the target.
### Decisions
- **Storage backend: SQLite.** We are moving away from Redis. `gn_libs.jobs`
(SQLite) is the target implementation; the existing Redis-backed
implementations must be adapted to use it for running jobs. No new
Redis-backed job code.
### Open questions
- **Process model:** subprocess-per-job (current) vs a resident worker/queue.
- **Status vocabulary:** unify names (`queued`/`running`/`completed`/`error`/
`stopped` vs `pending`/`success`/...).
- **External IDs:** how to preserve `jobs_by_external_id` semantics.
- **Migration path:** per-repo, behind the existing APIs, to avoid a big-bang
change.
## Multi-task jobs (design consideration)
Today a background job is effectively **one script == one job**. Some features
(e.g. genotype uploads) are one logical operation made of several dependent
stages. Splitting them into unrelated single-script jobs loses the relationship
between steps and makes gating/status awkward.
### Option A — keep "one script == one job"
- Each stage is its own job; the client orchestrates the sequence.
- ✅ No change to the job model; smallest near-term risk.
- ❌ Several unrelated jobs for one operation.
- ❌ Readiness/success gating duplicated in clients.
- ❌ Coherent per-operation status is hard.
### Option B — one job containing multiple tasks
- A job owns tasks; each task has a status; the job tracks overall progress.
- ✅ Natural for multi-stage operations.
- ✅ Central readiness/gating and per-task success.
- ✅ Better user-visible status (e.g. "collect info done, awaiting input").
- ❌ Rework of storage schema, runner, handlers and UI.
- ❌ Scope creep.
### Recommendation
Do not couple this to any single feature. Treat it as part of the shared-jobs
effort, and in the meantime allow Option A (or a thin orchestration layer) so
features are not blocked. If adopted, a task model would need: task statuses
(`pending`/`ready`/`running`/`success`/`error`/blocked), dependency gating,
per-task outputs passed to later tasks, derived overall status, and
retry/"fix and resume" (lower priority).
## Inventory of launch/tracking points
The detailed, migration-oriented checklist (with file paths) lives in
[tasks & todos](background-jobs-tasks-and-todos.md#migration-inventory-do-not-miss-any-spot). Keeping
it there means the refactor has one checklist of places not to miss. Broadly,
the points are:
- **gn-uploader:** `uploader/jobs.py` + `scripts/worker.py` (legacy) and
`uploader/background_jobs.py` + `gn_libs.jobs` (new); launch sites in
`uploader/population/rqtl2.py`, `uploader/phenotypes/views.py`,
`uploader/expression_data/{views,dbinsert}.py`, `uploader/samples/views.py`;
worker-side status imports in several `scripts/`.
- **genenetwork2:** `gn2/jobs/jobs.py`, `gn2/scripts/run_external.py`, launch at
`gn2/wqflask/views.py`, tracking UI in `gn2/wqflask/api/jobs.py` and
`gn2/wqflask/templates/jobs/`.
- **genenetwork3/4:** `gn3/commands.py` + `sheepdog/worker.py`; launch sites in
`gn3/api/gemma.py` and `gn3/api/correlation.py`; legacy `gn3/jobs.py`.
- **gn-auth:** `gn_auth/jobs.py`, `gn_auth/commands.py`,
`gn_auth/scripts/worker.py`; launch at
`gn_auth/auth/authorisation/data/views.py`; `gn_auth/scripts/search_phenotypes.py`.
## Related reading
- `gn-uploader/docs/dev/background_jobs.md` — earlier notes on per-user jobs,
multiprocessing within jobs, and fetching job status.
|