about summary refs log tree commit diff
path: root/gn_libs/WIP/background-jobs-design-notes.md
blob: cca3793563059b638b1c50e2c29d206c4fc79a3b (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
# Background jobs — design notes

- **Feature:** background jobs (cross-cutting, shared across repos)
- **Last updated:** 2026-10-22
- **Related:** [tasks & todos](background-jobs-tasks-and-todos.md) ·
  genotypes upload docs in `gn-uploader/WIP/`
- **Status:** living design. The unification work below is a **major, not yet
  committed** undertaking.

`gn_libs.jobs` is intended to become the single, shared background-jobs
implementation for the GeneNetwork repositories. Today there are several
independent implementations; this note records the target design, the current
state, and the open questions. The migration checklist lives in
[tasks & todos](background-jobs-tasks-and-todos.md).

## Current state: several implementations

| Repo | Storage | Create/queue | Runner/launcher | Notes |
|---|---|---|---|---|
| gn-libs | SQLite | `gn_libs/jobs/jobs.py` (`initialise_job`, `launch_job`) | `gn_libs/jobs/launcher.py` | Intended shared implementation; migrations via `init_app`. |
| gn-uploader | Redis (legacy) **and** SQLite | legacy `uploader/jobs.py`; new `gn_libs.jobs` | legacy `scripts/worker.py`; new `gn_libs.jobs.launcher` | Mid-migration; both coexist. |
| genenetwork2 | Redis (`gn2:jobs`) | `gn2/jobs/jobs.py` (`queue`, `run`) | `gn2/scripts/run_external.py` | Redis hash per job; debug blueprint. |
| genenetwork3 | Redis (`GN3::JOBS` + command queue) | `gn3/commands.py` (`queue_cmd`, `run_async_cmd`); `gn3/jobs.py` (`create_job`) | `sheepdog/worker.py` | `gn3/jobs.py` appears unused — verify. |
| genenetwork4 | Redis (mirror of gn3) | same files as gn3 | `sheepdog/worker.py` | Essentially duplicated from gn3. |
| gn-auth | Redis | `gn_auth/jobs.py` (`create_job`); `gn_auth/commands.py` (`queue_cmd`, `run_async_cmd`) | `gn_auth/scripts/worker.py` | Similar to gn3 but diverged. |

Consequences: duplicated status models, two storage backends (Redis vs SQLite),
inconsistent metadata/output handling, and UI/status code repeated per repo.

## Target design

- **One implementation in `gn-libs`** for: job creation, storage, metadata,
  status, std-stream capture, listing/expiry, and launching/running.
- **Repos supply job _definitions_ and _handlers_**, not their own job engines.
- A **single runner/launcher** contract, so every repo launches jobs the same
  way (no per-repo `scripts/worker.py` / `run_external.py` copies).
- Consistent status vocabulary and metadata keys across repos.
- Backwards-compatible handling of existing jobs/external IDs where feasible.
- **Move away from Redis:** the Redis-backed implementations are to be adapted
  to the SQLite implementation for running jobs; `gn_libs.jobs` is the target.

### Decisions

- **Storage backend: SQLite.** We are moving away from Redis. `gn_libs.jobs`
  (SQLite) is the target implementation; the existing Redis-backed
  implementations must be adapted to use it for running jobs. No new
  Redis-backed job code.

### Open questions

- **Process model:** subprocess-per-job (current) vs a resident worker/queue.
- **Status vocabulary:** unify names (`queued`/`running`/`completed`/`error`/
  `stopped` vs `pending`/`success`/...).
- **External IDs:** how to preserve `jobs_by_external_id` semantics.
- **Migration path:** per-repo, behind the existing APIs, to avoid a big-bang
  change.

## Multi-task jobs (design consideration)

Today a background job is effectively **one script == one job**. Some features
(e.g. genotype uploads) are one logical operation made of several dependent
stages. Splitting them into unrelated single-script jobs loses the relationship
between steps and makes gating/status awkward.

### Option A — keep "one script == one job"

- Each stage is its own job; the client orchestrates the sequence.
- ✅ No change to the job model; smallest near-term risk.
- ❌ Several unrelated jobs for one operation.
- ❌ Readiness/success gating duplicated in clients.
- ❌ Coherent per-operation status is hard.

### Option B — one job containing multiple tasks

- A job owns tasks; each task has a status; the job tracks overall progress.
- ✅ Natural for multi-stage operations.
- ✅ Central readiness/gating and per-task success.
- ✅ Better user-visible status (e.g. "collect info done, awaiting input").
- ❌ Rework of storage schema, runner, handlers and UI.
- ❌ Scope creep.

### Recommendation

Do not couple this to any single feature. Treat it as part of the shared-jobs
effort, and in the meantime allow Option A (or a thin orchestration layer) so
features are not blocked. If adopted, a task model would need: task statuses
(`pending`/`ready`/`running`/`success`/`error`/blocked), dependency gating,
per-task outputs passed to later tasks, derived overall status, and
retry/"fix and resume" (lower priority).

## Inventory of launch/tracking points

The detailed, migration-oriented checklist (with file paths) lives in
[tasks & todos](background-jobs-tasks-and-todos.md#migration-inventory-do-not-miss-any-spot). Keeping
it there means the refactor has one checklist of places not to miss. Broadly,
the points are:

- **gn-uploader:** `uploader/jobs.py` + `scripts/worker.py` (legacy) and
  `uploader/background_jobs.py` + `gn_libs.jobs` (new); launch sites in
  `uploader/population/rqtl2.py`, `uploader/phenotypes/views.py`,
  `uploader/expression_data/{views,dbinsert}.py`, `uploader/samples/views.py`;
  worker-side status imports in several `scripts/`.
- **genenetwork2:** `gn2/jobs/jobs.py`, `gn2/scripts/run_external.py`, launch at
  `gn2/wqflask/views.py`, tracking UI in `gn2/wqflask/api/jobs.py` and
  `gn2/wqflask/templates/jobs/`.
- **genenetwork3/4:** `gn3/commands.py` + `sheepdog/worker.py`; launch sites in
  `gn3/api/gemma.py` and `gn3/api/correlation.py`; legacy `gn3/jobs.py`.
- **gn-auth:** `gn_auth/jobs.py`, `gn_auth/commands.py`,
  `gn_auth/scripts/worker.py`; launch at
  `gn_auth/auth/authorisation/data/views.py`; `gn_auth/scripts/search_phenotypes.py`.

## Related reading

- `gn-uploader/docs/dev/background_jobs.md` — earlier notes on per-user jobs,
  multiprocessing within jobs, and fetching job status.