summaryrefslogtreecommitdiff
path: root/issues/gemma
diff options
context:
space:
mode:
authorPjotr Prins2021-11-25 09:25:55 +0100
committerPjotr Prins2021-11-25 09:25:55 +0100
commitd326fd0036a43b4e2b1edfe8137ff5c36570d221 (patch)
tree1801fc7ee454c7c2d8462f94570b2e24e4062a9d /issues/gemma
parentc97aa198bf95cb59ece70f3d8460d4537800e92c (diff)
downloadgn-gemtext-d326fd0036a43b4e2b1edfe8137ff5c36570d221.tar.gz
Fixed incomplete output for parallel gemma
Diffstat (limited to 'issues/gemma')
-rw-r--r--issues/gemma/gemma-wrapper-has-incomplete-files.gmi8
1 files changed, 7 insertions, 1 deletions
diff --git a/issues/gemma/gemma-wrapper-has-incomplete-files.gmi b/issues/gemma/gemma-wrapper-has-incomplete-files.gmi
index 6cb92f6..d530fb4 100644
--- a/issues/gemma/gemma-wrapper-has-incomplete-files.gmi
+++ b/issues/gemma/gemma-wrapper-has-incomplete-files.gmi
@@ -8,7 +8,7 @@ Gemma wrapper caches files - but it can happen a cached file is incomplete and n
## Tasks
-* [ ] parse parallel job log for failed tasks and remove the output files.
+* [X] parse parallel job log for failed tasks and remove the output files.
* [X] create a (global) lock file for gemma-wrapper
## Info
@@ -23,6 +23,12 @@ The problem is that it is NOT a catch all. If there is a hardware fault or a pro
It turns out that GNU parallel can keep track of jobs in a job log - and even rerun the ones missing using the `--joblog` and `--resume` switches. The last we don't need because we are using a cache. But we can use the log file to remove any incomplete output files! To me this is the obvious solution because 'parallel' is monitoring outside the GEMMA process and is a hardened piece of software. On failure it simply designates runs that way and we can clean up any (partly) produced files followed by a safe rerun. The lock routine below ascertains no processes are creating the same output at the same time.
+## Delete files on failure
+
+Implemented in
+
+=> https://github.com/genetics-statistics/gemma-wrapper/commit/624ed0d805f29ab682cffbe46bc104dffd0d713c
+
## Dealing with locks
There is another parallel issue (pun intended) where gemma-wrapper is invoked twice for the same job. This is quite possible when people get impatient waiting for a first job to finish.