diff options
| author | Pjotr Prins | 2021-11-20 12:45:16 +0100 |
|---|---|---|
| committer | Pjotr Prins | 2021-11-20 12:45:16 +0100 |
| commit | 1d1d0f4f448a2b0e15f10a3a7030d93f955cb421 (patch) | |
| tree | eda94f6f67f627747a9920fd7bc887b1c5417b3d | |
| parent | 78143cb46fb98b2152d9e40d5f885e5eea2414c3 (diff) | |
| download | gn-gemtext-1d1d0f4f448a2b0e15f10a3a7030d93f955cb421.tar.gz | |
gemma-wrapper: add lock support
| -rw-r--r-- | issues/gemma/gemma-wrapper-has-incomplete-files.gmi | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/issues/gemma/gemma-wrapper-has-incomplete-files.gmi b/issues/gemma/gemma-wrapper-has-incomplete-files.gmi index 4bea71d..3c9a7ad 100644 --- a/issues/gemma/gemma-wrapper-has-incomplete-files.gmi +++ b/issues/gemma/gemma-wrapper-has-incomplete-files.gmi @@ -17,6 +17,10 @@ GNU parallel can fail, but does not tell how individual processes did. Need to c Turns out GNU parallel can keep track of jobs in a job log - and even rerun the ones missing. The last we don't need because we are using a cache. But we can use the log file to remove any incomplete output files! +## Dealing with locks + There is another parallel issue (pun intended) where gemma-wrapper is invoked twice for the same job. This is quite possible when people get impatient waiting for a first job to finish. One solution is to write a lock file using the inputs as a hash. The lock file can contain a PID and we can check if that is still alive. I should do the same for sheepdog locks(!) + +=> https://github.com/genetics-statistics/gemma-wrapper/commit/e7e516ec5a6ffc5b398302fa204685a40e76e171 Added locking support |