From 1d1d0f4f448a2b0e15f10a3a7030d93f955cb421 Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Sat, 20 Nov 2021 12:45:16 +0100 Subject: gemma-wrapper: add lock support --- issues/gemma/gemma-wrapper-has-incomplete-files.gmi | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/issues/gemma/gemma-wrapper-has-incomplete-files.gmi b/issues/gemma/gemma-wrapper-has-incomplete-files.gmi index 4bea71d..3c9a7ad 100644 --- a/issues/gemma/gemma-wrapper-has-incomplete-files.gmi +++ b/issues/gemma/gemma-wrapper-has-incomplete-files.gmi @@ -17,6 +17,10 @@ GNU parallel can fail, but does not tell how individual processes did. Need to c Turns out GNU parallel can keep track of jobs in a job log - and even rerun the ones missing. The last we don't need because we are using a cache. But we can use the log file to remove any incomplete output files! +## Dealing with locks + There is another parallel issue (pun intended) where gemma-wrapper is invoked twice for the same job. This is quite possible when people get impatient waiting for a first job to finish. One solution is to write a lock file using the inputs as a hash. The lock file can contain a PID and we can check if that is still alive. I should do the same for sheepdog locks(!) + +=> https://github.com/genetics-statistics/gemma-wrapper/commit/e7e516ec5a6ffc5b398302fa204685a40e76e171 Added locking support -- cgit v1.2.3