From 4a59004bad69ea1bd2cadcf1e2c3446ccbd85be8 Mon Sep 17 00:00:00 2001 From: Munyoki Kilyungi Date: Tue, 23 Aug 2022 21:37:49 +0300 Subject: Add a design-doc for better logging --- topics/better-logging.gmi | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 topics/better-logging.gmi (limited to 'topics/better-logging.gmi') diff --git a/topics/better-logging.gmi b/topics/better-logging.gmi new file mode 100644 index 0000000..4c216aa --- /dev/null +++ b/topics/better-logging.gmi @@ -0,0 +1,38 @@ +# Improving Logging in GN2 + +## What Are We Trying To Solve? + +We prioritise maintaining user functionality over speed in GN [with time this speed will be improved]. As such we should be pay more attention at not breaking any currently working GN2 functionality. And when/if we do, trouble-shooting should be easy. On this front, one way is to stream-line logging in both GN2/GN3 and make it more script friendly - only report when something fails, not to instrument variables - and in so doing make the process of monitoring easier. + +## Goals + +- Remove noise from GN2. + +- Separate logging into different files: error logs, info logs. Add this somewhere with Flask itself instead of re-directing STDOUT to a file. + +### Non-goals + +- Logging in GN3. + +- Parsing logs to extract goals. + +- Getting rid of "gn.db" global object and in so doing removing "MySqlAlchemy" [that we really shouldn't be using]. + +- Adding log messages to existing functions. + +## Actual Design + +- Configure logger to separate logs into different files: + - INFO file - contains initial bootstrap messages and instrumentation we may have to do. + - ERROR file - contains real errors. + +- Have those settings - the different log files - be part of GN2 start-up script. + +- For error messages, use the following format for different error messages: + - "DATABASE: " + - "REDIS: " + - "COMPUTATION: " + - "API: " + - "MISC: " + +- Have time-stamped logs by month. E.g. "genenetwork2-08-2022.error.log" and "genenetwork2-08-2022.info.log". This way in future, we can actually run an analysis on what breaks often in GN2. -- cgit v1.2.3 From cfdcd069af40a0cc55014c4ba8c154b41d8cce87 Mon Sep 17 00:00:00 2001 From: Munyoki Kilyungi Date: Tue, 23 Aug 2022 22:14:25 +0300 Subject: Fix sub-lists in "better-logging" --- topics/better-logging.gmi | 7 +++++++ 1 file changed, 7 insertions(+) (limited to 'topics/better-logging.gmi') diff --git a/topics/better-logging.gmi b/topics/better-logging.gmi index 4c216aa..9ca8e42 100644 --- a/topics/better-logging.gmi +++ b/topics/better-logging.gmi @@ -23,16 +23,23 @@ We prioritise maintaining user functionality over speed in GN [with time this sp ## Actual Design - Configure logger to separate logs into different files: + - INFO file - contains initial bootstrap messages and instrumentation we may have to do. + - ERROR file - contains real errors. - Have those settings - the different log files - be part of GN2 start-up script. - For error messages, use the following format for different error messages: + - "DATABASE: " + - "REDIS: " + - "COMPUTATION: " + - "API: " + - "MISC: " - Have time-stamped logs by month. E.g. "genenetwork2-08-2022.error.log" and "genenetwork2-08-2022.info.log". This way in future, we can actually run an analysis on what breaks often in GN2. -- cgit v1.2.3 From 6eae4be03f06c29ceb9f167d6b95653378f39087 Mon Sep 17 00:00:00 2001 From: Munyoki Kilyungi Date: Tue, 23 Aug 2022 23:15:34 +0300 Subject: Update better-logging topic --- topics/better-logging.gmi | 4 ++++ 1 file changed, 4 insertions(+) (limited to 'topics/better-logging.gmi') diff --git a/topics/better-logging.gmi b/topics/better-logging.gmi index 9ca8e42..ea9fd26 100644 --- a/topics/better-logging.gmi +++ b/topics/better-logging.gmi @@ -6,6 +6,8 @@ We prioritise maintaining user functionality over speed in GN [with time this sp ## Goals +- Have script-friendly error/info logs. + - Remove noise from GN2. - Separate logging into different files: error logs, info logs. Add this somewhere with Flask itself instead of re-directing STDOUT to a file. @@ -43,3 +45,5 @@ We prioritise maintaining user functionality over speed in GN [with time this sp - "MISC: " - Have time-stamped logs by month. E.g. "genenetwork2-08-2022.error.log" and "genenetwork2-08-2022.info.log". This way in future, we can actually run an analysis on what breaks often in GN2. + +- Get rid of "utility.logger" module and replace it with Flask's or Python's in-built logging. -- cgit v1.2.3 From 185ab8cc3e621f3257b525a8ca5ae7a2281d055e Mon Sep 17 00:00:00 2001 From: Munyoki Kilyungi Date: Mon, 29 Aug 2022 12:32:05 +0300 Subject: Update topic on better logging. * topics/better-logging.gmi: Update design-doc. --- topics/better-logging.gmi | 44 +++++++++++--------------------------------- 1 file changed, 11 insertions(+), 33 deletions(-) (limited to 'topics/better-logging.gmi') diff --git a/topics/better-logging.gmi b/topics/better-logging.gmi index ea9fd26..8de3fb3 100644 --- a/topics/better-logging.gmi +++ b/topics/better-logging.gmi @@ -6,44 +6,22 @@ We prioritise maintaining user functionality over speed in GN [with time this sp ## Goals -- Have script-friendly error/info logs. - -- Remove noise from GN2. - -- Separate logging into different files: error logs, info logs. Add this somewhere with Flask itself instead of re-directing STDOUT to a file. +* Have script-friendly error/info logs. +* Remove noise from GN2. +* Separate logging into different files: error logs, info logs. Add this somewhere with Flask itself instead of re-directing STDOUT to a file. ### Non-goals -- Logging in GN3. - -- Parsing logs to extract goals. - -- Getting rid of "gn.db" global object and in so doing removing "MySqlAlchemy" [that we really shouldn't be using]. - -- Adding log messages to existing functions. +* Logging in GN3. +* Parsing logs to extract goals. +* Getting rid of "gn.db" global object and in so doing removing "MySqlAlchemy" [that we really shouldn't be using]. +* Adding log messages to existing functions. ## Actual Design -- Configure logger to separate logs into different files: - - - INFO file - contains initial bootstrap messages and instrumentation we may have to do. - - - ERROR file - contains real errors. - -- Have those settings - the different log files - be part of GN2 start-up script. - -- For error messages, use the following format for different error messages: - - - "DATABASE: " - - - "REDIS: " - - - "COMPUTATION: " - - - "API: " - - - "MISC: " +* Get rid of "utility.logger" module and replace it with Flask's or Python's in-built logging. +* Configure the logging system to automatically add the module name, line number, time-stamps etc. -- Have time-stamped logs by month. E.g. "genenetwork2-08-2022.error.log" and "genenetwork2-08-2022.info.log". This way in future, we can actually run an analysis on what breaks often in GN2. +## Resources -- Get rid of "utility.logger" module and replace it with Flask's or Python's in-built logging. +=> https://realpython.com/python-logging/ Logging in Python -- cgit v1.2.3