Correlation api (#2)

* add file for correlation api

* register initial correlation api

* add correlation package

* add function  for getting page data

* delete loading page api

* modify code for correlation

* add tests folder for correlations

* fix error in correlation api

* add tests for correlation

* add tests for  correlation loading data

* add module for correlation computations

* modify api to return json when computing correlation

* add tests for computing correlation

* modify code for loading correlation data

* modify tests for correlation computation

* test loading correlation data using api endpoint

* add tests for asserting error in creating Correlation object

* add do correlation method

* add dummy tests for do_correlation method

* delete unused modules

* add tests for creating trait and dataset

* add intergration test for correlation api

* add tests for correlation api

* edit docorrelation method

* modify integration tests for correlation api

* modify tests for show_corr_results

* add create dataset function

* pep8 formatting and fix return value for api

* add more test data for doing correlation

* modify tests for correlation

* pep8 formatting

* add getting formatted corr type method

* import json library

add process samples method for correlation

* fix issue with sample_vals key_error

* create utility module for correlation

* refactor endpoint for /corr_compute

* add test and mocks for compute_correlation function

* add compute correlation function  and pep8 formatting

* move get genofile samplelist to utility module

* refactor code for CorrelationResults object

* pep8 formatting for module

* remove CorrelationResults from Api

* add base package

initialize data_set module with create_dataset,redis and Dataset_Getter

* set dataset_structure if redis is empty

* add callable for DatsetType

* add set_dataset_key method If name is not in the object's dataset dictionary

* add Dataset object and MrnaAssayDataSet

* add db_tools

* add mysql client

* add DatasetGroup object

* add species module

* get mapping method

* import helper functions and new dataset

* add connection to db before request

* add helper functions

* add logger module

* add get_group_samplelists module

* add logger for debug

* add code for adding sample_data

* pep8 formatting

* Add chunks module

* add correlation helper module

* add  get_sample_r_and_p_values method

add get_header_fields function

* add generate corr json method

* add function to retrieve_trait_info

* remove comments and clean up code in show_corr_results

* remove comments and clean up code for data_set module

* pep8 formatting for helper_functions module

* pep8 formatting for trait module

* add module for species

* add Temp Dataset Object

* add Phenotype Dataset

* add Genotype Dataset

* add rettrieve sample_sample_data method

* add webqtlUtil module

* add do lit correlation for all traits

* add webqtlCaseData:Settings not ported

* return the_trait for create trait method

* add correlation_test json data

* add tests fore show corr results

* add dictfier package

* add tests for show_corr_results

* add assertion for trait_id

* refactor code for show_corr_results

* add test file for compute_corr intergration tests

* add scipy dependency

* refactor show_corr_results object

add do lit correlation for trait_list

* add hmac module

* add bunch module:Dictionary using object notation

* add correlation functions

* add rpy2 dependency

* add hmac module

* add MrnaAssayTissueData object and get_symbol_values_pairs function

* add config module

* add get json_results method

* pep8 formatting remove comments

* add config file

* add db package

* refactor correlatio compuatation module

* add do tissue correlation for trait list

* add  do lit correlation for all traits

* add do tissue correlation for all traits

* add do_bicor for bicor method

* raise error for when initital start vars is None

* add support for both form and json data when for correlation input

* remove print statement and pep8 formatting

* add default settings file

* add tools module for locate_ignore_error

* refactor code remove comments for trait module

* Add new test data for  computing correlation

* pep8 formatting and use pickle

* refactor function for filtering form/json data

* remove unused imports

* remove mock functions in correlation_utility module

* refactor tests for compute correlation and pep8 formatting

* add tests for show_correlation results

* modify tests for show_corr_results

* add json files for tests

* pep8 formatting for show_corr_results

* Todo:Lint base files

* pylint for intergration tests

* add test module for test_corr_helpers

* Add test chunk module

* lint utility package

* refactoring and pep8 formatting

* implement simple metric for correlation

* add  hmac utility file

* add correlation prefix

* fix merge conflict

* minor fixes for endpoints

* import:python-scipy,python-sqlalchemy from guix

* add python mysqlclient

* remove pkg-resources from requirements

* add python-rpy3 from guix

* refactor code for species module

* pep8 formatting and refactor code

* add tests for genereating correlation results

* lint correlation functions

* fix failing tests for show_corr_results

* add new correlation test data fix errors

* fix issues related to getting group samplelists

* refactor intergration tests for correlation

* add todo  for refactoring_wanted_inputs

* replace custom Attribute setter with SimpleNamespace

* comparison of sample r correlation results btwn genenenetwork2 and genenetwork3

* delete AttributeSetter

* test request for /api/correlation/compute_correlation took 18.55710196495056 Seconds

* refactor tests and show_correlation results

* remove  unneccessary comments and print statements

* edit requirement txt file

* api/correlation took 114.29814600944519 Seconds for correlation resullts:20000

 - corr-type:lit

- corr-method:pearson

corr-dataset:corr_dataset:HC_M2_0606_P

* capture SQL_URI and GENENETWORK FILES path

* pep8 formatting edit && remove print statements

* delete filter_input function

update test and data for correlation

* add docstring for required correlation_input

* /api/correlation took 12.905632972717285 Seconds

 *  pearson

 * lit

 *dataset:HX_M2_0606_P

trait_id :1444666

p_range:(lower->-0.60,uppper->0.74)

corr_return_results: 100

* update integration and unittest for correlation

* add simple markdown docs for correlation

* update docs

* add tests and catch for invalid correlation_input

* minor fix for api

* Remove jupyter from deps

* guix.scm: Remove duplicate entry

* guix.scm: Add extra action items as comments

* Trim requirements.txt file

Co-authored-by: BonfaceKilz <me@bonfacemunyoki.com>

1 files changed, 163 insertions, 0 deletions

diff --git a/gn3/utility/logger.py b/gn3/utility/logger.py
new file mode 100644
index 0000000..4245a02
--- /dev/null
+++ b/gn3/utility/logger.py
@@ -0,0 +1,163 @@
+"""
+# GeneNetwork logger
+#
+# The standard python logging module is very good. This logger adds a
+# few facilities on top of that. Main one being that it picks up
+# settings for log levels (global and by module) and (potentially)
+# offers some fine grained log levels for the standard levels.
+#
+# All behaviour is defined here.  Global settings (defined in
+# default_settings.py).
+#
+# To use logging and settings put this at the top of a module:
+#
+#   import utility.logger
+#   logger = utility.logger.getLogger(__name__ )
+#
+# To override global behaviour set the LOG_LEVEL in default_settings.py
+# or use an environment variable, e.g.
+#
+#    env LOG_LEVEL=INFO ./bin/genenetwork2
+#
+# To override log level for a module replace that with, for example,
+#
+#   import logging
+#   import utility.logger
+#   logger = utility.logger.getLogger(__name__,level=logging.DEBUG)
+#
+# We'll add more overrides soon.
+"""
+# todo incomplete file
+
+# pylint: disable-all
+import logging
+import datetime
+from inspect import isfunction
+from inspect import stack
+
+from pprint import pformat as pf
+
+
+# from utility.tools import LOG_LEVEL, LOG_LEVEL_DEBUG, LOG_SQL
+
+LOG_SQL = True
+
+
+class GNLogger:
+    """A logger class with some additional functionality, such as
+    multiple parameter logging, SQL logging, timing, colors, and lazy
+    functions.
+
+    """
+
+    def __init__(self, name):
+        self.logger = logging.getLogger(name)
+
+    def setLevel(self, value):
+        """Set the undelying log level"""
+        self.logger.setLevel(value)
+
+    def debug(self, *args):
+        """Call logging.debug for multiple args. Use (lazy) debugf and
+level=num to filter on LOG_LEVEL_DEBUG.
+
+        """
+        self.collect(self.logger.debug, *args)
+
+    def debug20(self, *args):
+        """Call logging.debug for multiple args. Use level=num to filter on
+LOG_LEVEL_DEBUG (NYI).
+
+        """
+        if level <= LOG_LEVEL_DEBUG:
+            if self.logger.getEffectiveLevel() < 20:
+                self.collect(self.logger.debug, *args)
+
+    def info(self, *args):
+        """Call logging.info for multiple args"""
+        self.collect(self.logger.info, *args)
+
+    def warning(self, *args):
+        """Call logging.warning for multiple args"""
+        self.collect(self.logger.warning, *args)
+        # self.logger.warning(self.collect(*args))
+
+    def error(self, *args):
+        """Call logging.error for multiple args"""
+        now = datetime.datetime.utcnow()
+        time_str = now.strftime('%H:%M:%S UTC %Y%m%d')
+        l = [time_str]+list(args)
+        self.collect(self.logger.error, *l)
+
+    def infof(self, *args):
+        """Call logging.info for multiple args lazily"""
+        # only evaluate function when logging
+        if self.logger.getEffectiveLevel() < 30:
+            self.collectf(self.logger.debug, *args)
+
+    def debugf(self, level=0, *args):
+        """Call logging.debug for multiple args lazily and handle
+        LOG_LEVEL_DEBUG correctly
+
+        """
+        # only evaluate function when logging
+        if level <= LOG_LEVEL_DEBUG:
+            if self.logger.getEffectiveLevel() < 20:
+                self.collectf(self.logger.debug, *args)
+
+    def sql(self, sqlcommand, fun=None):
+        """Log SQL command, optionally invoking a timed fun"""
+        if LOG_SQL:
+            caller = stack()[1][3]
+            if caller in ['fetchone', 'fetch1', 'fetchall']:
+                caller = stack()[2][3]
+            self.info(caller, sqlcommand)
+        if fun:
+            result = fun(sqlcommand)
+            if LOG_SQL:
+                self.info(result)
+            return result
+
+    def collect(self, fun, *args):
+        """Collect arguments and use fun to output"""
+        out = "."+stack()[2][3]
+        for a in args:
+            if len(out) > 1:
+                out += ": "
+            if isinstance(a, str):
+                out = out + a
+            else:
+                out = out + pf(a, width=160)
+        fun(out)
+
+    def collectf(self, fun, *args):
+        """Collect arguments and use fun to output one by one"""
+        out = "."+stack()[2][3]
+        for a in args:
+            if len(out) > 1:
+                out += ": "
+                if isfunction(a):
+                    out += a()
+                else:
+                    if isinstance(a, str):
+                        out = out + a
+                    else:
+                        out = out + pf(a, width=160)
+        fun(out)
+
+# Get the module logger. You can override log levels at the
+# module level
+
+
+def getLogger(name, level=None):
+    """method to get logger"""
+    gnlogger = GNLogger(name)
+    _logger = gnlogger.logger
+
+    # if level:
+    #     logger.setLevel(level)
+    # else:
+    #     logger.setLevel(LOG_LEVEL)
+
+    # logger.info("Log level of "+name+" set to "+logging.getLevelName(logger.getEffectiveLevel()))
+    return gnlogger