diff options
| -rw-r--r-- | doc/README.org | 48 | ||||
| -rw-r--r-- | wqflask/base/data_set.py | 4 | ||||
| -rw-r--r-- | wqflask/wqflask/ctl/ctl_analysis.py | 53 | ||||
| -rw-r--r-- | wqflask/wqflask/static/new/javascript/ctl_graph.js | 199 | ||||
| -rw-r--r-- | wqflask/wqflask/templates/ctl_results.html | 36 | ||||
| -rw-r--r-- | wqflask/wqflask/templates/ctl_setup.html | 8 |
6 files changed, 319 insertions, 29 deletions
diff --git a/doc/README.org b/doc/README.org index 0f56914a..b38ea664 100644 --- a/doc/README.org +++ b/doc/README.org @@ -31,7 +31,7 @@ of GN2. Large system deployments can get very [[http://biogems.info/contrib/genenetwork/gn2.svg ][complex]]. In this document we explain the GeneNetwork version 2 (GN2) reproducible deployment system -which is based on GNU Guix (see also Pjotr's [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]). The Guix +which is based on GNU Guix (see also [[https://github.com/pjotrp/guix-notes/blob/master/README.md][Guix-notes]]). The Guix system can be used to install GN with all its files and dependencies. The official installation path is from a checked out version of the @@ -52,15 +52,16 @@ Linux distribution (including CentOS). For more elaborate installation instructions see [[#source-deployment][Source deployment]]. Note that GN2 consists of an approx. 5 GB installation including -database. +database. If you use a virtual machine we recommend to use at least +double. ** Step 1: Install GNU Guix Fetch the GNU Guix binary from [[https://www.gnu.org/software/guix/download/][here]] (middle panel) and follow -[[https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html][instructions]]. Essentially you have to download and unpack the tar ball -(which creates directories in /gnu and /var/guix), add build users and -group (Guix builds software as unpriviliged users) and run the Guix -daemon after fixing the paths (also known as the 'profile'). +[[https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html][instructions]]. Essentially, download and unpack the tar ball (which +creates directories in /gnu and /var/guix), add build users and group +(Guix builds software as unpriviliged users) and run the Guix daemon +after fixing the paths (also known as the 'profile'). Once you have succeeded, you have to [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#set-the-key][set the key]] (getting permission to download binaries from the GNU server) and you should be able to @@ -104,17 +105,23 @@ guix package -i git export GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt #+end_src -check out the git repsitories (gn-latest branch) +check out the git repositories (gn-deploy branch) #+begin_src bash cd ~ mkdir genenetwork cd genenetwork -git clone --branch gn-latest https://github.com/genenetwork/guix-bioinformatics -git clone --branch gn-latest --recursive https://github.com/genenetwork/guix guix-gn-latest -cd guix-gn-latest +git clone --branch gn-deploy https://github.com/genenetwork/guix-bioinformatics +git clone --branch gn-deploy --recursive https://github.com/genenetwork/guix guix-gn-deploy +cd guix-gn-deploy #+end_src bash +To test whether this is working try: + +#+begin_src bash +#+end_src bash + + ** Step 3: Authorize the GN Guix server GN2 has its own GNU Guix binary distribution server. To trust it you have @@ -146,7 +153,7 @@ GNU Guix package path by pointing the package path to our repository: #+begin_src bash rm /root/.config/guix/latest -ln -s ~/genenetwork/guix-gn-latest/ /root/.config/guix/latest +ln -s ~/genenetwork/guix-gn-deploy/ /root/.config/guix/latest #+end_src Now check whether you can find the GN2 package with @@ -164,13 +171,16 @@ And install with #+begin_src bash env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ \ guix package -i genenetwork2 \ - --substitute-urls="http://guix.genenetwork.org:8080 https://mirror.guixsd.org" \ - --fallback + --substitute-urls="http://guix.genenetwork.org" #+end_src Note: the order of the substitute url's may make a difference in speed (put the one first that is fastest for your location and time of day). +Note: if your system starts building or gives an error it may well be +Step 3 did not succeed. The installation should actually be smooth at +this point and only do binary installs (no compiling). + After installation you should be able to run genenetwork2 after updating the Guix suggested environment vars. Check the output of @@ -367,7 +377,7 @@ Create, install and run a recent version of the guix-daemon by compiling the guix repository you have installed with git in step 2. Follow [[https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#building-gnu-guix-from-source-using-guix][these]] steps carefully after -: cd ~/genenetwork/guix-gn-latest +: cd ~/genenetwork/guix-gn-deploy Make sure to restart the guix daemon and run guix client from this directory. @@ -377,7 +387,7 @@ directory. Reinstall genenetwork2 using the new tree #+begin_src bash -env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ ./pre-inst-env guix package -i genenetwork2 --substitute-urls="http://guix.genenetwork.org:8080 https://mirror.guixsd.org" +env GUIX_PACKAGE_PATH=~/genenetwork/guix-bioinformatics/ ./pre-inst-env guix package -i genenetwork2 --substitute-urls="http://guix.genenetwork.org https://mirror.guixsd.org" #+end_src bash Note the use of ./pre-inst-env here! @@ -486,7 +496,7 @@ and a download of the test database. <pjotrp> right? <user01> yep <user01> set to the ones in ~/.guix-profile/ -<pjotrp> good, and you are in gn-latest-guix repo [07:06] +<pjotrp> good, and you are in gn-deploy-guix repo [07:06] <user01> yep [07:07] <pjotrp> git log shows @@ -645,7 +655,7 @@ The following derivations would be built: <pjotrp> and see what this lists [08:31] <pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics ./pre-inst-env guix package -i genenetwork2 - --substitute-urls=http://guix.genenetwork.org:8080 --dry-run + --substitute-urls=http://guix.genenetwork.org --dry-run <pjotrp> should be all binary installs <user01> it's not.. [08:32] <user01> if I remove --substitute-urls, the list changes, does that mean I @@ -716,7 +726,7 @@ The following derivations would be built: <pjotrp> should not [09:24] <pjotrp> what does env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env guix package -i genenetwork2 - --substitute-urls="http://guix.genenetwork.org:8080" --dry-run + --substitute-urls="http://guix.genenetwork.org" --dry-run [09:25] <pjotrp> say for r-prepocesscore <pjotrp> download or build? @@ -869,7 +879,7 @@ The following derivations would be built: <pjotrp> I wrote an elixir package for guix :) <pjotrp> env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env guix package -A elixir - --substitute-urls="http://guix.genenetwork.org:8080" [10:08] + --substitute-urls="http://guix.genenetwork.org" [10:08] <pjotrp> elixir 1.2.3 out ../guix-bioinformatics/gn/packages/elixir.scm:31:2 <pjotrp> diff --git a/wqflask/base/data_set.py b/wqflask/base/data_set.py index 94b38e13..41c5d8ba 100644 --- a/wqflask/base/data_set.py +++ b/wqflask/base/data_set.py @@ -749,8 +749,8 @@ class PhenotypeDataSet(DataSet): if this_trait.lrs: query = """ select Geno.Chr, Geno.Mb from Geno, Species - where Species.Name = %s and - Geno.Name = %s and + where Species.Name = '%s' and + Geno.Name = '%s' and Geno.SpeciesId = Species.Id """ % (species, this_trait.locus) logger.sql(query) diff --git a/wqflask/wqflask/ctl/ctl_analysis.py b/wqflask/wqflask/ctl/ctl_analysis.py index 1b1d7155..9515d23a 100644 --- a/wqflask/wqflask/ctl/ctl_analysis.py +++ b/wqflask/wqflask/ctl/ctl_analysis.py @@ -6,6 +6,8 @@ import scipy as sp # SciPy import rpy2.robjects as ro # R Objects import rpy2.rinterface as ri +import simplejson as json + from base.webqtlConfig import GENERATED_IMAGE_DIR from utility import webqtlUtil # Random number for the image from utility import genofile_parser # genofile_parser @@ -73,8 +75,30 @@ class CTL(object): self.r_CTLnetwork = ro.r["CTLnetwork"] # Map the CTLnetwork function self.r_CTLprofiles = ro.r["CTLprofiles"] # Map the CTLprofiles function self.r_plotCTLobject = ro.r["plot.CTLobject"] # Map the CTLsignificant function + self.nodes_list = [] + self.edges_list = [] print("Obtained pointers to CTL functions") + def addNode(self, gt): + node_dict = { 'data' : {'id' : str(gt.name) + ":" + str(gt.dataset.name), + 'sid' : str(gt.name), + 'dataset' : str(gt.dataset.name), + 'label' : gt.name, + 'symbol' : gt.symbol, + 'geneid' : gt.geneid, + 'omim' : gt.omim } } + self.nodes_list.append(node_dict) + + def addEdge(self, gtS, gtT, significant, x): + edge_data = {'id' : str(gtS.symbol) + '_' + significant[1][x] + '_' + str(gtT.symbol), + 'source' : str(gtS.name) + ":" + str(gtS.dataset.name), + 'target' : str(gtT.name) + ":" + str(gtT.dataset.name), + 'lod' : significant[3][x], + 'color' : "#ff0000", + 'width' : significant[3][x] } + edge_dict = { 'data' : edge_data } + self.edges_list.append(edge_dict) + def run_analysis(self, requestform): print("Starting CTL analysis on dataset") self.trait_db_list = [trait.strip() for trait in requestform['trait_list'].split(',')] @@ -99,7 +123,7 @@ class CTL(object): genofilelocation = locate(dataset.group.name + ".geno", "genotype") parser = genofile_parser.ConvertGenoFile(genofilelocation) parser.process_csv() - + print(dataset.group) # Create a genotype matrix individuals = parser.individuals markers = [] @@ -129,9 +153,11 @@ class CTL(object): rPheno = r_t(ro.r.matrix(r_as_numeric(r_unlist(traits)), nrow=len(self.trait_db_list), ncol=len(individuals), dimnames = r_list(self.trait_db_list, individuals), byrow=True)) + print(rPheno) + # Use a data frame to store the objects - rPheno = r_data_frame(rPheno) - rGeno = r_data_frame(rGeno) + rPheno = r_data_frame(rPheno, check_names = False) + rGeno = r_data_frame(rGeno, check_names = False) # Debug: Print the genotype and phenotype files to disk #r_write_table(rGeno, "~/outputGN/geno.csv") @@ -156,7 +182,7 @@ class CTL(object): self.r_lineplot(res, significance = significance) r_dev_off() - n = 2 + n = 2 # We start from 2, since R starts from 1 :) for trait in self.trait_db_list: # Create the QTL like CTL plots self.results['imgurl' + str(n)] = webqtlUtil.genRandStr("CTL_") + ".png" @@ -169,6 +195,24 @@ class CTL(object): # Flush any output from R sys.stdout.flush() + # Create the interactive graph for cytoscape visualization (Nodes and Edges) + print(type(significant)) + if not type(significant) == ri.RNULLType: + for x in range(len(significant[0])): + print(significant[0][x], significant[1][x], significant[2][x]) # Debug to console + tsS = significant[0][x].split(':') # Source + tsT = significant[2][x].split(':') # Target + gtS = TRAIT.GeneralTrait(name = tsS[0], dataset_name = tsS[1]) # Retrieve Source info from the DB + gtT = TRAIT.GeneralTrait(name = tsT[0], dataset_name = tsT[1]) # Retrieve Target info from the DB + self.addNode(gtS) + self.addNode(gtT) + self.addEdge(gtS, gtT, significant, x) + + significant[0][x] = gtS.symbol + " (" + gtS.name + ")" # Update the trait name for the displayed table + significant[2][x] = gtT.symbol + " (" + gtT.name + ")" # Update the trait name for the displayed table + + self.elements = json.dumps(self.nodes_list + self.edges_list) + def loadImage(self, path, name): print("pre-loading imgage results:", self.results[path]) imgfile = open(self.results[path], 'rb') @@ -188,6 +232,7 @@ class CTL(object): print("Processing CTL output") template_vars = {} template_vars["results"] = self.results + template_vars["elements"] = self.elements self.render_image(self.results) sys.stdout.flush() return(dict(template_vars)) diff --git a/wqflask/wqflask/static/new/javascript/ctl_graph.js b/wqflask/wqflask/static/new/javascript/ctl_graph.js new file mode 100644 index 00000000..94bd7e9d --- /dev/null +++ b/wqflask/wqflask/static/new/javascript/ctl_graph.js @@ -0,0 +1,199 @@ +window.onload=function() { + // id of Cytoscape Web container div + //var div_id = "cytoscapeweb"; + + var cy = cytoscape({ + container: $('#cytoscapeweb'), // container to render in + + elements: elements_list, + + style: [ // the stylesheet for the graph + { + selector: 'node', + style: { + 'background-color': '#666', + 'label': 'data(symbol)', + 'font-size': 10 + } + }, + + { + selector: 'edge', + style: { + 'width': 'data(width)', + 'line-color': 'data(color)', + 'target-arrow-color': '#ccc', + 'target-arrow-shape': 'none', + 'font-size': 8, + 'curve-style': 'bezier' + } + } + ], + + zoom: 12, + layout: { name: 'circle', + fit: true, // whether to fit the viewport to the graph + padding: 30 // the padding on fit + //idealEdgeLength: function( edge ){ return edge.data['correlation']*10; }, + }, + + + zoomingEnabled: true, + userZoomingEnabled: true, + panningEnabled: true, + userPanningEnabled: true, + boxSelectionEnabled: false, + selectionType: 'single', + + // rendering options: + styleEnabled: true + }); + + var eles = cy.$() // var containing all elements, so elements can be restored after being removed + + var defaults = { + zoomFactor: 0.05, // zoom factor per zoom tick + zoomDelay: 45, // how many ms between zoom ticks + minZoom: 0.1, // min zoom level + maxZoom: 10, // max zoom level + fitPadding: 30, // padding when fitting + panSpeed: 10, // how many ms in between pan ticks + panDistance: 10, // max pan distance per tick + panDragAreaSize: 75, // the length of the pan drag box in which the vector for panning is calculated (bigger = finer control of pan speed and direction) + panMinPercentSpeed: 0.25, // the slowest speed we can pan by (as a percent of panSpeed) + panInactiveArea: 8, // radius of inactive area in pan drag box + panIndicatorMinOpacity: 0.5, // min opacity of pan indicator (the draggable nib); scales from this to 1.0 + zoomOnly: false, // a minimal version of the ui only with zooming (useful on systems with bad mousewheel resolution) + fitSelector: undefined, // selector of elements to fit + animateOnFit: function(){ // whether to animate on fit + return false; + }, + fitAnimationDuration: 1000, // duration of animation on fit + + // icon class names + sliderHandleIcon: 'fa fa-minus', + zoomInIcon: 'fa fa-plus', + zoomOutIcon: 'fa fa-minus', + resetIcon: 'fa fa-expand' + }; + + cy.panzoom( defaults ); + + function create_qtips(cy){ + cy.nodes().qtip({ + content: function(){ + gn_link = '<b>'+'<a href="http://gn2.genenetwork.org/show_trait?trait_id=' + this.data().sid + '&dataset=' + this.data().dataset + '" >'+this.data().id +'</a>'+'</b><br>' + ncbi_link = '<a href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=gene&cmd=Retrieve&dopt=Graphics&list_uids=' + this.data().geneid + '" >NCBI<a>'+'<br>' + omim_link = '<a href="http://www.ncbi.nlm.nih.gov/omim/' + this.data().omim + '" >OMIM<a>'+'<br>' + qtip_content = gn_link + ncbi_link + omim_link + return qtip_content + //return '<b>'+'<a href="http://gn2.genenetwork.org/show_trait?trait_id=' + this.data().id + '&dataset=' + this.data().dataset + '" >'+this.data().id +'<a>'+'</b>' + }, + // content: { + // title: '<b>'+'<a href="http://gn2.genenetwork.org/show_trait?trait_id=' + this.target() + '&dataset=' + this.dataset() + '" >'+this.target() +'<a>'+'</b>', + // text: this.target, + // button: true + // }, + position: { + my: 'top center', + at: 'bottom center' + }, + style: { + classes: 'qtip-bootstrap', + tip: { + width: 16, + height: 8 + } + } + }); + + cy.edges().qtip({ + content: function(){ + edge_ID = '<b>Edge: ' + this.data().id + '</b><br>' + lod_score = 'LOD: ' + this.data().lod + '<br>' + return edge_ID + lod_score + }, + position: { + my: 'top center', + at: 'bottom center' + }, + style: { + classes: 'qtip-bootstrap', + tip: { + width: 16, + height: 8 + } + } + }); + } + + create_qtips(cy) + + $('#slide').change(function() { + eles.restore() + + console.log(eles) + + // nodes_to_restore = eles.filter("node[max_corr >= " + $(this).val() + "], edge[correlation >= " + $(this).val() + "][correlation <= -" + $(this).val() + "]") + // nodes_to_restore.restore() + + // edges_to_restore = eles.filter("edge[correlation >= " + $(this).val() + "][correlation <= -" + $(this).val() + "]") + // edges_to_restore.restore() + + //cy.$("node[max_corr >= " + $(this).val() + "]").restore(); + //cy.$("edge[correlation >= " + $(this).val() + "][correlation <= -" + $(this).val() + "]").restore(); + + cy.$("node[max_corr < " + $(this).val() + "]").remove(); + cy.$("edge[correlation < " + $(this).val() + "][correlation > -" + $(this).val() + "]").remove(); + + cy.layout({ name: $('select[name=layout_select]').val(), + fit: true, // whether to fit the viewport to the graph + padding: 25 // the padding on fit + }); + + }); + + $('#reset_graph').click(function() { + eles.restore() + $('#slide').val(0) + cy.layout({ name: $('select[name=layout_select]').val(), + fit: true, // whether to fit the viewport to the graph + padding: 25 // the padding on fit + }); + }); + + $('select[name=focus_select]').change(function() { + focus_trait = $(this).val() + + eles.restore() + cy.$('edge[source != "' + focus_trait + '"][target != "' + focus_trait + '"]').remove() + + cy.layout({ name: $('select[name=layout_select]').val(), + fit: true, // whether to fit the viewport to the graph + padding: 25 // the padding on fit + }); + }); + + $('select[name=layout_select]').change(function() { + layout_type = $(this).val() + console.log("LAYOUT:", layout_type) + cy.layout({ name: layout_type, + fit: true, // whether to fit the viewport to the graph + padding: 25 // the padding on fit + }); + }); + + $("a#image_link").click(function(e) { + var pngData = cy.png(); + + $(this).attr('href', pngData); + $(this).attr('download', 'network_graph.png'); + + console.log("TESTING:", image_link) + + }); + + +}; + + diff --git a/wqflask/wqflask/templates/ctl_results.html b/wqflask/wqflask/templates/ctl_results.html index 00ccecb6..969ca18a 100644 --- a/wqflask/wqflask/templates/ctl_results.html +++ b/wqflask/wqflask/templates/ctl_results.html @@ -1,17 +1,30 @@ {% extends "base.html" %} +{% block css %} + <link rel="stylesheet" type="text/css" href="/static/new/css/network_graph.css" /> + <link rel="stylesheet" type="text/css" href="/static/packages/cytoscape/css/cytoscape.js-panzoom.css" /> + <link rel="stylesheet" type="text/css" href="http://cdnjs.cloudflare.com/ajax/libs/qtip2/2.2.0/jquery.qtip.css"> + <style> + /* The Cytoscape Web container must have its dimensions set. */ + html, body { height: 100%; width: 100%; padding: 0; margin: 0; } + #cytoscapeweb { width: 70%; height: 70%; } + </style> +{% endblock %} + {% block title %}CTL results{% endblock %} {% block content %} <!-- Start of body --> <div class="container"> <h1>CTL Results</h1> {{(request.form['trait_list'].split(',')|length)}} phenotypes as input<br> - <h3>Network Figure</h3> + + <!-- <a href="/tmp/{{ results['imgurl1'] }}"> <img alt="Embedded Image" src="data:image/png;base64, {% for elem in results['imgdata1'] -%} {% print("%c"|format(elem)) %} {%- endfor %} - " /></a> + " /></a> --> + <h3>CTL/QTL Plots for individual traits</h3> {% for r in range(2, (request.form['trait_list'].split(',')|length +1)) %} <a href="/tmp/{{ results['imgurl' + r|string] }}"> @@ -39,9 +52,26 @@ </tr> {% endfor %} </table> + <h3>Network Figure</h3> + <div id="cytoscapeweb" class="col-xs-9" style="min-height:700px !important;"></div> +</div> +{% endblock %} +{% block js %} + <script> + elements_list = {{ elements | safe }} + </script> + <script language="javascript" type="text/javascript" src="/static/new/packages/DataTables/js/jquery.js"></script> + <script language="javascript" type="text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/qtip2/2.2.0/jquery.qtip.js"></script> + <script language="javascript" type="text/javascript" src="/static/packages/underscore/underscore-min.js"></script> + <script language="javascript" type="text/javascript" src="/static/packages/cytoscape/js/min/cytoscape.min.js"></script> + <script language="javascript" type="text/javascript" src="/static/packages/cytoscape/js/min/AC_OETags.min.js"></script> + <script language="javascript" type="text/javascript" src="/static/packages/cytoscape/js/min/json2.min.js"></script> + <script language="javascript" type="text/javascript" src="/static/packages/cytoscape/js/src/cytoscape-panzoom.js"></script> + <script language="javascript" type="text/javascript" src="/static/packages/cytoscape/js/src/cytoscape-qtip.js"></script> -</div> + <script language="javascript" type="text/javascript" src="/static/new/javascript/ctl_graph.js"></script> {% endblock %} + diff --git a/wqflask/wqflask/templates/ctl_setup.html b/wqflask/wqflask/templates/ctl_setup.html index a05379a8..a7ad5759 100644 --- a/wqflask/wqflask/templates/ctl_setup.html +++ b/wqflask/wqflask/templates/ctl_setup.html @@ -11,7 +11,13 @@ Please make sure you select enough traits to perform CTL. Your collection needs to contain at least 2 different traits. You provided {{request.form['trait_list'].split(',')|length}} traits as input. </div> {% else %} - <h1>CTL analysis parameters</h1> + <h1>CTL analysis</h1> + CTL analysis is published as open source software, if you are using this method in your publications, please cite:<br><br> + Arends D, Li Y, Brockmann GA, Jansen RC, Williams RW, Prins P<br> + Correlation trait locus (CTL) mapping: Phenotype network inference subject to genotype.<br> + The Journal of Open Source Software (2016)<br> + Published in <a href="http://joss.theoj.org/papers/10.21105/joss.00087"><img src="http://joss.theoj.org/papers/10.21105/joss.00087/status.svg"></a> + <br><br> {{(request.form['trait_list'].split(',')|length)}} traits as input <form action="/ctl_results" method="post" class="form-horizontal"> |