# Migrate GN1 Clustering GeneNetwork1 has clustering output that we want to migrate to GN2. For example, go to => http://gn1-lily.genenetwork.org/ GN1 on Lily Select Type -> CRTD mRNA data and do a search for 'synap*'. Click on the top 10 results checkboxes and Add to the basket (add is an icon). Now click on 'Select all' and the 'Heat map' icons. Computation may take a while. Next click on 'Cluster traits' and 'Redraw' map. This is the one we want in GN3/GN2! => ./heatmap.png ## Members * fredm * pjotr ## Useful links => https://github.com/genenetwork/genenetwork1/tree/master/web/webqtl/heatmap Implementation ## Implementation As a first step the computations should move to GN3 with proper regression/unit testing scripts. Next wire them up as REST API endpoints and add it to the GN2 web interface: - [ ] Move computation to GN3 - [ ] Create REST API endpoints - [ ] Add regression/unit tests in GN3 - [ ] Add to GN2 web interface @pjotr I couldn't quite figure out what "CRTD mRNA", so I selected "Cartilage mRNA" , to try and produce a heat map, and I did get some results. The selected data was: * Species: Mouse (mm10) * Group: BXD Family * Type: Cartilage mRNA I got => http://gn1-lily.genenetwork.org/image/Heatmap_8CJbN7Iy.png the initial heatmap and => http://gn1-lily.genenetwork.org/image/Heatmap_syjdiq5z.png the 'Cluster Traits' heatmap is that anything like what I should expect? I (@fredmanglis) have noticed that the computation of the heatmap data is intertwined with the drawing of the heatmap. I think this will be among the first things to separate, as part of moving the computation over to GN3. My initial impression of this, is that the GN3 should handle the computation and GN2 the display. Please correct me if I am wrong. ## 2021-08-11 I have had a look at the Python implementation of Plotly, and tried out a few of the examples to get a feel for the library and its capabilities. I think it is possible to use the library to draw the heatmaps, though I'll need more time to fully grasp how it works, and how I could use it to actually provide some of the features in the clustering heatmaps that do not seem to be out-of-the-box with plotly, e.g. the lines with clustering distance. It does also seem like the library might provide more complex heatmaps, such as => https://plotly.com/python/imshow/#display-an-xarray-image-with-pximshow heatmap of xarray data which sort of approximates the heatmap example linked in => https://github.com/genenetwork/genenetwork3/pull/31#issuecomment-891539359 zsloan's comment I wrote a simple script to test out the library as shown: ``` import random import plotly.express as px def generate_random_data(width=10, height=30): return [[random.uniform(0,2) for i in range(0, width)] for j in range(0, height)] def main(): fig = px.imshow(generate_random_data()) fig.show() if __name__ == "__main__": main() ``` Thankfully, the following python packages seem to already be packaged with guix-bioinformatics: * python-plotly * python-pandas * python-xarray * python-scipy The only (seemingly) missing package is `python-pooch` ## python-pooch python-pooch is in guix as of commit 211c933 in master from this March: => https://guix.gnu.org/packages/python-pooch-1.3.0/ python-pooch package listing => https://issues.guix.gnu.org/47022 python-pooch patch thread ## 2021-08-12 When the "Single Spectrum" colour scheme is selected, the heatmap's "colour-scale" in genenetwork1 is a single spectrum that flows from Blue, through green, to red. This one is easy to reproduce somewhat by setting the colour scale with something like: ``` fig.update_coloraxes(colorscale=[ [0.0, '#0000FF'], [0.5, '#00FF00'], [1.0, '#FF0000']]) ``` When the "Blue + Red" colour scheme is selected, the heatmap's "colour-scale" in genenetwork1 is split into 2 separate colour scales depending on whether the data value corresponds to one of: * C57BL/6J + * DBA/2J + When the "Grey + Blue + Red" colour scheme is selected, the heatmap's "colour-scale" in genenetwork1 goes from a dark-grey at 0 to a light-grey at 0.5, before splitting into 2 separate colour scales for values greater than 0.5, depending on whether the data value corresponds to one of: * C57BL/6J + * DBA/2J + I (@fredmanglis) have not yet figured out how to represent these more complex splits on the Plotly heatmaps, but I have a suspicion this might be achieved if there is a way to label the data as it goes into the `px.imshow(...)` call. Maybe look into using the => https://plotly.com/python/creating-and-updating-figures/#conditionally-updating-traces conditional trace update feature to set up the colours as appropriate, when different colour-schemes are selected. Failing that, have a look at the => https://plotly.com/python/colorscales/ colour scales documentation => https://plotly.com/python/plotly-fundamentals/ plotly fundamentals page => https://plotly.com/python/categorical-axes/ categorical axes ## 2021-08-17 Tried providing the data-points as a dictionary instead of number, to test out the categorical layout of the plots: something along the lines of: ``` data = [[{'value': 0.07039128483638035, 'category': 'C57BL/6J +'}, {'value': 1.8493427990767048, 'category': 'C57BL/6J +'}, ..., {'value': 1.3089193078506003, 'category': 'C57BL/6J +'}]] ``` but that did work as expected. Paused on heatmap generation to first test out the database access code. Added tests and fixed issues with older db-access code to get a sample of the data for drawing heatmaps.