# Migrate GN1 Clustering

GeneNetwork1 has clustering output that we want to migrate to GN2. For example, go to

=> http://gn1-lily.genenetwork.org/ GN1 on Lily

Select Type -> CRTD mRNA data and do a search for 'synap*'. Click on the top 10 results checkboxes and Add to the basket (add is an icon). Now click on 'Select all' and the 'Heat map' icons. Computation may take a while.

Next click on 'Cluster traits' and 'Redraw' map. This is the one we want in GN3/GN2!

=> ./heatmap.png

## Members

* fredm
* pjotr

## Useful links

=> https://github.com/genenetwork/genenetwork1/tree/master/web/webqtl/heatmap Implementation

## Implementation

As a first step the computations should move to GN3 with proper regression/unit testing scripts. Next wire them up as REST API endpoints and add it to the GN2 web interface:

- [ ] Move computation to GN3
- [ ] Create REST API endpoints
- [ ] Add regression/unit tests in GN3
- [ ] Add to GN2 web interface


@pjotr I couldn't quite figure out what "CRTD mRNA", so I selected "Cartilage mRNA" , to try and produce a heat map, and I did get some results.

The selected data was:

* Species: Mouse (mm10)
* Group: BXD Family
* Type: Cartilage mRNA

I got
=> http://gn1-lily.genenetwork.org/image/Heatmap_8CJbN7Iy.png the initial heatmap
and
=> http://gn1-lily.genenetwork.org/image/Heatmap_syjdiq5z.png the 'Cluster Traits' heatmap

is that anything like what I should expect?



I (@fredmanglis) have noticed that the computation of the heatmap data is intertwined with the drawing of the heatmap.

I think this will be among the first things to separate, as part of moving the computation over to GN3. My initial impression of this, is that the GN3 should handle the computation and GN2 the display.

Please correct me if I am wrong.


## 2021-08-11

I have had a look at the Python implementation of Plotly, and tried out a few of the examples to get a feel for the library and its capabilities. I think it is possible to use the library to draw the heatmaps, though I'll need more time to fully grasp how it works, and how I could use it to actually provide some of the features in the clustering heatmaps that do not seem to be out-of-the-box with plotly, e.g. the lines with clustering distance.

It does also seem like the library might provide more complex heatmaps, such as

=> https://plotly.com/python/imshow/#display-an-xarray-image-with-pximshow heatmap of xarray data

which sort of approximates the heatmap example linked in

=> https://github.com/genenetwork/genenetwork3/pull/31#issuecomment-891539359 zsloan's comment

I wrote a simple script to test out the library as shown:

```
import random
import plotly.express as px

def generate_random_data(width=10, height=30):
    return [[random.uniform(0,2) for i in range(0, width)]
            for j in range(0, height)]

def main():
    fig = px.imshow(generate_random_data())
    fig.show()

if __name__ == "__main__":
    main()
```

Thankfully, the following python packages seem to already be packaged with guix-bioinformatics:

* python-plotly
* python-pandas
* python-xarray
* python-scipy

The only (seemingly) missing package is `python-pooch`

## python-pooch

python-pooch is in guix as of commit 211c933 in master from this March:

=> https://guix.gnu.org/packages/python-pooch-1.3.0/ python-pooch package listing
=> https://issues.guix.gnu.org/47022 python-pooch patch thread