diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/API_readme.md | 42 | ||||
| -rw-r--r-- | doc/GUIX-Reproducible-from-source.org | 1 | ||||
| -rw-r--r-- | doc/README.org | 151 |
3 files changed, 85 insertions, 109 deletions
diff --git a/doc/API_readme.md b/doc/API_readme.md index 652376a0..be6668dc 100644 --- a/doc/API_readme.md +++ b/doc/API_readme.md @@ -6,17 +6,17 @@ To get a list of species with data available in GN (and their associated names and ids): ``` -curl http://gn2.genenetwork.org/api/v_pre1/species +curl http://genenetwork.org/api/v_pre1/species [ { "FullName": "Mus musculus", "Id": 1, "Name": "mouse", "TaxonomyId": 10090 }, ... { "FullName": "Populus trichocarpa", "Id": 10, "Name": "poplar", "TaxonomyId": 3689 } ] ``` Or to get a single species info: ``` -curl http://gn2.genenetwork.org/api/v_pre1/species/mouse +curl http://genenetwork.org/api/v_pre1/species/mouse ``` OR ``` -curl http://gn2.genenetwork.org/api/v_pre1/species/mouse.json +curl http://genenetwork.org/api/v_pre1/species/mouse.json ``` *For all queries where the last field is a user-specified name/ID, there will be the option to append a file format type. Currently there is only JSON (and it will default to JSON if none is provided), but other formats will be added later* @@ -26,33 +26,33 @@ curl http://gn2.genenetwork.org/api/v_pre1/species/mouse.json This query can optionally filter by species: ``` -curl http://gn2.genenetwork.org/api/v_pre1/groups (for all species) +curl http://genenetwork.org/api/v_pre1/groups (for all species) ``` OR ``` -curl http://gn2.genenetwork.org/api/v_pre1/groups/mouse (for just mouse groups/RISets) +curl http://genenetwork.org/api/v_pre1/groups/mouse (for just mouse groups/RISets) [ { "DisplayName": "BXD", "FullName": "BXD RI Family", "GeneticType": "riset", "Id": 1, "MappingMethodId": "1", "Name": "BXD", "SpeciesId": 1, "public": 2 }, ... { "DisplayName": "AIL LGSM F34 and F39-43 (GBS)", "FullName": "AIL LGSM F34 and F39-43 (GBS)", "GeneticType": "intercross", "Id": 72, "MappingMethodId": "2", "Name": "AIL-LGSM-F34-F39-43-GBS", "SpeciesId": 1, "public": 2 } ] ``` ## Fetch Genotypes for Group/RISet ## ``` -curl http://gn2.genenetwork.org/api/v_pre1/genotypes/bimbam/BXD -curl http://gn2.genenetwork.org/api/v_pre1/genotypes/BXD.bimbam +curl http://genenetwork.org/api/v_pre1/genotypes/bimbam/BXD +curl http://genenetwork.org/api/v_pre1/genotypes/BXD.bimbam ``` Returns a group's genotypes in one of several formats - bimbam, rqtl2, or geno (a format used by qtlreaper which is just a CSV file consisting of marker positions and genotypes) Rqtl2 genotype queries can also include the dataset name and will return a zip of the genotypes, phenotypes, and gene map (marker names/positions). For example: ``` -curl http://gn2.genenetwork.org/api/v_pre1/genotypes/rqtl2/BXD/HC_M2_0606_P.zip +curl http://genenetwork.org/api/v_pre1/genotypes/rqtl2/BXD/HC_M2_0606_P.zip ``` ## Fetch Datasets ## ``` -curl http://gn2.genenetwork.org/api/v_pre1/datasets/bxd +curl http://genenetwork.org/api/v_pre1/datasets/bxd ``` OR ``` -curl http://gn2.genenetwork.org/api/v_pre1/datasets/mouse/bxd +curl http://genenetwork.org/api/v_pre1/datasets/mouse/bxd [ { "AvgID": 1, "CreateTime": "Fri, 01 Aug 2003 00:00:00 GMT", "DataScale": "log2", "FullName": "UTHSC/ETHZ/EPFL BXD Liver Polar Metabolites Extraction A, CD Cohorts (Mar 2017) log2", "Id": 1, "Long_Abbreviation": "BXDMicroArray_ProbeSet_August03", "ProbeFreezeId": 3, "ShortName": "Brain U74Av2 08/03 MAS5", "Short_Abbreviation": "Br_U_0803_M", "confidentiality": 0, "public": 0 }, ... { "AvgID": 3, "CreateTime": "Tue, 14 Aug 2018 00:00:00 GMT", "DataScale": "log2", "FullName": "EPFL/LISP BXD CD Liver Affy Mouse Gene 1.0 ST (Aug18) RMA", "Id": 859, "Long_Abbreviation": "EPFLMouseLiverCDRMAApr18", "ProbeFreezeId": 181, "ShortName": "EPFL/LISP BXD CD Liver Affy Mouse Gene 1.0 ST (Aug18) RMA", "Short_Abbreviation": "EPFLMouseLiverCDRMA0818", "confidentiality": 0, "public": 1 } ] ``` (I added the option to specify species just in case we end up with the same group name across multiple species at some point, though it's currently unnecessary) @@ -61,11 +61,11 @@ curl http://gn2.genenetwork.org/api/v_pre1/datasets/mouse/bxd ### For mRNA Assay/"ProbeSet" ### ``` -curl http://gn2.genenetwork.org/api/v_pre1/dataset/HC_M2_0606_P +curl http://genenetwork.org/api/v_pre1/dataset/HC_M2_0606_P ``` OR ``` -curl http://gn2.genenetwork.org/api/v_pre1/dataset/bxd/HC_M2_0606_P``` +curl http://genenetwork.org/api/v_pre1/dataset/bxd/HC_M2_0606_P``` { "confidential": 0, "data_scale": "log2", "dataset_type": "mRNA expression", "full_name": "Hippocampus Consortium M430v2 (Jun06) PDNN", "id": 112, "name": "HC_M2_0606_P", "public": 2, "short_name": "Hippocampus M430v2 BXD 06/06 PDNN", "tissue": "Hippocampus mRNA", "tissue_id": 9 } ``` (This also has the option to specify group/riset) @@ -73,26 +73,26 @@ curl http://gn2.genenetwork.org/api/v_pre1/dataset/bxd/HC_M2_0606_P``` ### For "Phenotypes" (basically non-mRNA Expression; stuff like weight, sex, etc) ### For these traits, the query fetches publication info and takes the group and phenotype 'ID' as input. For example: ``` -curl http://gn2.genenetwork.org/api/v_pre1/dataset/bxd/10001 +curl http://genenetwork.org/api/v_pre1/dataset/bxd/10001 { "dataset_type": "phenotype", "description": "Central nervous system, morphology: Cerebellum weight, whole, bilateral in adults of both sexes [mg]", "id": 10001, "name": "CBLWT2", "pubmed_id": 11438585, "title": "Genetic control of the mouse cerebellum: identification of quantitative trait loci modulating size and architecture", "year": "2001" } ``` ## Fetch Sample Data for Dataset ## ``` -curl http://gn2.genenetwork.org/api/v_pre1/sample_data/HSNIH-PalmerPublish.csv +curl http://genenetwork.org/api/v_pre1/sample_data/HSNIH-PalmerPublish.csv ``` Returns a CSV file with sample/strain names as the columns and trait IDs as rows ## Fetch Sample Data for Single Trait ## ``` -curl http://gn2.genenetwork.org/api/v_pre1/sample_data/HC_M2_0606_P/1436869_at +curl http://genenetwork.org/api/v_pre1/sample_data/HC_M2_0606_P/1436869_at [ { "data_id": 23415463, "sample_name": "129S1/SvImJ", "sample_name_2": "129S1/SvImJ", "se": 0.123, "value": 8.201 }, { "data_id": 23415463, "sample_name": "A/J", "sample_name_2": "A/J", "se": 0.046, "value": 8.413 }, { "data_id": 23415463, "sample_name": "AKR/J", "sample_name_2": "AKR/J", "se": 0.134, "value": 8.856 }, ... ] ``` ## Fetch Trait List for Dataset ## ``` -curl http://gn2.genenetwork.org/api/v_pre1/traits/HXBBXHPublish.json +curl http://genenetwork.org/api/v_pre1/traits/HXBBXHPublish.json [ { "Additive": 0.0499967532467532, "Id": 10001, "LRS": 16.2831307029479, "Locus": "rs106114574", "PhenotypeId": 1449, "PublicationId": 319, "Sequence": 1 }, ... ] ``` @@ -101,7 +101,7 @@ Both JSON and CSV formats can be specified, with JSON as default. There is also ## Fetch Trait Info (Name, Description, Location, etc) ## ### For mRNA Expression/"ProbeSet" ### ``` -curl http://gn2.genenetwork.org/api/v_pre1/trait/HC_M2_0606_P/1436869_at +curl http://genenetwork.org/api/v_pre1/trait/HC_M2_0606_P/1436869_at { "additive": -0.214087568058076, "alias": "HHG1; HLP3; HPE3; SMMCI; Dsh; Hhg1", "chr": "5", "description": "sonic hedgehog (hedgehog)", "id": 99602, "locus": "rs8253327", "lrs": 12.7711275309832, "mb": 28.457155, "mean": 9.27909090909091, "name": "1436869_at", "p_value": 0.306, "se": null, "symbol": "Shh" } ``` @@ -110,7 +110,7 @@ For phenotypes this just gets the max LRS, its location, and additive effect (a Since each group/riset only has one phenotype "dataset", this query takes either the group/riset name or the group/riset name + "Publish" (for example "BXDPublish", which is the dataset name in the DB) as input ``` -curl http://gn2.genenetwork.org/api/v_pre1/trait/BXD/10001 +curl http://genenetwork.org/api/v_pre1/trait/BXD/10001 { "additive": 2.39444435069444, "id": 4, "locus": "rs48756159", "lrs": 13.4974911471087 } ``` @@ -130,7 +130,7 @@ Each method's query takes the following parameters respectively (more will be ad Example query: ``` -curl http://gn2.genenetwork.org/api/v_pre1/mapping?trait_id=10015&db=BXDPublish&method=gemma&use_loco=true +curl http://genenetwork.org/api/v_pre1/mapping?trait_id=10015&db=BXDPublish&method=gemma&use_loco=true ``` ### R/qtl ### @@ -146,7 +146,7 @@ curl http://gn2.genenetwork.org/api/v_pre1/mapping?trait_id=10015&db=BXDPublish& Example query: ``` -curl http://gn2.genenetwork.org/api/v_pre1/mapping?trait_id=1418701_at&db=HC_M2_0606_P&method=rqtl&num_perm=100 +curl http://genenetwork.org/api/v_pre1/mapping?trait_id=1418701_at&db=HC_M2_0606_P&method=rqtl&num_perm=100 ``` Some combinations of methods/models may not make sense. The R/qtl manual should be referred to for any questions on its use (specifically the scanone function in this case) @@ -164,6 +164,6 @@ This query currently takes the following parameters (though more will be added): Example query: ``` -curl http://gn2.genenetwork.org/api/v_pre1/correlation?trait_id=1427571_at&db=HC_M2_0606_P&target_db=BXDPublish&type=sample&return_count=100 +curl http://genenetwork.org/api/v_pre1/correlation?trait_id=1427571_at&db=HC_M2_0606_P&target_db=BXDPublish&type=sample&return_count=100 [ { "#_strains": 6, "p_value": 0.004804664723032055, "sample_r": -0.942857142857143, "trait": 20511 }, { "#_strains": 6, "p_value": 0.004804664723032055, "sample_r": -0.942857142857143, "trait": 20724 }, { "#_strains": 12, "p_value": 1.8288943424888848e-05, "sample_r": -0.9233615170820528, "trait": 13536 }, { "#_strains": 7, "p_value": 0.006807187408935392, "sample_r": 0.8928571428571429, "trait": 10157 }, { "#_strains": 7, "p_value": 0.006807187408935392, "sample_r": -0.8928571428571429, "trait": 20392 }, ... ] ``` diff --git a/doc/GUIX-Reproducible-from-source.org b/doc/GUIX-Reproducible-from-source.org index 83adce99..19e4d14f 100644 --- a/doc/GUIX-Reproducible-from-source.org +++ b/doc/GUIX-Reproducible-from-source.org @@ -33,7 +33,6 @@ GNU Guix has to be installed as root. I tested this recipe on a fresh install of Debian 8.3.0 (in KVM) though it should work on any modern Linux distribution (including CentOS). - Note that GN2 consists of an approx. 5 GB installation including database. If you use a virtual machine we recommend to use at least double. diff --git a/doc/README.org b/doc/README.org index 620c946c..c2ef2d57 100644 --- a/doc/README.org +++ b/doc/README.org @@ -3,17 +3,12 @@ * Table of Contents :TOC: - [[#introduction][Introduction]] - [[#install][Install]] - - [[#tarball][Tarball]] - - [[#docker][Docker]] - - [[#with-source][With source]] - [[#running-gn2][Running GN2]] - - [[#run-mysql-server][Run MySQL server]] - - [[#install-mysql-with-gnu-guix][Install MySQL with GNU GUIx]] + - [[#run-mariadb-server][Run MariaDB server]] + - [[#install-mariadb-with-gnu-guix][Install MariaDB with GNU GUIx]] - [[#load-the-small-database-in-mysql][Load the small database in MySQL]] - [[#gn2-dependency-graph][GN2 Dependency Graph]] - [[#working-with-the-gn2-source-code][Working with the GN2 source code]] - - [[#running-elasticsearch][Running ElasticSearch]] - - [[#systemd][SystemD]] - [[#read-more][Read more]] - [[#trouble-shooting][Trouble shooting]] - [[#importerror-no-module-named-jinja2][ImportError: No module named jinja2]] @@ -22,6 +17,8 @@ - [[#rpy2-error-show-now-found][Rpy2 error 'show' now found]] - [[#mysql-cant-connect-server-through-socket-error][Mysql can't connect server through socket ERROR]] - [[#irc-session][IRC session]] + - [[#notes][NOTES]] + - [[#deploying-gn2-official][Deploying GN2 official]] * Introduction @@ -40,44 +37,10 @@ an example of the [[#gn2-dependency-graph][GN2 Dependency Graph]]. * Install -The quickest way to install GN2 is by using a binary installation -(tarball or Docker image). These installations are bundled by GNU -Guix and include all dependencies. You can install GeneNetwork on most -Linux distributions, including Debian, Ubuntu, Fedora and CentOS, -provided you have administrator privileges (root). The alternative is -a Docker installation. - -** Tarball - -Download the ~800Mb tarball from -[[http://files.genenetwork.org/software/binary_tarball/]]. Validate the checksum and -unpack to root, for example - -: tar xvzf genenetwork2-2.10rc3-1538ffd-tarball-pack.tar.gz -: mv /gnu / -: mv /opt/genenetwork2 /opt/ - -Now you shoud be able to start the server with - -: /opt/genenetwork2/bin/genenetwork2 - -When the server stops with a MySQL error [[#run-mysql-server][Run MySQL server]] -and set SQL_URI to point at it. For example: - -: export SQL_URI=mysql://gn2:mysql_password@127.0.0.1/db_webqtl_s - -See also [[#mysql-cant-connect-server-through-socket-error][Mysql can't connect server through socket ERROR]]. - -** Docker - -Docker images are also available through -[[http://files.genenetwork.org/software/]]. Validate the checksum and run -with [[https://docs.docker.com/engine/reference/commandline/load/][Docker load]]. - -** With source - -For more elaborate installation instructions on deploying GeneNetwork from -source see [[#source-deployment][Source deployment]]. +Make sure to install GNU Guix using the binary download instructions +on the main website. Follow the instructions on +[[GUIX-Reproducible-from-source.org]] to download pre-built binaries. Note +the download amounts to several GBs of data. * Running GN2 @@ -94,39 +57,54 @@ or you can set environment variables to override individual parameters, e.g. the debug and logging switches can be particularly useful when developing GN2. -* Run MySQL server -** Install MySQL with GNU GUIx +* Running Redis + +Install redis. Make sure you add the setting: + +: appendonly yes + +* Run MariaDB server +** Install MariaDB with GNU GUIx + +/Note: we moved to MariaDB/ These are the steps you can take to install a fresh installation of -mysql (which comes as part of the GNU Guix genenetwork2 install). +mariadb (which comes as part of the GNU Guix genenetwork2 install). As root configure and run #+BEGIN_SRC bash -adduser mysql && addgroup mysql -mysqld --datadir=/var/mysql --initialize-insecure -mkdir -p /var/run/mysqld -chown mysql.mysql ~/mysql /var/run/mysqld -mysqld -u mysql --datadir=/var/mysql --explicit_defaults_for_timestamp -P 12048" +adduser mariadb && addgroup mariadb +mysqld --datadir=/home/mariadb/database --initialize-insecure +mkdir -p /var/run/mariadbd +chown mariadb.mariadb /var/run/mariadbd +mysqld -u mariadb --datadir=/home/mariadb/database/mariadb --explicit_defaults_for_timestamp -P 12048" #+END_SRC If you want to run as root you may have to set : /etc/my.cnf -: [mysqld] +: [mariadbd] : user=root +You also need to set + +: ft_min_word_len = 3 + +To make sure word text searches (shh) work and rebuild the tables if +required. + To check error output in a file on start-up run with something like -: mysqld -u mysql --console --explicit_defaults_for_timestamp --datadir=/gnu/mysql --log-error=~/test.log +: mariadbd -u mariadb --console --explicit_defaults_for_timestamp --datadir=/gnu/mariadb --log-error=~/test.log -Other tips are that Guix installs mysqld in your profile, so this may work +Other tips are that Guix installs mariadbd in your profile, so this may work -: /home/user/.guix-profile/bin/mysqld -u mysql --explicit_defaults_for_timestamp --datadir=/gnu/mysql +: /home/user/.guix-profile/bin/mariadbd -u mariadb --explicit_defaults_for_timestamp --datadir=/gnu/mariadb When you get errors like: -: qlalchemy.exc.IntegrityError: (_mysql_exceptions.IntegrityError) (1215, 'Cannot add foreign key constraint') +: qlalchemy.exc.IntegrityError: (_mariadb_exceptions.IntegrityError) (1215, 'Cannot add foreign key constraint') you may need to set @@ -196,35 +174,6 @@ http://biogems.info/contrib/genenetwork/gn2.svg See [[development.org]]. -* Running ElasticSearch - -In order to start up elasticsearch: -Penguin - change user to "elasticsearch" and use the following command: "env JAVA_HOME=/opt/jdk-9.0.4 /opt/elasticsearch-6.2.1/bin/elasticsearch" - - -** SystemD - -New server - as root run "systemctl restart elasticsearch" - -#+BEGIN_SRC -tux01:/etc/systemd/system# cat elasticsearch.service -[Unit] -Description=Run Elasticsearch - -[Service] -ExecStart=/opt/elasticsearch-6.2.1/bin/elasticsearch -Environment=JAVA_HOME=/opt/jdk-9.0.4 -Environment="ES_JAVA_OPTS=-Xms1g -Xmx8g" -Environment="PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/jdk-9.0.4/bin" -LimitNOFILE=65536 -StandardOutput=syslog -StandardError=syslog -User=elasticsearch - -[Install] -WantedBy=multi-user.target -#+END_SRC - * Read more If you want to understand the architecture of GN2 read @@ -708,3 +657,31 @@ The following derivations would be built: <pjotrp> I am building it on guix.genenetwork.org right now [10:09] <user01> nice [10:10] #+end_src + +* NOTES + +** Deploying GN2 official + +Let's see how fast we can deploy a second copy of GN2. + +- [-] Base install + + [X] First install a Debian server with GNU Guix on board + + [X] Get Guix build going + - [X] Build the correct version of Guix + - [X] Check out the correct gn-stable version of guix-bioinformatics http://git.genenetwork.org/pjotrp/guix-bioinformatics + - [X] guix package -i genenetwork2 -p /usr/local/guix-profiles/gn2-stable + + [X] Create a gn2 user and home with space + + [X] Install redis (currently debian) + - [X] add to systemd + - [X] update redis.cnf + - [X] update database + + [X] Install mariadb (currently debian mariadb-server) + - [X] add to systemd + - [X] system stop mysql + - [X] update mysql.cnf + - [X] update database (see gn-services/services/mariadb.md) + - [X] check tables + + [ ] run gn2 (rust-qtlreaper not working) + + [X] update nginx + + [ ] install genenetwork3 + - [ ] add to systemd |