Merge https://github.com/zsloan/genenetwork

Resolved conflicts:
	wqflask/base/trait.py
	wqflask/wqflask/correlation/correlationFunction.py
	wqflask/wqflask/correlation/correlation_function.py
	wqflask/wqflask/correlation/correlation_functions.py
	wqflask/wqflask/correlation/show_corr_results.py

3 files changed, 283 insertions, 30 deletions

diff --git a/misc/gn_installation_notes.txt b/misc/gn_installation_notes.txt
index bad21226..6329586f 100644
--- a/misc/gn_installation_notes.txt
+++ b/misc/gn_installation_notes.txt
@@ -1,6 +1,37 @@
+UFW - default firewall confirguation tool for Ubuntu; eases iptables firewall configuration
+
+Enable UFW:
+sudo ufw enable
+Check status of UFW:
+sudo ufw status verbose
+
+Limit ssh:
+sudo ufw limit ssh
+
+Enable http, https:
+sudo ufw allow http
+sudo ufw allow https
+
+===========================================
+
+http://evilrouters.net/2011/02/18/using-etckeeper-with-git-on-ubuntu/
+
+Install etckeeper:
+sudo apt-get install etckeeper
+
+Edit /etc/etckeeper/etckeeper.conf to comment out VCS="bzr" and comment in VCS="git"
+
+Initialize repository:
+sudo etckeeper init
+
+Do initial commit:
+sudo etckeeper commit "Initial commit."
+
+===========================================
+
 Get git:
 
-sudo apt-get install git-core
+sudo apt-get install git
 
 ===========================================
 
@@ -16,9 +47,10 @@ git pull origin flask(or whatever the branch is)
 Search for package with a specified file that can be installed with apt-get
 apt-file search _______
 
-Install from requirements.txt:
-pip install -r gene/wqflask/requirements.txt -t ve27
+============================================
 
+Create trash directory:
+mkdir ~/.trash
 
 ============================================
 
@@ -29,19 +61,6 @@ dpkg -l | less
 
 ===========================================
 
-Using Yolk
-
-Install Yolk:
-pip install yolk
-
-Check packages installed in this virtual environment:
-yolk -l
-
-Checks packages that have updates available:
-yolk -U
-
-===========================================
-
 Installing virtualenv:
 sudo pip install virtualenv
 
@@ -53,20 +72,31 @@ source ~/ve27/bin/activate
 
 ===========================================
 
-Installing yaml
+Install libmysqlclient-dev (Mysql-Python dependency)
+sudo apt-get install libmysqlclient-dev
 
-Install libyaml-dev:
-sudo apt-get install libyaml-dev
+Install python-dev (numpy dependency)
+sudo apt-get install python-dev
 
-Install yaml:
-pip install pyyaml
+Install scipy dependencies:
+sudo apt-get install libatlas-base-dev gfortran g++
 
 ===========================================
 
-Install MySQL Client
+Install pip:
+sudo apt-get install python-pip
 
-To fix error "mysql_config not found" while installing packages with pip:
-sudo apt-get install libmysqlclient-dev
+REMEMBER TO SOURCE VE BEFORE INSTALLING 
+
+Comment out in requirements.txt:
+Reaper
+numarray
+
+Before installing from requirements.txt, install numpy separately:
+pip install numpy==1.7.0 (or whatever version we're using)
+
+Install from requirements.txt (after activating virtualenv):
+pip install -r gene/misc/requirements.txt
 
 ===========================================
 
@@ -75,10 +105,7 @@ Installing QTL Reaper
 wget http://downloads.sourceforge.net/project/qtlreaper/qtlreaper/1.1.1/qtlreaper-1.1.1.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fqtlreaper%2Ffiles%2Flatest%2Fdownload&ts=1358975786&use_mirror=iweb
 mv -v qtlreaper-1.1.1.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fqtlreaper%2Ffiles%2Flatest%2Fdownload&ts=1358975786&use_mirror=iweb qtlreaper-1.1.1.tar.gz
 tar xvf qtlreaper-1.1.1.tar.gz (to unzip)
-python setup.py build
-sudo mkdir /home/zas1024/ve27/include/python2.7/Reaper
-sudo chown /home/zas1024/ve27/include/python2.7/Reaper (or whereever the directory is; the problem
-involved the fact that doing "sudo python setup.py install" doesn't install within the virtualenv)
+mkdir /home/zas1024/ve27/include/python2.7/Reaper
 python setup.py install
 
 ===========================================
@@ -92,6 +119,153 @@ sudo python setup.py install
 
 ===========================================
 
+Installing nginx
+
+sudo -s
+nginx=stable # use nginx=development for latest development version
+echo "deb http://ppa.launchpad.net/nginx/$nginx/ubuntu lucid main" > /etc/apt/sources.list.d/nginx-$nginx-lucid.list
+apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C300EE8C
+apt-get update 
+apt-get install nginx
+
+Create configuration file in ~/gene/wqflask/other_config/nginx_conf/ (modeled off of the others)
+Create symbolic link to config file in /etc/nginx/sites-enabled/:
+ln -s dir_to_link_to linking_dir
+
+Run nginx:
+sudo /usr/sbin/nginx
+
+============================================
+
+Copy over zach_settings.py to /home/zas1024 directory (or whatever is home directory)
+
+export WQFLASK_SETTINGS=~/gene/wqflask/cfg/zach_settings.py (or wherever file is located)
+
+============================================
+
+Install MySQL Server
+
+sudo apt-get install mysql-server
+
+mkdir /mnt/big
+fdisk /dev/sdb
+m: for help
+n: new partion
+...
+w: write to table and exit
+
+Start MySQL server:
+service mysql start
+
+Stop MySQL server:
+service mysql stop
+
+Change root password:
+mysql> UPDATE mysql.user SET Password=PASSWORD('your password') WHERE User='root';
+
+Setup accounts in MySQL (first need to delete anonymous/non-root accounts):
+#; use mysql;
+#; select * from user;
+#; delete from user where Host!="localhost";
+#; delete from user where User!="root";
+#; update user set Password = Password('yourpassword') where User='root';
+#; GRANT ALL ON *.* TO 'yourusername'@'%' IDENTIFIED BY 'yourpassword';
+#; select * from user;
+
+Use Percona to optimise my.cfg file:
+https://tools.percona.com/wizard
+
+To check # CPUs:
+grep -c processor /proc/cpuinfo
+
+To check memory, can use htop
+
+============================================
+
+Check RSA key:
+ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key
+
+03:2c:d7:01:01:f0:31:3a:c8:df:e4:98:62:2c:59:d2  root@penguin (RSA)
+
+============================================
+
+Installing redis server
+
+In the dls directory:
+$ wget http://redis.googlecode.com/files/redis-2.6.14.tar.gz
+$ tar xzf redis-2.6.14.tar.gz
+$ cd redis-2.6.14
+$ make
+
+Run redis (from inside redis-2.6.14 directory):
+src/redis-server
+
+For the rest of the installation instructions, refer to:
+http://redis.io/topics/quickstart
+
+To continue installation (from the README):
+
+In order to install Redis binaries into /usr/local/bin just use:
+
+    % make install
+
+Make install will just install binaries in your system, but will not configure
+init scripts and configuration files in the appropriate place. This is not
+needed if you want just to play a bit with Redis, but if you are installing
+it the proper way for a production system, we have a script doing this
+for Ubuntu and Debian systems:
+
+    % cd utils
+    % ./install_server
+
+The script will ask you a few questions and will setup everything you need
+to run Redis properly as a background daemon that will start again on
+system reboots.
+
+In the redis config file (/etc/redis/6379.conf) uncomment the following line:
+bind 127.0.0.1
+
+This is so only the local computer has access
+
+============================================
+
+Check RSA key:
+ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key
+
+03:2c:d7:01:01:f0:31:3a:c8:df:e4:98:62:2c:59:d2  root@penguin (RSA)
+
+============================================
+
+Using Yolk
+
+Install Yolk:
+pip install yolk
+
+Check packages installed in this virtual environment:
+yolk -l
+
+Checks packages that have updates available:
+yolk -U
+
+===========================================
+
+Installing yaml
+
+Install libyaml-dev:
+sudo apt-get install libyaml-dev
+
+Install yaml:
+pip install pyyaml
+
+===========================================
+
+Install MySQL Client
+
+To fix error "mysql_config not found" while installing packages with pip:
+sudo apt-get install libmysqlclient-dev
+
+===========================================
+
 Installing R
 
 sudo apt-get install r-base-dev
@@ -132,7 +306,7 @@ Start up virtual environment:
 source ~/ve27/bin/activate
 
 To set WQFLASK_SETTINGS environment variable:
-export WQFLASK_SETTINGS=~/gene/wqflask/cfg/zach_settings.py (or wherever file is located)
+export WQFLASK_SETTINGS=~/zach_settings.py (or wherever file is located)
 
 To change screen environment variable (if man not working or to get color, for example):
 export TERM=screen
@@ -146,4 +320,9 @@ Run web server:
 Run server:
 python runserver.py
 
+===========================================
+
+To do full upgrade (as opposed to apt-get upgrade)
+sudo aptitude full-upgrade
+
 ===========================================
\ No newline at end of file
diff --git a/misc/notes.txt b/misc/notes.txt
index 92ed267b..91a0e67c 100644
--- a/misc/notes.txt
+++ b/misc/notes.txt
@@ -92,6 +92,15 @@ python runserver.py
 
 ===========================================
 
+UFW - default firewall confirguation tool for Ubuntu; eases iptables firewall configuration
+
+Enable UFW:
+sudo ufw enable
+Check status of UFW:
+sudo ufw status verbose
+
+===========================================
+
 Update user password:
 use mysql;
 UPDATE user SET password=PASSWORD('whatever') WHERE user='webqtl';
@@ -117,7 +126,7 @@ pylint --help
 
 Start screen session:
 
-If "no option -R":
+If "no option -R" or running for first time:
 byobu-select-backend
 2. screen
 
@@ -236,6 +245,11 @@ To iterate through dictionary items: for X, Y in MyDictionary.items():
 
 ===========================================
 
+Change import path from command line
+export PYTHONPATH=~/gene/wqflask (or whatever)
+
+===========================================
+
 Create symbolic link:
 ln -s dir_to_link_to linking_dir
 
@@ -255,7 +269,33 @@ grep -ir (search string) (directory)
 
 ===========================================
 
+Change owner/group:
+
+chown zas1024 somefile (change owner of somefile to zas1024)
+chown zas1024: somefile (change owner and group of somefile to zas1024)
+sudo chown root: somefile (change owner/group of somefile to root)
+
+Change permissions:
+
+chmod g-wr (remove write/read privileges for group)
+chmod g+w (add write privileges for group)
+chmod u+rwx (add read, write, and execute privileges for user)
+etc.
+
+===========================================
+
 MySQL Things -
 
 Create fulltext (MATCH AGAINST syntax) index:
 CREATE FULLTEXT INDEX __________ ON Table (Column);
+
+If <4 character searches not working:
+Add ft_min_word_len = 2 (to make te minimum characters 2) to config file
+Rebuild relevant tables with fulltext indexes with "REPAIR TABLE _______ QUICK"
+
+===========================================
+
+To delete all .pyc files (for example if there's some issue you're having trouble resolving):
+find . -name '*.pyc' -delete
+http://stackoverflow.com/questions/785519/how-do-i-remove-all-pyc-files-from-a-project
+
diff --git a/misc/requirements.txt b/misc/requirements.txt
new file mode 100644
index 00000000..ec02a2c2
--- /dev/null
+++ b/misc/requirements.txt
@@ -0,0 +1,34 @@
+BeautifulSoup==3.2.1
+Flask==0.9
+Flask-Login==0.1.3
+Flask-Mail==0.7.6
+Flask-Principal==0.3.4
+Flask-SQLAlchemy==0.16
+Flask-Security==1.6.0
+Flask-WTF==0.8.3
+Jinja2==2.6
+MySQL-python==1.2.4
+PyYAML==3.10
+#Reaper==1.0
+Reindent==0.1.1
+SQLAlchemy==0.8.0
+WTForms==1.0.3
+Werkzeug==0.8.3
+apache-libcloud==0.12.3
+argparse==1.2.1
+blinker==1.2
+itsdangerous==0.17
+logging-tree==1.2
+logilab-astng==0.24.3
+logilab-common==0.59.1
+#numarray==1.5.2
+numpy==1.7.0
+passlib==1.6.1
+pp==1.6.3
+pylint==0.27.0
+redis==2.7.2
+requests==1.1.0
+scipy==0.11.0
+simplejson==3.0.7
+wsgiref==0.1.2
+yolk==0.4.3