Adding code descriptions

3 files changed, 307 insertions, 17 deletions

diff --git a/BNW_data_flow.txt b/BNW_data_flow.txt
new file mode 100644
index 00000000..665122fe
--- /dev/null
+++ b/BNW_data_flow.txt
@@ -0,0 +1,166 @@
+This file will provide an overview of data flow within BNW.
+This focuses on what happens after clicking on "Learn network model from data" on homepage (home.php).
+
+Step 1: Upload data
+
+Clicking "Learn network model from data" on homepage (home.php) goes to: 
+bn_file_load_gom.php which is responsible for uploading the input file
+   and preparing it for structure/parameter learning.
+
+Specifically, uploading a file runs the 'run_prep_input' script in /sourcecodes.
+This runs the prepareInput.m Matlab/Octave function.
+
+One additional note:
+BNW assigns each network a randomly generated 3 letter prefix when the file is uploaded.
+I will use ??? to signify this prefix.
+
+The following 13 files are written to /sourcecodes/data directory during this step:
+1) ???continuous_input_orig.txt: This is an exact copy of the input file uploaded by the user.
+2) ???continuous_input.txt: Input file after processing by prepareInput.m.
+   -A couple of notable changes are made to the input file.
+     a) An additional header type row containing the variable/node type is added as the second row of the file.
+         See ???type.txt below.
+     b) Discrete variable values are recoded to be integers.
+3) ???input_desc.txt: Variable descriptions that can be viewed by user clicking on "View uploaded variables and data"
+4) ???nnode.txt: The number of variables/nodes in the input file (# of columns in input file).
+5) ???nrows.txt: The number of cases/samples in the input file (# of rows in the input file, excluding header lines).
+
+The following 3 files are used to keep track of variable names, variable types, and discrete variable states.
+6) ???name.txt: The names of the variables in the input file in the same order as they are presented in the input file.
+7) ???nlevels.txt: The original possible states of discrete variables in the input file. The order of these states
+      corresponds to the integers in ???continuous_input.txt (i.e., the first listed state for the variable has a value
+      of 1 in ???continuous_input.txt, the second listed state has a value of 2, . . .)
+8) ???type.txt: The type of each variable/node. Continuous nodes have a value of 1. Discrete nodes have a value equal to the
+      number of possible states.
+
+The following 5 files contain default values of structure learning options.
+They will be modified if the structural constraint interface is used.
+9) ???ban.txt: List of banned edges (will contain only a header line at this point).
+10) ???white.txt: List of required edges (will contain only a header line at this point).
+11) ???k.txt: Number of top-scoring models to include in model averaging (default value is 1, indicating no model averaging).
+12) ???parent.txt: Maximum number of parents for each node during structure learning (default value is 4).
+13) ???thr.txt: Threshold value to include edges after model averaging (default value is 0.5). 
+
+Step 2a: Structure learning with default parameters
+
+Click "Perform Bayesian network modeling using default settings"
+This is a link to executionprogress.php which just displays a "progress bar" and starts structure learning.
+To start structure learning, it runs execute_bn_gom.php.
+execute_bn_gom.php which runs the run.sh script and prepares a couple of files.
+
+The run.sh contains the commands to run the structure learning codes.
+
+Output files from structure learning that are written to /sourcecodes/data:
+1) ???structure_input.txt: The structure matrix, containing the edges included in the network model.
+   The format of this file is a header line containing the variable names.
+   The rest is the strucure matrix where 1 indicates an edge is included in the network and a 2 is not.
+   The matrix had dimensions n x n where n is the number of variables.
+   A '1' in the ith column in the jth row indicates that there is an edge from variable i to variable j.
+2) ???structure_input_temp.txt: Structure matrix with model averaging data.
+   This file has the model averaging probability of each edge.
+   The ???structure_input.txt file is generated from this file. If the value of an edge is this file
+   is greater than the value in ???thr.txt, the edge is included in the network (a '1' in ???structure_input.txt).
+
+execute_bn_gom.php also creates the following graphviz:
+3) ???grviz_name_file.txt: A list of the names of the variables that is used by graphviz/dot. 
+4) ???graphviz.txt: Input file for dot that creates the edges included in the network.
+   0 is the first node listed in ???grviz_name_file.txt, 1 is the second node, . . .
+
+
+Step 2b: Structure learning using the structural constraint interface
+
+Before the process described in Step 2a, additional constraints or changes to default parameters 
+  can be added using the structural constraint interface.
+
+This can be accessed by clicking on "Go to structure learning settings and the BNW
+     structural constraint interface" which links to create_tiers_gom.php
+
+The top section of create_tiers_gom.php is "Global structure learning settings"
+  Modifying this section will change the values in ???parent.txt, ???k.txt, and ???thr.txt.
+
+The remaining sections will modify the ???ban.txt and ???white.txt files.
+
+Clicking on "Perform Bayesian network modeling" goes to executionprogress.php and Step 2a is followed.
+
+
+Step 3: Parameter learning of the original network.
+
+The step begins near the end of execute_bn_gom.php file where the run_octave script is run.
+
+run_octave starts up the runBN_initial.m Matlab/Octave script.
+This prepares parameter learning for the data file in ???continuous_input.txt and 
+    network structure defined in ???structure_input.txt.
+
+runBN_initial makes 2 main output files and 2 files that may be used by other Matlab/Octave scripts.
+1) ???net_figure.txt: This file provides the data that is used to draw the network in BNW.  I will describe 
+      the format of this file elsewhere.
+2) ???parameters.txt: The parameters of the network using the original data set.
+    This file is broken into different sections that correspond to each variable/node.
+    Each section has the name of the variable and whether it is discrete or continuout.
+    If it is discrete, the number of possible states is also provided.
+    For discrete nodes, the percentage of each state is listed.
+    For continuous nodes, the mean and standard deviation of the Gaussian fit to that variable is provided.
+3) ???map.txt: The standard deviations and means of the original data set.
+    The parameter learning tools used in BNW work best if continuous data is normalized.
+    Each line of this file provides the variable name, the standard deviation, and the mean of the original data
+         for each variable for use in standardizing data in the Matlab/Octave scripts.
+4) ???mapdata.txt: The topological order of the nodes used in parameter learning.
+    Sometimes before parameter learning, the nodes are resorted so they are in topological order (i.e., 
+      parents are always listed before parents).
+
+Step 4: Display of the network.
+  
+This step is initiated at the end of execute_bn_gom.php when layout.php is opened.
+
+layout.php is the main page that is used when the network is displayed.
+
+The network is drawn in a division using either 
+network_layout_evd.php if Evidence mode is being used or 
+network_layout_inv.php if Intervention mode is being used.
+
+These php files basically just read and ???net_figure.txt and run dot using the graphviz file
+   described above to draw the network.
+
+Step 5a: Adding evidence or intervention to the network.
+
+Clicking on a node and entering evidence/intervention runs add_evd.php/add_inv.php.
+
+This creates 3 files:
+(I am going to describe these as evidence here, intervention mode is the same.)
+1) ???vardata.txt: The value of the evidence that was entered.
+2) ???varname.txt: The name of the node for which evidence was entered.
+3) ???var.txt: The index of the node for which evidence was entered in the ???mapdata.txt file.
+If evidence is entered for more than one node, these files will contain tab-separated lists.
+
+add_evd.php/add_inv.php then runs a script called either run_octave_evd or run_octave_inv.
+
+This script runs the Predictmultiple or Predictmultipleintervention Matlab/Octave functions.
+
+The output files from this script are:
+4) ???net_figure_new.txt: The net_figure file after entering evidence/intervention.
+5) ???parameters_ev.txt: The parameters of the network considering the evidence/intervention.
+
+Then, network_layout_evd_2.php or network_layout_inv_2.php is used to draw the network with the evidence/intervention.
+
+Step 5b: Additional files created in Intervention mode.
+
+In Intervention mode, the network structure can change. The node with intervention no longer depends on its parents.
+
+Several files must be created to account for this. They are the same as described above, but with the new network structure.
+1) ???graphviz_new.txt
+2) ???grviz_name_file_new.txt
+3) ???structure_input_new.txt
+
+
+
+Structure learning:
+
+
+
+
+
+
+
+
+
+
diff --git a/BNW_file_overview.txt b/BNW_file_overview.txt
index 881605ef..40528f49 100644
--- a/BNW_file_overview.txt
+++ b/BNW_file_overview.txt
@@ -1,4 +1,4 @@
-This file will provide an overview of the directory structure and files used by BNW.

+This file will provide an overview of the BNW code.

 

 At this point, BNW requires two packages that are not currently included in the BNW directory.

 These are Octave and dot/graphviz. The BNW code assumes:

@@ -6,9 +6,9 @@ a) dot can be executed using /usr/bin/dot
 b) octave can be executed using /usr/bin/octave

 

 BNW is currently located at /var/www/html/compbio/BNW_x.xx

-(The current development version is BNW_1.12).

+(Current development version is BNW_1.12).

 

-The main directory contains two basic files (index.html and home.php) and two directories.

+It contains two basic files (index.html and home.php) and two directories.

 home.php can be modified/updated to use other versions of BNW

 

 downloads: This contains the downloadable version of the BNW structure learning package.

@@ -17,13 +17,13 @@ sourcecodes: This contains the files and codes that are used by the webserver.
 The main purpose of this file is to provide an overview of the files that are contained in 'sourcecodes'.

 

 1. Directories within sourcecodes:

-a) bnt-master: This contains the code for Bayes Net Toolbox which is used in parameter learning in BNW.

-	It has been slightly modified from its original version to work with Octave instead of MATLAB.

+a) bnt-master: This contains the code for BayesNetToolbox which is used in parameter learning in BNW.

+	It has been slightly modified to work with Octave instead of MATLAB.

 

 b) parameter_learning: This contains the MATLAB/Octave code that we have written for parameter_learning.

-        It also prepares data that has been uploaded to BNW so it can be used with the structure learning code.

+        It also prepares data that has been uploaded to BNW for use with the structure learning code.

 

-c) k-best and localscore: These two directories contain the sourcecode used for structure learning.

+c) k-best and localscore: These two directories contains the code used for structure learning.

 

 d) BNW_workflow_test_files, example_datasets, and examples: These three directories contain files that are used by the tutorials and examples on the BNW homepage.

 

@@ -32,15 +32,14 @@ e) data: The data for files generated by using BNW is written to this directory.
 

 2. Files within sourcecodes:

 a) Major BNW php files:

-Many of these will be described in more detail in another file that I am working on.

+Many of these will be described in more detail in another file.

 bn_file_load_gom.php: Used in uploading data to be used for structure learning.

 create_tiers_gom.php: Structural constraint interface.

 home_upload.php: Used in uploading a data file and a structure file when the structure is known. 

 layout_example.php: Main php file for displaying/making predictions with network using example data.

-layout.php: Main php file for displaying a network and using it to make predictions.

-net_structure.php: Used in uploading the structure file when the network structure is known.

-upload_structure_file.php: Used in uploading the structure file when the network structure is known.

-The following are self-explanatory:

+layout.php: Main php file for displaying/making predictions with network using user uploaded data.

+net_structure.php: Used in uploading structure file when the network structure is known.

+upload_structure_file.php: Used in uploading structure file when the network structure is known.

 faq.php

 getting_started.php

 help.php

@@ -71,7 +70,6 @@ my_new_style.css
 sf-homepage.css

 

 e) PHP files that support layout.php or layout_example.php:

-I will describe these further in another file.

 add_evd_example.php

 add_evd.php

 add_inv_example.php

@@ -111,16 +109,16 @@ graphviz_structure.php
 

 mat_structure.php

 -Called by add_inv.php and add_inv_example.php when making predictions after an intervention.

--Interventions can change the network structure and this file helps account for these changes.

+-Interventions can change the network structure and this file help account for these changes.

 

 matrix.php

--Allows for viewing the structure matrix file.

+-Allows for display of the structure matrix file.

 

 input_check.php

--Allows for viewing the input file and variable description file after it is uploaded.

+-Allows for display of the input file and variable description files.

 

 parameter_display.php

--Allows for viewing the network parameters of the original data set and after entering evidence/intervention.

+-Allows for display of the network parameters of the original data set and after entering evidence/intervention.

 

 

 

diff --git a/BNW_network_file_overview.txt b/BNW_network_file_overview.txt
new file mode 100644
index 00000000..633aa3ac
--- /dev/null
+++ b/BNW_network_file_overview.txt
@@ -0,0 +1,126 @@
+This file will provide an overview of the BNW code.
+
+At this point, BNW requires two packages that are not currently included in the BNW directory.
+These are Octave and dot/graphviz. The BNW code assumes:
+a) dot can be executed using /usr/bin/dot
+b) octave can be executed using /usr/bin/octave
+
+BNW is currently located at /var/www/html/compbio/BNW_x.xx
+(Current development version is BNW_1.12).
+
+It contains two basic files (index.html and home.php) and two directories.
+home.php can be modified/updated to use other versions of BNW
+
+downloads: This contains the downloadable version of the BNW structure learning package.
+
+sourcecodes: This contains the files and codes that are used by the webserver.
+The main purpose of this file is to provide an overview of the files that are contained in 'sourcecodes'.
+
+1. Directories within sourcecodes:
+a) bnt-master: This contains the code for BayesNetToolbox which is used in parameter learning in BNW.
+	It has been slightly modified to work with Octave instead of MATLAB.
+
+b) parameter_learning: This contains the MATLAB/Octave code that we have written for parameter_learning.
+        It also prepares data that has been uploaded to BNW for use with the structure learning code.
+
+c) k-best and localscore: These two directories contains the code used for structure learning.
+
+d) BNW_workflow_test_files, example_datasets, and examples: These three directories contain files that are used by the tutorials and examples on the BNW homepage.
+
+e) data: The data for files generated by using BNW is written to this directory. It also contains several sub-directories that contain example data and networks.
+
+
+2. Files within sourcecodes:
+a) Major BNW php files:
+Many of these will be described in more detail in another file.
+bn_file_load_gom.php: Used in uploading data to be used for structure learning.
+create_tiers_gom.php: Structural constraint interface.
+home_upload.php: Used in uploading a data file and a structure file when the structure is known. 
+layout_example.php: Main php file for displaying/making predictions with network using example data.
+layout.php: Main php file for displaying/making predictions with network using user uploaded data.
+net_structure.php: Used in uploading structure file when the network structure is known.
+upload_structure_file.php: Used in uploading structure file when the network structure is known.
+faq.php
+getting_started.php
+help.php
+home.php
+workflow.php
+
+b) Files related to the structure learning package:
+build.sh
+network_score
+structure
+
+c) Files related to home page, help files, example data sets, and tutorials:
+BNW_overview.png
+BNW_workflow_1.htm
+BNW_workflow_2.htm
+BNW_workflow_net1.htm
+BNW_workflow_sci.htm
+index.html
+example.php
+examplefilecopy.sh
+
+d) General html/css/inc files:
+header_batchsearch.inc
+header_new.inc
+js-graph-it.css
+js-graph-it.js
+my_new_style.css
+sf-homepage.css
+
+e) PHP files that support layout.php or layout_example.php:
+add_evd_example.php
+add_evd.php
+add_inv_example.php
+add_inv.php
+network_layout_evd_2_example.php
+network_layout_evd_2.php
+network_layout_evd_2xample.php
+network_layout_evd.php
+network_layout_inv_2_example.php
+network_layout_inv_2.php
+network_layout_inv_example.php
+network_layout_inv.php
+
+f) Scripts used in structure and parameter learning:
+run_octave: perform parameter learning with original data set and no evidence or intervention
+run_octave_evd: perform parameter learning when considering evidence
+run_octave_inv: perform parameter learning when considering intervention
+run_prep_input: perform prepare data uploaded to BNW for use with structure/parameter learning packages
+run.sh: perform structure learning
+
+g) Files that perform specific functions:
+tier_description_processing_gom.php
+-Used by create_tiers_gom.php for the structure constraint interface.
+
+clear_example.php, clear.php, and filecopy.sh
+-Used by layout.php to clear entered evidence/intervention 
+
+execute_bn_gom.php
+-Runs the script (run.sh) that performs structure learning after uploading data/adding constraints
+-It is actually called by executionprogress.php
+
+executionprogress.php, runtime_check.php, k_only_table.txt, and maxparent_table.txt
+-Used to provide an estimate of how long structure learning will take.
+
+graphviz_structure.php
+-Called by net_structure.php and used to generate files when users upload data with a known structure.
+
+mat_structure.php
+-Called by add_inv.php and add_inv_example.php when making predictions after an intervention.
+-Interventions can change the network structure and this file help account for these changes.
+
+matrix.php
+-Allows for display of the structure matrix file.
+
+input_check.php
+-Allows for display of the input file and variable description files.
+
+parameter_display.php
+-Allows for display of the network parameters of the original data set and after entering evidence/intervention.
+
+
+
+
+