summaryrefslogtreecommitdiff
path: root/Code
AgeCommit message (Collapse)Author
2020-05-04merge_edges.py: make sqlite insert really fast. Commit in the end instead ↵Hui Lan
of commit for each insertion.
2020-05-04merge_edges.py: string does not have isnumeric method.Hui Lan
2020-05-03download_and_map.py: when download_log.txt contains blank lines, this script ↵Hui Lan
will stop working. Make it work when the log file contains blank lines. These blank lines usually appear in the end of the log file.
2020-05-03configure.py: fewer daily downloadsHui Lan
2020-05-02download_and_map.py: lower the requirement of available disk space to make ↵Hui Lan
downloading and mapping proceed.
2020-03-27update_network_by_force.py: the number of %s should be the same as the ↵Hui Lan
number of arguments.
2020-03-27html_network.py: string like 'F3'H' will cause syntax error in JavaScript. ↵Hui Lan
json.dumps could not fix that.
2020-03-27html_network.py: string like 'F3'H' will cause syntax error in JavaScript.Hui Lan
2020-03-27update_network_by_force.py: code for build table of regulators and regulatees.Hui Lan
2020-03-26html_network.py: don't show intermediate resultHui Lan
2020-03-26html_network.py: remove many lines to make it simplerHui Lan
2020-02-15merge_edges.py: remove old function make_new_edge()Hui Lan
2020-02-15update_network_by_force.py: enable visiting the web app each time the ↵Hui Lan
network is updated The Apache web server will be restarted daily to free memory. sudo service apache2 restart
2020-02-15merge_edges.py: now db_fname has a place to printHui Lan
2020-02-15merge_edges.py: an integer in a list prevents the join method from workingHui Lan
2020-02-15update_network_by_force.py: disable visiting the web app each time when ↵Hui Lan
running update_network_by_force.py
2020-02-15merge_edges.py: a more memory efficient method to compute an edge's net strengthHui Lan
Compute an edge's strength on the fly instead of saving everything and then computing the net strength. The new function make_new_edge2 will replace make_new_edge.
2020-02-15merge_edges.py: do not show edge file names in network.logHui Lan
2020-02-15merge_edges.py: datetime.now() dose not work. Should be datetime.datetime.now()Hui Lan
2020-02-14merge_edges.py: write edge file names to network.logHui Lan
When merging many big edge files, the computer may run out of memory. Save the edge files that have been considered thus far and figure out where merging stopped.
2020-02-11merge_edges.py: use the most recent update date as the merged edge's date.Hui Lan
2020-02-11merge_edges.py: add a few comments for function make_new_edge.Hui Lan
2020-02-11merge_edges.py: log more information in network.log.Hui Lan
2020-02-11merge_edges.py: log number of edge files scanned.Hui Lan
2020-02-11merge_edges.py: what is the purpose of variable d.Hui Lan
2020-02-11merge_edges.py: make a better keyHui Lan
Use a combination of target gene ID and tf gene ID as a key. So if we having the following: Target: AT5G09445 AT5G09445 TF: AT1G53910 RAP2.12 Then the key will be "AT5G09445_AT1G53910". Before it was "AT5G09445 AT5G09445 AT1G53910 RAP2.12". This is OK in most cases, as long a gene ID's corresponding gene name is consistent. But if "AT1G53910" has a different gene name, then we will have a DIFFERENT key, which is not what we want.
2020-02-11merge_edges.py: why 10?Hui Lan
2020-02-11merge_edges.py: consider all files in directory EDGE_POOL whose file name ↵Hui Lan
starts with 'edegs'
2020-02-06backup_files.py: remove the cd command - we do not need to cd to a directory ↵Hui Lan
to gzip a file inside it.
2020-02-06backup_files.py: fix misspellingHui Lan
2020-01-21update_network_by_force.py: update G.pickle and SOURCE_NODES.pickle with ↵Hui Lan
edges.txt Visit the following link to update two pickle files used the Webapp, G.pickle and SOURCE_NODES.pickle. http://118.25.96.118/brain/before The visit could be done using the command line tool curl, as follows: curl http://118.25.96.118/brain/before -Hui
2020-01-21merge_edges.py: write all edge information to an SQLite database file called ↵Hui Lan
edges.sqlite When I saved a static html page for each edge (e.g., http://118.25.96.118/static/edges/AT1G20910_AT1G30100_0.html), it took 5GB disk space for saving 1 million html pages. Not very disk space efficient. An alternative is to save all edge information in a database table (i.e., edge), and query this database table for a particular edge. The database file edges.sqlite takes less than 200MB for 1 million edges, requiring 10 times smaller space than the static approach. The reason is that we do not have a lot of HTML tags in the database. Quite happy about that, though it seems that filling a database is a bit slower (2 hours??? for 1 million rows). Also updated two files that were affected: update_network.py and update_network_by_force.py. Now instead of copying 1 million static html page to the Webapp, I just need to copy edges.sqlite to static/edges/. Faster. In the Webapp, I updated start_webapp.py and added a file templates/edge.html for handling dynamic page generation. -Hui
2020-01-18download_and_map.py: write "no enough space" warning message to network log fileHui Lan
If there is no enough space left in the disk, download_and_map.py will refuse to download any data. This can be quite mysterious for a maintainer. So, write the reason to the network log file. The reason is something like: "[download_and_map.py] home directory does not have enough space (only 13 G available)." -Hui
2019-12-26merge_edges.py: save memory by removing the dictionary variable duniqHui Lan
The purpose of duniq is to avoid duplicated edge lines. Now, just make sure we don't insert the same tuple. -Hui
2019-12-12update_network_by_force.py: summarize the association strength in the edge fileHui Lan
It would be interesting to see how edges' association strengths change over time, as time is an input variable for the function that computes the association strength.
2019-12-07create backup_files.py for backing up filesHui Lan
Define a function copy_and_backup_file(src_file, dest_dir) to do backup and compression work. The function copy_and_backup_file is used in update_network_by_force.py. -Hui
2019-12-06draw_subnetwork.py: add a known network (G0) from the thermomorphogenesis paper.Hui Lan
Fixed a bug. Now I close figure (plt.close()) before creating a new one, to avoid that the current figure is drawn on top of the old one. -Hui
2019-12-05draw_subnetwork.py for generating a subnetwork consisting of genes that ↵Hui Lan
might be responsible for thermomorphogenesis Use networkx and matplotlib. Reference: Quint et al. (2016) Molecular and genetic control of plant thermomorphogenesis. Nature Plants.
2019-12-05add count_AGI_in_cDNA.py for counting number of unique genes in a cdna file.Hui Lan
2019-12-04merge_edges.py: run dos2unix on merge_edges.py to remove ^M characters.Hui Lan
2019-12-04merge_edges.py: clean up source code by removing commented lines and editing ↵Hui Lan
the head comments.
2019-12-04brain: add python and R code to local repository.Hui Lan
generated by cgit v1.2.1 (git 2.18.0) at 2025-03-15 15:37:25 +0000