| Age | Commit message (Collapse) | Author | |
|---|---|---|---|
| 8 days | Compute the data_link_ids from the request results. | Frederick Muriuki Muriithi | |
| 8 days | Fix bug in data for creating a new resource. | Frederick Muriuki Muriithi | |
| 8 days | Compute request data for …/data/link/phenotypes endpoint. | Frederick Muriuki Muriithi | |
| 8 days | Add debug logging to track process. | Frederick Muriuki Muriithi | |
| 8 days | Fix return: Don't use `RETURNING *`. | Frederick Muriuki Muriithi | |
| Fetching the results of the insert query with `RETURNING *` does not work as expected with MySQLdb. This commit, thus, returns the explicitly computed values instead. | |||
| 12 days | Break up long lines. | Frederick Muriuki Muriithi | |
| 12 days | `update_auth()` scaffolding | Frederick Muriuki Muriithi | |
| Add untested scaffolding for how the `update_auth()` function will work. Include notes to help with figuring the feature out. | |||
| 12 days | Request token for use with script to update authorisations | Frederick Muriuki Muriithi | |
| After the script has completed loading the data into the database, the user who uploads the data will need to access the data they've uploaded. This commit requests a new token to use with the script for updating the authorisations as appropriate, to grant the user access to their data. | |||
| 12 days | Only update authorisation after all data is entered. | Frederick Muriuki Muriithi | |
| Pull the auth update function out of the database connection contextmanager to prevent any problems with an auth update from causing a rollback of all the data. We can always manually update the authorisation later, therefore, we do not want a failure in that causing the (potentially) time-consuming data entry process from being rolled back. | |||
| 2025-06-03 | Add more logging to track progress. | Frederick Muriuki Muriithi | |
| 2025-06-03 | Save data with `LOAD DATA INFILE …` query | Frederick Muriuki Muriithi | |
| To help speed up the saving of the data (for really huge files) into the database, use the `LOAD DATA INFILE …` command if available, and if not fallback to the one using raw queries. | |||
| 2025-06-03 | Extract building of data items into helper function. | Frederick Muriuki Muriithi | |
| 2025-06-03 | Increase number of rows per query. | Frederick Muriuki Muriithi | |
| 2025-06-03 | Debug the query: useful to figure out what's happening. | Frederick Muriuki Muriithi | |
| 2025-06-03 | Memory saving: Don't reload saved data | Frederick Muriuki Muriithi | |
| We already have all the data in memory, so we do not need to reload it and keep yet another copy. | |||
| 2025-06-02 | Compute data in files lazily. | Frederick Muriuki Muriithi | |
| 2025-06-02 | Process NA-strings in the worker script. | Frederick Muriuki Muriithi | |
| 2025-06-02 | Add note in documentation. | Frederick Muriuki Muriithi | |
| 2025-06-02 | Revert "Convert N/A values to NoneType objects." | Frederick Muriuki Muriithi | |
| This reverts commit 6870c08d484f482cc2f2501d28b474636dd0810d. The "read_csv_file" should return the data in the CSV file, as is, before processing the N/A values. | |||
| 2025-06-02 | Update placeholder function | Frederick Muriuki Muriithi | |
| We'll probably provide a token to use to update the authorisation for the new data. The commit also explicitly raises a `NotImplement` exception for now. | |||
| 2025-06-02 | Add missing import. | Frederick Muriuki Muriithi | |
| 2025-06-02 | Fix bug: Compute data correctly. Replace N/A strings with NoneType. | Frederick Muriuki Muriithi | |
| 2025-06-02 | Add some debugging statements. | Frederick Muriuki Muriithi | |
| 2025-06-02 | Parametrize the "DataId" column for different tables. | Frederick Muriuki Muriithi | |
| 2025-06-02 | Explicitly query for newly entered data. | Frederick Muriuki Muriithi | |
| The `cursor.executemany()` call only returned the last row that was inserted, rather than ALL the rows. This is not the correct thing to do, therefore, this commit fixes that by explicitly querying for all the newly entered data. | |||
| 2025-06-02 | Convert N/A values to NoneType objects. | Frederick Muriuki Muriithi | |
| 2025-06-02 | Pass the phenotypes name-to-id map to functions. | Frederick Muriuki Muriithi | |
| 2025-06-02 | Compute the map from a phenotype's name to its ID. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Refactor: Extract common pattern into separate function | Frederick Muriuki Muriithi | |
| Extract the common pattern into a separate, more generalized function and pass the new function the data it requires to perform its tasks for the different file types. | |||
| 2025-05-30 | Add typing to load_data function and invoke it. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Save standard errors and counts. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Get the species, population and dataset. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Add more tables to lock list. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Fix query: Add missing field and parameter. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Fix function names. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Fix processing of numerical data to save to the database. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Fix file paths: append the path to the extraction directory. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Add new file type. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Add dataset_id and bundle path to job metadata. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Fix typo: Fix table name. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Bug: Loop until the data is exhausted. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Fix typo, and bug. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Fix indentation. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Refactor out common `__row_to_dataitems__` function. | Frederick Muriuki Muriithi | |
| 2025-05-30 | Use full table names rather than aliases | Frederick Muriuki Muriithi | |
| Using aliases leads to errors when you have to use table locking to prevent data corruption. This commit updates queries to use the full table names rather than aliases, in order to prevent such troubles. | |||
| 2025-05-28 | Bug: Fix actual data value, not result set. | Frederick Muriuki Muriithi | |
| 2025-05-28 | Bug: Add missing commas. | Frederick Muriuki Muriithi | |
| 2025-05-28 | Maintain single list of tables — build logs and query from list | Frederick Muriuki Muriithi | |
| Maintain a list of tables to lock when running the script and build the logging messages, and lock query from the list for easier maintenance. | |||
| 2025-05-28 | Add missing import. | Frederick Muriuki Muriithi | |
| 2025-05-28 | Pass path to bundle extraction dir, and make paths relative to it. | Frederick Muriuki Muriithi | |
| Pass the path where the bundle was extracted to the function doing the work. Make the paths for the files being used be relative to the extraction directory thus passed. | |||
 |
index : gn-uploader | |
| gn-uploader: Data uploader for GeneNetwork |
| aboutsummaryrefslogtreecommitdiff |