aboutsummaryrefslogtreecommitdiff
path: root/scripts/insert_data.py
AgeCommit message (Collapse)Author
2024-11-22Use gn-libs code for db connection.Frederick Muriuki Muriithi
Use the code in gn-libs to connect to the database, rather than a local module.
2024-07-25Rename module: qc_app --> uploaderFrederick Muriuki Muriithi
2024-03-29Quiet linter.Frederick Muriuki Muriithi
2024-03-22Notify user if identifiers are not consistent.Frederick Muriuki Muriithi
2024-03-22Map names in files to names in database.Frederick Muriuki Muriithi
2024-03-22Fix linting issue.Frederick Muriuki Muriithi
2024-03-20Fix bug: correctly merge standard-error values in file to data in dbFrederick Muriuki Muriithi
`read_datavalues(…)` function returns a dict of the form: ``` { ProbeSetName01: ({…}, …), ProbeSetName02: ({…}, …), ︙ } ``` Previously, the generator would thus try to index into the keys of the datavalues, which were strings, leading to an error. This commit changes the generator to return the values of the datavalues dict as a flattened list of values.
2024-02-28Fix bug: fetch from cursor, not return from cursor.execute(…)load-raw-data-no-parsingFrederick Muriuki Muriithi
2024-02-12Provide better error messagingFrederick Muriuki Muriithi
Provide a better error message that can help the user figure out what they did wrong and fix it for themselves instead of the older, cryptic and extremely unhelpful error message.
2024-01-15Update imports: `take` was moved to different package.Frederick Muriuki Muriithi
2023-12-20Fix ID column keyFrederick Muriuki Muriithi
Fix the wrong assumption that the ID key will always be "ProbeSetID". The key for the ID column could change from study to study or dataset to dataset.
2023-11-30Bug: Fix insertion errorFrederick Muriuki Muriithi
Fix a bug that was causing the data insertion to fail.
2023-11-30Fix errors caught by pylint and mypy.Frederick Muriuki Muriithi
2023-11-15Log out DB errors even when handledFrederick Muriuki Muriithi
Log out any DB errors that occur to help with tracking and debugging failures.
2023-11-15Remove debugging statement.Frederick Muriuki Muriithi
2023-11-15Bug: Insert ProbeSets before average data and cross-refsFrederick Muriuki Muriithi
Insert the ProbeSets first before inserting the average data and cross-referencing it.
2023-11-15mypy: Fix typing errors.Frederick Muriuki Muriithi
2023-11-15Linting: Fix linting errors.Frederick Muriuki Muriithi
2023-10-30Prevent overwrite. Avoid inserting same probeset multiple times.Frederick Muriuki Muriithi
* Avoid overwriting existing data by adding a random string to the name of the ProbeSet * Read ProbeSet names separately from the means for the strains, to avoid entering the same ProbeSet record over and over with varying names.
2023-10-18Link to `ProbeSet.Id` in XRef TableFrederick Muriuki Muriithi
The "ProbeSetId" value in the file is actually stored in the database in the `ProbeSet.Name` field, whereas the `ProbeSetXRef` table actually uses the `ProbeSet.Id` value to define the relationships. This commit fixes that issue.
2023-10-16Fix bug: Insert ProbeSets if they do not exist.Frederick Muriuki Muriithi
2023-10-16Fix bug: Provide the platform_id to `insert_means`Frederick Muriuki Muriithi
2023-10-16Fix bug: parsing fields from query.Frederick Muriuki Muriithi
2022-08-05Update job status. Display stdout and stderr outputsFrederick Muriuki Muriithi
* Display the status of the job, as it is running * Display STDERR output if an error occurs * Display STDOUT output as job is running and on successful completion of the job
2022-08-04Format queries for output. Get rid of extraneous output.Frederick Muriuki Muriithi
2022-07-19Save standard error data. Fix linting and typing errors.Frederick Muriuki Muriithi
2022-07-19Implement data insertion (averages/means)Frederick Muriuki Muriithi
* quality_control/file_utils.py (new file): extract common file utilities. * quality_control/parsing.py (refactor): extract common file utilities. * scripts/insert_data.py: Implement data insertion for averages/means