Message boards :
Code and Servers :
Database stored Input Output files
Message board moderation
Author | Message |
---|---|
Tomáš Brada Project administrator Volunteer developer Send message Joined: 3 Feb 19 Deprecated: Creation of dynamic property BoincUser::$nposts is deprecated in /var/boincadm/prj/html/inc/forum.inc on line 613 Posts: 667 Credit: 432,784 RAC: 0 |
The input and output files are very small in this project. Storing them in one big directory (tree) is not very fast. All SPT output files are first uploaded and stored in directory, then the asim picks them up and stores them in database and finally, the cleaner deletes them. What if I modified the file_upload_handler to store the small output files directly in the database? And do this for input files too. Another issue of directory is that sometimes the file is forgotten and then it stays in the directory forever. For larger files, file storage is better, because the page cache and socket/file interaction is more effective. But for files of few kilobytes, db storage is effective. Also DB allows to utilize indices (indexes), whereas directory has the be read from start to end when searching for glob patterns. |
Tomáš Brada Project administrator Volunteer developer Send message Joined: 3 Feb 19 Posts: 667 Credit: 432,784 RAC: 0 |
Coded support for input files stored in database. The fuh fcgi serves them. Not yet used. |
Tomáš Brada Project administrator Volunteer developer Send message Joined: 3 Feb 19 Posts: 667 Credit: 432,784 RAC: 0 |
Yay, it works! It will take some work to migrate over to the new FUH (assimilators and work generators need change as well as database rows need to be moved). So the traditional FUH mode remains in use. |
Tomáš Brada Project administrator Volunteer developer Send message Joined: 3 Feb 19 Posts: 667 Credit: 432,784 RAC: 0 |
Check the state of result before allowing the upload, allow only in progress. Update the result state after upload to allow assimilation. Figure out which field to update. |
Tomáš Brada Project administrator Volunteer developer Send message Joined: 3 Feb 19 Posts: 667 Credit: 432,784 RAC: 0 |
It appears that some change to the transitioner is needed to support what I am trying to do here. Luckily, transitioner is the only of the boinc-standard programs that needs change. |
Tomáš Brada Project administrator Volunteer developer Send message Joined: 3 Feb 19 Posts: 667 Credit: 432,784 RAC: 0 |
Upload of output files to database was verified today. It was possible even without changes to transitioner, but it was not painless. The issues were mainly caused by outdated documentation. Download was not tested yet. |
Tomáš Brada Project administrator Volunteer developer Send message Joined: 3 Feb 19 Posts: 667 Credit: 432,784 RAC: 0 |
Application tot5 and the server side support was fully migrated to DB-stored input and output files. Both download and upload seems to work. |
Tomáš Brada Project administrator Volunteer developer Send message Joined: 3 Feb 19 Posts: 667 Credit: 432,784 RAC: 0 |
Application SPT asimilator was migrated. Will be tested on next batch. |
©2024 Tomáš Brada