Database stored Input Output files

Message boards : Code and Servers : Database stored Input Output files
Message board moderation

To post messages, you must log in.

AuthorMessage
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 628
Credit: 445,023
RAC: 258
Message 4004 - Posted: 6 Feb 2020, 9:11:28 UTC

The input and output files are very small in this project. Storing them in one big directory (tree) is not very fast. All SPT output files are first uploaded and stored in directory, then the asim picks them up and stores them in database and finally, the cleaner deletes them. What if I modified the file_upload_handler to store the small output files directly in the database? And do this for input files too.
Another issue of directory is that sometimes the file is forgotten and then it stays in the directory forever.
For larger files, file storage is better, because the page cache and socket/file interaction is more effective. But for files of few kilobytes, db storage is effective. Also DB allows to utilize indices (indexes), whereas directory has the be read from start to end when searching for glob patterns.
ID: 4004 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 628
Credit: 445,023
RAC: 258
Message 4064 - Posted: 22 Mar 2020, 21:20:57 UTC

Coded support for input files stored in database. The fuh fcgi serves them. Not yet used.
ID: 4064 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 628
Credit: 445,023
RAC: 258
Message 4066 - Posted: 23 Mar 2020, 10:36:54 UTC

Yay, it works!
It will take some work to migrate over to the new FUH (assimilators and work generators need change as well as database rows need to be moved). So the traditional FUH mode remains in use.
ID: 4066 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 628
Credit: 445,023
RAC: 258
Message 4067 - Posted: 23 Mar 2020, 11:02:56 UTC

Check the state of result before allowing the upload, allow only in progress. Update the result state after upload to allow assimilation. Figure out which field to update.
ID: 4067 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 628
Credit: 445,023
RAC: 258
Message 4068 - Posted: 24 Mar 2020, 21:09:21 UTC

It appears that some change to the transitioner is needed to support what I am trying to do here. Luckily, transitioner is the only of the boinc-standard programs that needs change.
ID: 4068 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 628
Credit: 445,023
RAC: 258
Message 4069 - Posted: 25 Mar 2020, 20:06:59 UTC

Upload of output files to database was verified today. It was possible even without changes to transitioner, but it was not painless. The issues were mainly caused by outdated documentation. Download was not tested yet.
ID: 4069 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 628
Credit: 445,023
RAC: 258
Message 4070 - Posted: 28 Mar 2020, 10:20:43 UTC

Application tot5 and the server side support was fully migrated to DB-stored input and output files. Both download and upload seems to work.
ID: 4070 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomáš Brada
Project administrator
Volunteer developer
Avatar

Send message
Joined: 3 Feb 19
Posts: 628
Credit: 445,023
RAC: 258
Message 4071 - Posted: 29 Mar 2020, 21:09:38 UTC

Application SPT asimilator was migrated. Will be tested on next batch.
ID: 4071 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Message boards : Code and Servers : Database stored Input Output files

©2021 Tomáš Brada