03.03.2020: Somewhere around 20.02.2020 another database crash occurred. This time we did not loose any data (all was backed up). After a disk swap, most of the functionality has been restored. I am still slowly working on restoring the science result database and work generator.

30.01.2020: Unfortunately, the database here suffered a corruption and crashed. Fortunately, the backup was less than 24 hours old, so we did not loose much. I am attempting to recover recent results that are lost in database from uploaded files. While this process is running, scheduler will be disabled and you may see strange numbers in the server status page. After that process is complete, you will have time to upload results in progress. The results that are definitely lost will be resent.
No other results were lost.

I've got the xml files prior to the crash if you'd like to use them to restore credits ??

https://stats.free-dc.org/files/brada.zip

Thank you for keeping us informed.
Good communication goes a long way in the eyes of us crunchers :D

---

Dear Bok. I do not think I need the xml files. Most of the credit has been granted for recovered work. And the rest would be just too much hassle to import.

I just lost 75 SPT completed and uploaded tasks:

17822	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597201352000000000_0 is no longer usable	
17823	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597187597000000000_0 is no longer usable	
17824	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597197422000000000_0 is no longer usable	
17825	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597195457000000000_0 is no longer usable	
17826	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597321217000000000_0 is no longer usable	
17827	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597164017000000000_0 is no longer usable	
17828	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597209212000000000_0 is no longer usable	
17829	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597205282000000000_0 is no longer usable	
17830	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597203317000000000_0 is no longer usable	
17831	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597189562000000000_0 is no longer usable	
17832	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597191527000000000_0 is no longer usable	
17833	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597207247000000000_0 is no longer usable	
17834	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597199387000000000_0 is no longer usable	
17835	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597193492000000000_0 is no longer usable	
17836	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597317287000000000_0 is no longer usable	
17837	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_597319252000000000_0 is no longer usable	
17838	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_596592202000000000_0 is no longer usable	
17839	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_588235057000000000_0 is no longer usable	
17840	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_583566217000000000_0 is no longer usable	
17841	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_584246107000000000_1 is no longer usable	
17842	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_579760012000000000_0 is no longer usable	
17843	T.Brada Experiment Grid	1/30/2020 6:03:50 PM	Result spt_58_580868272000000000_0 is no longer usable	

and

23716	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582624982000000000_0 is no longer usable	
23717	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583927777000000000_0 is no longer usable	
23718	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583923847000000000_0 is no longer usable	
23719	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_587034442000000000_0 is no longer usable	
23720	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583855072000000000_0 is no longer usable	
23721	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583721452000000000_0 is no longer usable	
23722	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583849177000000000_0 is no longer usable	
23723	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583915987000000000_0 is no longer usable	
23724	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583969042000000000_0 is no longer usable	
23725	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583355962000000000_0 is no longer usable	
23726	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583147672000000000_0 is no longer usable	
23727	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583359892000000000_0 is no longer usable	
23728	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582695722000000000_0 is no longer usable	
23729	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583725382000000000_0 is no longer usable	
23730	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582630877000000000_0 is no longer usable	
23731	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583418842000000000_0 is no longer usable	
23732	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583717522000000000_0 is no longer usable	
23733	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583957252000000000_0 is no longer usable	
23734	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582626947000000000_0 is no longer usable	
23735	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583947427000000000_0 is no longer usable	
23736	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582931522000000000_0 is no longer usable	
23737	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583381507000000000_0 is no longer usable	
23738	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582650527000000000_0 is no longer usable	
23739	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583931707000000000_0 is no longer usable	
23740	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582933487000000000_0 is no longer usable	
23741	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583921882000000000_0 is no longer usable	
23742	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582634807000000000_0 is no longer usable	
23743	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583385437000000000_0 is no longer usable	
23744	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583794157000000000_0 is no longer usable	
23745	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583383472000000000_0 is no longer usable	
23746	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583929742000000000_0 is no longer usable	
23747	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_584024062000000000_0 is no longer usable	
23748	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583917952000000000_0 is no longer usable	
23749	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583959217000000000_0 is no longer usable	
23750	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583723417000000000_0 is no longer usable	
23751	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583949392000000000_0 is no longer usable	
23752	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583851142000000000_0 is no longer usable	
23753	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582628912000000000_0 is no longer usable	
23754	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_587680927000000000_0 is no longer usable	
23755	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583845247000000000_0 is no longer usable	
23756	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583976902000000000_0 is no longer usable	
23757	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583971007000000000_0 is no longer usable	
23758	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_584016202000000000_0 is no longer usable	
23759	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583963147000000000_0 is no longer usable	
23760	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582699652000000000_0 is no longer usable	
23761	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583847212000000000_0 is no longer usable	
23762	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583974937000000000_0 is no longer usable	
23763	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583719487000000000_0 is no longer usable	
23764	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583972972000000000_0 is no longer usable	
23765	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583019947000000000_0 is no longer usable	
23766	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_584018167000000000_0 is no longer usable	
23767	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583155532000000000_0 is no longer usable	
23768	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583357927000000000_0 is no longer usable	
23769	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583919917000000000_0 is no longer usable	
23770	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582632842000000000_0 is no longer usable	
23771	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_583955287000000000_0 is no longer usable	
23772	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582927592000000000_0 is no longer usable	
23773	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582579787000000000_0 is no longer usable	
23774	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582308617000000000_0 is no longer usable	
23775	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582161242000000000_0 is no longer usable	
23776	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582878467000000000_0 is no longer usable	
23777	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582666247000000000_0 is no longer usable	
23778	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_581638552000000000_0 is no longer usable	
23779	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_584483872000000000_1 is no longer usable	
23780	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_582169102000000000_0 is no longer usable	
23781	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_580508677000000000_0 is no longer usable	
23782	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_580913467000000000_0 is no longer usable	
23783	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_580361302000000000_0 is no longer usable	
23784	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_585309172000000000_1 is no longer usable	
23785	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_579329677000000000_0 is no longer usable	
23786	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_579514387000000000_0 is no longer usable	
23787	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_579671587000000000_0 is no longer usable	
23788	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_578170327000000000_0 is no longer usable	
23789	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546500422000000000_1 is no longer usable	
23790	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546339292000000000_1 is no longer usable	
23791	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546486667000000000_1 is no longer usable	
23792	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546319642000000000_1 is no longer usable	
23793	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546488632000000000_1 is no longer usable	
23794	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546490597000000000_1 is no longer usable	
23795	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546333397000000000_1 is no longer usable	
23796	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546294097000000000_1 is no longer usable	
23797	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546311782000000000_1 is no longer usable	
23798	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546321607000000000_1 is no longer usable	
23799	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546337327000000000_1 is no longer usable	
23800	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546323572000000000_1 is no longer usable	
23801	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_57_546459157000000000_1 is no longer usable	
23802	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_596234572000000000_2 is no longer usable	
23803	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_598985572000000000_2 is no longer usable	
23804	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_596289592000000000_2 is no longer usable	
23805	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_596232607000000000_2 is no longer usable	
23806	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_599003257000000000_2 is no longer usable	
23807	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_599015047000000000_2 is no longer usable	
23808	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_599011117000000000_2 is no longer usable	
23809	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_596291557000000000_2 is no longer usable	
23810	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_599013082000000000_2 is no longer usable	
23811	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_596269942000000000_2 is no longer usable	
23812	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_599044522000000000_2 is no longer usable	
23813	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_596230642000000000_2 is no longer usable	
23814	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_599052382000000000_2 is no longer usable	
23815	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_596236537000000000_2 is no longer usable	
23816	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_596267977000000000_2 is no longer usable	
23817	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_599048452000000000_2 is no longer usable	
23818	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_598983607000000000_2 is no longer usable	
23819	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_599046487000000000_2 is no longer usable	
23820	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_596266012000000000_2 is no longer usable	
23821	T.Brada Experiment Grid	1/30/2020 6:05:31 PM	Result spt_58_599050417000000000_2 is no longer usable	

I don't have the fastest computers, so that represents over a day's work wasted.

On top of those results that my computers were crunching while your website was unreachable, I have no idea how much, if any, credit I lost from tasks returned before the crash.

Dear Bok. I do not think I need the xml files. Most of the credit has been granted for recovered work. And the rest would be just too much hassle to import.

This user feels quite differently. I'm probably not alone.

---

Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.

i agree with Mike - i've lost 142 units that were in progress during the crash, but uploaded complete, and a similar number that were uploaded prior to the crash which were awarded credit but that has not been recredited. So that doesn't seem like "most of the credit" to me.

This user feels quite differently. I'm probably not alone.

Really ?

Perhaps he could just create another sub-project with a credit total, eg Lost credit, could have a best guess at the value of credit lost at each database crash and add it on. Obviously don't need to add it to the project total credit or export it, it's not like we are keeping track of sub-projects properly.

Dear Bok. I do not think I need the xml files. Most of the credit has been granted for recovered work. And the rest would be just too much hassle to import.

This is the result of the crash...for some devastating

Dear Bok. I do not think I need the xml files. Most of the credit has been granted for recovered work. And the rest would be just too much hassle to import.

This is the result of the crash...for some devastating

Thanks for that information fro Free-DC!

That's only a fraction of the problem, however.

In addition to the loss shown on Free-DC, there's also the tasks processed while the website was down, plus all tasks returned after the last stats dump but before the website died.

All told, there's three time frames for lost tasks:

1) Returned after T.Brada's last database backup and before the last stats dump. This is the number shown on on the Free-DC website. In my case, this is 3139 credits.
2) Returned after the last stats dump, but before the T.Brada website went down. There's no easy way to determine what was lost here, although an examination of some of the logs on in the BOINC client could let you tell approximately how many tasks were returned from that host computer during this time frame.
3) Returned (or, more accurately, attempted to be returned) after the T.Brada website came back up. This is relatively easy to determine from the main client BOINC log as it tries to return the tasks, as in my earlier post. That was about 5100 credits in my case.

So the missing credits for me are:
1) 3139
2) unknown
3) ~5100

In my case, I happen to know they were all SPT tasks, but it would be much more complicated had I been running a mix of tasks.

Multiply this by the number of active users on the system.

---

Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.

This is the result of the crash...for some devastating

I took a couple of screenshots yesterday to preserve the event for a bit longer.

First the major losers, shout out to emoga who only made 4th but if he could add on the million plus points he lost at QuChemPedIA yesterday would be the biggest loser...



At least some did get their credit back, the major winners being...

If you know the extent of the losses, is it not possible to apply a manual correction?

Ouch!

---

#5 on the first list, not on the 2nd list .. I'd rather it was the other way around ..

I'll go with ouch & not happy

All tasks that did *exist* (unsent, received or validated) in the database at the time of last backup, and were *uploaded* at some point, are, or will be validated and credited. Some of that process is still running.
From tasks that were not in database at time of backup were mostly lost. Some were validated tho.

I am sorry for your loss. But accidents happen.
Ps: the database wasn't actually running from sd-card, like I joked on discord.

Ok .. I have to stop this project. My current tasks list says I have 53 pending and none in progress. Yet I am actively working 23 units, have 3 ready to start and 2 ready to report.

Something is still very wrong.

I am sorry for your loss. But accidents happen.

I agree.
lost credits doesn't matter.

When the database crashed in the Stop@home project, nothing was restored.
All results were lost.
The project was stopped.

Tomáš Brada has a backup.
He is working on resuming normal project work.

Please, be patient!

I'm not worried yet, I'll give Thomas a few days to finish sorting things out.

Recovered and granted credit for additional about 600 padls and 1020 spt tasks.

I think that what I done, is the most I can do. The lost or aborted results and credits are lost.
Still, it was less than 22 hours worth of data. What if it was three days? I think my time will be better spent on getting those incremental xtradb backups to work, which would allow faster backup period (perhaps twice a day). And loading more tasks from the new search interval.

If somebody really wants their credits back, go ahead and make a program that turns the xml stat files (diffs!) into sql update statements.

The project will be operating normally from now. Hit the update button in your manager, abort ghost tasks if any.

---