I think we will need to design a filter on the server side that, based on the run time estimates, prevents such small WUs from being sent out to clients in the future. Although we have more than 200,000 WUs left, little is computed due to the extremely heavy MySQL load that is caused by the massive connections. So, another lesson learned, I would say. These small WUs in fact can easily run on some of the (mostly idle) server CPU cores and should also finish within a few days.
Michael.

---

Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.

Have you considered a new type of workunit that is essentially a script to run several (perhaps 10) of what is now the workunit size, one after the other, with code to write some kind of checkpoints in between?

Have you considered a new type of workunit that is essentially a script to run several (perhaps 10) of what is now the workunit size, one after the other, with code to write some kind of checkpoints in between?

I thought about it. Checkpointing is not required given these short run times, though. ;-)
Michael.

---

Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.

Would the checkpoint files need to contain much more than the number of sections that have already been completed and therefore do not need to be repeated? That would allow you to build much longer workunits without doing much more than building a script to run the same program over and over, with a different set of input files each time. Also, if you select which old-style workunits to combine into one new-style workunit primarily by picking those which share the same largest input files, you could reduce the communications load on your server - assuming that there is enough sharing.

Also, I've seen a number of BOINC projects that reduce the communications load by sending most of the workunit files, and getting the output files, in compressed form (ZIP or similar) and having the workunits include steps to decompress the input files and compress the output files.

First of all, our traffic is always in compressed format to reduce bandwidth. Secondly, as said before, we have thought about combinatorial WU archives. It is not advisable because it would require to significantly change the validator. That in turn makes not much sense as we will not send out archives with so many WUs that are this short again. We have learnt a "MySQL lesson" from this and that's what our current testing phase is meant for - although we are processing scientifically meaningful work throughout our "test" runs. ;-)
Michael.

---

Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.

there may be few solutions, do not know if they can be easily implemented in BOINC project, but

use different tables for different WU sizes. You would need to tweak all queries so WU lookup will be done depending on some (precalculated) WU size. With this, you can apply different mysql cache/keys settings to different WU tables

or use BOINC server setting to connect not faster than in X minutes (like in pirates@home - 1 hour, or LHC@home - 15 mins). So clients wont disturb server too often

---

I crunch for Ukraine

We actually know the sources of the issue and have solved it.
Michael.

---

Rechenkraft.net e.V. - Verein zur Foerderung von Bildung, Forschung und Wissenschaft durch Einsatz vernetzter Computer.