excessive CPU usage and loading time seemingly related to lenght of the game

Some really interesting stuff here. First loading performance.

Widelands loads a game first, than saves it and reloads it. There are some reasons for that, on is that starting a new game and loading is the same code path. Another is that we have a fresh load that has no backwards compatibility cruft anymore.

All timings on my system with a fast CPU, a lot of RAM and a SSD, so the times are rather small. On the long game, the initial load takes ~2 seconds, saving ~6, reloading ~1s. The stuff that is slow is between saving and realoading, we delete everything (as we have to), and that takes forever (~90 seconds on my system).

These are the timings:
Running Time Self (ms) Symbol Name
87682.0ms 86.5% 87682,0 Widelands::Warehouse::launch_worker(Widelands::Game&, unsigned char, Widelands::Requirements const&)
87674.0ms 86.4% 0,0 Widelands::Warehouse::cleanup(Widelands::EditorGameBase&)
87674.0ms 86.4% 0,0 Widelands::ObjectManager::cleanup(Widelands::EditorGameBase&)
87674.0ms 86.4% 0,0 Widelands::EditorGameBase::cleanup_for_load()
87674.0ms 86.4% 0,0 Widelands::Game::cleanup_for_load()
87674.0ms 86.4% 0,0 Widelands::ReplayWriter::ReplayWriter(Widelands::Game&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)

So the main culprit here is launch_worker() when Warehouses are destroyed. It creates a ton of new workers that are immediately destroyed again too - so a lot of unnecessary work is done. Warehouse::cleanup() should launch all those workers if the warehouse is destroyed in-game though, so a distinction between those two code paths is all that is needed here.

Second problem was about CPU usage while the game is paused. On my system that is 83% CPU in the long game. I've know about this issue for a long time and actually addressed it in

https://code.launchpad.net/~widelands-dev/widelands/render_queue
and
https://code.launchpad.net/~widelands-dev/widelands/split_overlay_manager/+merge/254496

unfortunately, these changes had other bugs that I was unable to reproduce, so that did not go anywhere. I still want to get at least the renderqueue branch in. It will drop the CPU usage to ~10%

"Widelands loads a game first, than saves it and reloads it. There are some reasons for that, on is that starting a new game and loading is the same code path. Another is that we have a fresh load that has no backwards compatibility cruft anymore."

most times people keep using the same version. wouldn't it be more efficient to have the game check if the saved game is the same version, and if it is skip the saving and reloading part? if the savegame file does not contain information on the version, it could be added.

It would be more efficient, but it would come at the cost of additional code complexity. The more complex code path (the one we current use in the code) would see much less use - bugs in it would be detected later and would be harder to reproduce (since you only have one chance).

And the efficiency you are searching for is not needed - as pointed out, the long load times are due to a not-needed worker launching. We can introduce a much shorter and simpler alternative code path in this inner code which solves the issue and incurs much less of the maintenance issues.

I have pushed a branch which does try to implement the alternative code branch for the loading sequence.

The load performance branch has been merged. Keeping the bug open for the ingame performance topic.

The render_queue branch should make the drawing overhead already a bit better, helping this one too.

Fixed in build19-rc1.