Unleashed queues -- memory usage too high

There's a simple change we can do to save money... be sure that all the ActionQueueCommands define __slots__ (remember that __slots__ is not inherited from parent).

On Sun, 13 Mar 2011 16:24:14 -0000, Facundo Batista <email address hidden> wrote:
> There's a simple change we can do to save money... be sure that all the
> ActionQueueCommands define __slots__ (remember that __slots__ is not
> inherited from parent).

most of them do use __slots__ already...

Attached is a zipped file with two graphs I made as a result of a memory analysis of the Ubuntu One client, queuing almost a quarter million of commands.

The first graph is the client memory shown by top, annotated with what I did in the test:

- throw files inside Ubuntu One (the number of files with the 'f' at the end, note that for every file, two commands are queued).

- disconnect and connect

- u1sdtool --waiting

Now see the graph called sdmem_timed.png

Some conclusions:

- We don't have a measurable memory impact when disconnecting/connecting; I explicitly did that test because we had an issue regarding this in the past, but I made fixes in AQ with this in mind.

- Memory growth seems to be linear.

- The '--waiting' takes a lot of memory. Of course, then releases it, but leaves holes in memory that may be used later (see that when I added the last bunch of files, it didn't start to use more memory at that moment), or may not (memory fragmentation). For an alternative to this, check bug #754050.

The second graph is much simpler, just the quantity of queued operations, and the memory the client used (see sdmem_growths.png).

Yes, the memory usage is pretty much linear.

The "memory per operation" (including all data structures to make it happen) varies heavily, I guess because of --waiting memory usage, but it is around 5KB.

Despite all said, I'll do the minor memory optimization in AQ that I said above: assure that *all* commands have __slots__ declared.

Nice Work Facundo.

This bug was fixed in the package ubuntuone-client - 1.6.0-0ubuntu1

---------------
ubuntuone-client (1.6.0-0ubuntu1) natty; urgency=low

  * New upstream release.
    - Avoid conflict when same file already exists locally (LP: #711389)
    - Must move fils to trash on deletion (LP: #690673)
    - Support disconnection during initial connect attempt (LP: #711211)
    - Log version number on startup (LP: #715455)
    - Memory usage too high (LP: #721483)
    - Unsubscribed folders show as synced, if they exist (LP: #741835)
    - gnome-settings-daemon crash in g_str_hash (LP: #744383)
    - Should not re-queue uploads directly (LP: #744608)
    - No feedback when creating folder from symlink fails (LP: #747299)
    - [FFE] Use API to set urgency from background process (LP: #747677)
    - Open control panel to volumes when quota exceeded (LP: #702176)
    - Set the launcher to urgent when quota exceeded (LP: #702183)
    - nautilus crash in g_str_hash (LP: #724882)
    - Disable/enable file sync is buggy (LP: #744980)

  [Chris Coulson]
  * gnome-settings-daemon crash in g_return_fail_if_warning (LP: #744980)
 -- Rodney Dawes <email address hidden> Sun, 10 Apr 2011 21:24:30 -0400