[FFE] Please merge couchdb 1.1.0-1 (main) from Debian unstable (main)

As per lp:837190, this (1.1.0, or fixes from 1.0.3) is needed for couchdb to work with the version of erlang in oneiric.

So following chipaca's advice, I've built and installed 1.1.0. I've been beating up on it with the dmedia unit tests. I haven't encountered any failures, but there seems to be a dramatic performance regression in 1.1.0.

For example, from 1.0.1 (plus Ubuntu patches) to this 1.1.0 package:

Microfiber benchmark: 198 => 24.5 ops/second

PyCouchDB benchmark: 55.3 => 18.4 ops/second

== Running against CouchDB 1.0.1 ==

jderose@jgd-test:~/bzr/microfiber/trunk$ python3 benchmark_microfiber.py
*** Benchmarking microfiber ***
Python: 3.2.2rc1, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_microfiber'...
    Seconds: 12.10
    Saves per second: 165.2
  Getting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 6.21
    Gets per second: 321.9
  Deleting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 11.99
    Deletes per second: 166.8
Total seconds: 30.30
Total ops per second: 198.0

jderose@jgd-test:~/bzr/microfiber/trunk$ python benchmark_pycouchdb.py
*** Benchmarking python-couchdb ***
Python: 2.7.2+, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_pycouchdb'...
    Seconds: 83.59
    Saves per second: 23.9
  Getting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 10.99
    Gets per second: 182.1
  Deleting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 13.92
    Deletes per second: 143.7
Total seconds: 108.50
Total ops per second: 55.3

== Running against CouchDB 1.1.0 ==

jderose@jgd-test:~/bzr/microfiber/trunk$ ./benchmark_microfiber.py
*** Benchmarking microfiber ***
Python: 3.2.2rc1, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_microfiber'...
    Seconds: 82.96
    Saves per second: 24.1
  Getting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 80.03
    Gets per second: 25.0
  Deleting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 82.34
    Deletes per second: 24.3
Total seconds: 245.33
Total ops per second: 24.5

jderose@jgd-test:~/bzr/microfiber/trunk$ ./benchmark_pycouchdb.py
*** Benchmarking python-couchdb ***
Python: 2.7.2+, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_pycouchdb'...
    Seconds: 161.63
    Saves per second: 12.4
  Getting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 80.52
    Gets per second: 24.8
  Deleting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 83.73
    Deletes per second: 23.9
Total seconds: 325.88
Total ops per second: 18.4

Okay, if you uncomment line 26 in /etc/couchdb/default.ini:

http://bazaar.launchpad.net/~wibblymat/ubuntu/oneiric/couchdb/update-to-1.1.0/view/head:/etc/couchdb/default.ini.tpl.in#L26

socket_options = [{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]

Then the performance is back up to what is was in 1.0.1. I guess CouchDB was perhaps running with nodelay by default before, and now it isn't? Anyway, gory benchmark details running with that line uncommented:

jderose@jgd-test:~/bzr/microfiber/trunk$ ./benchmark_microfiber.py
*** Benchmarking microfiber ***
Python: 3.2.2rc1, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_microfiber'...
    Seconds: 13.62
    Saves per second: 146.9
  Getting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 6.31
    Gets per second: 316.9
  Deleting 2000 documents from db 'test_benchmark_microfiber'...
    Seconds: 10.27
    Deletes per second: 194.8
Total seconds: 30.20
Total ops per second: 198.7

jderose@jgd-test:~/bzr/microfiber/trunk$ ./benchmark_pycouchdb.py
*** Benchmarking python-couchdb ***
Python: 2.7.2+, x86_64, Linux
  Saving 2000 documents in db 'test_benchmark_pycouchdb'...
    Seconds: 84.01
    Saves per second: 23.8
  Getting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 9.09
    Gets per second: 220.0
  Deleting 2000 documents from db 'test_benchmark_pycouchdb'...
    Seconds: 11.65
    Deletes per second: 171.7
Total seconds: 104.75
Total ops per second: 57.3

To help this get more testing, I went ahead and built it in the Novacut Daily Builds PPA:

https://launchpad.net/~novacut/+archive/daily?field.series_filter=oneiric

https://twitter.com/#!/novacut/status/108805962009296896

I've been beating up on it pretty hard, haven't encountered any problems yet. After further thought, perhaps we don't want to change /etc/couchdb/default.ini, should instead keep it the way upstream ships. But regardless, we should make sure desktopcouch is launching CouchDB with:

[httpd]
socket_options = [{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]

In one of its config files. Perhaps we should add it to the other static file desktopcouch includes:

  /etc/xdg/desktop-couch/compulsory-auth.ini

Okay, think I'm encountering a problem. After running for a while, CouchDB seems to hang when a request is made to a specific database. My experimental dc3 has a monitor that once a minute requests the welcome string like this:

  GET /

Requesting the welcome string continued to work fine. However requests like this would hang:

  GET /dmedia

I restarted dc3 and then everything worked fine again. I think I encountered the same thing testing against desktopcouch, but I didn't narrow things down as nicely. I'll see if I can reliably reproduce this, although it seems to only happen after CouchDB has been running several hours.

I tried the couchdb-bin from the Novacut ppa. It seemed to work fine and solved LP: #780972. After trying it and then going back to the current Oneiric version I found that I soon couldn't access my database (lernid preferences). I had to delete the database and recreate it to go on==not a big deal for Lernid, but something to be aware of in terms of migration planning if it is the general situation, particularly for contacts and bookmarks.

Tracking in oneiric, as this currently blocks bug 743096

I have filed bug #844995 requesting the demotion of couchdb and its dependents to universe from main.

Since it's late in the cycle, I've proposed to merge a cherry picked fix for LP: #780972.

As ubuntuone is pretty much the only (important) rdepends of couchdb in oneiric now, it would not be an unnecessarily big risk to update it now. My main concerns here are the performance regression (which seems to be just a question of where to apply the workaround/fix you mentioned, in couchdb or in desktopcouch), and the mentioned crash. If such a crash didn't happen with the current version, that does seem to be a reason for investigating and fixing it first before upgrading?

Is there anything else except for the cherrypicked fix for #780972 which you need for oneiric? If so, I'm fine with going 1.1 if the crash gets fixed, otherwise a cherrypick sounds appropriate at this time.

Martin,

I haven't again encountered the crash I mentioned. It occurred using my experimental desktopcouch-like `dc3`, so there's a good chance I was doing something stupid :P

I feel I've tested CouchDB 1.1.0 pretty comprehensively, aside from one key area: apps that use desktopcouch through libdesktopcouch-glib-1.0-2, libcouchdb-glib-1.0-2 (evolution-couchdb is probably the most widely used consumer). If we could get some testing via evolution-couchdb, and make sure U1 sync works okay still using it, then I personally feel pushing CouchDB 1.1.0 into Oneiric is a reasonable move.

As far as cherry picking, the CouchDB 1.0.3 release notes mention three Erlang R14B02 fixes:

* Fixed compatibility issues with Erlang R14B02
* Fixed error when restarting replications in OTP R14B02
* Fixed OAuth signature computation in OTP R14B02

BTW, the Novacut components all talk to CouchDB directly, so they're kinda weird cases in desktopcouch land... although still good testing.

Thanks Jason for the followup. So I'm fine with a full upgrade if the U1 team will sort out the MIR and performance regression, and I'm also fine with cherrypicking the three patches.

I have been, for the whole Oneiric cycle, testing evo-couchdb with the system-wide CouchDB instance (that is, no auth), and everything worked ok. So I'd be happy to test dc with the new couchdb and fix any issue that comes up. Although I think it should just work if the "unauthenticated" crashes are fixed. But anyway, just let me know where to get an updated couchdb package from, and I can do a round of testing before/after you upload to the archive

On Fri, 09 Sep 2011 08:49:01 -0000, Rodrigo Moya <email address hidden> wrote:
> I have been, for the whole Oneiric cycle, testing evo-couchdb with the
> system-wide CouchDB instance (that is, no auth), and everything worked
> ok. So I'd be happy to test dc with the new couchdb and fix any issue
> that comes up. Although I think it should just work if the
> "unauthenticated" crashes are fixed. But anyway, just let me know where
> to get an updated couchdb package from, and I can do a round of testing
> before/after you upload to the archive

you can build the package off of the merge proposal associated with this
bug. Or, https://launchpad.net/~novacut/+archive/daily :)

I hope this gets fixed quick

I have having this issue due to couchdb accessing google contacts everytime I open thunderbird

I don't know what issue you are having, as you haven't actually described one. But this bug is a sync request for a new version of couchdb, and couchdb doesn't access google contacts itself. If you have an issue, please open a new bug.

well the issue I am having is thunderbird constantly throwing this error...

There was a problem opening the address book "my <email address hidden>" - the message returned was: Cannot open book: Cannot process, book backend is opening.

I subscribed to this bug: https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/825280, which was closed with this message: Marking as a duplicate of bug 817656, as that should fix the current desktopcouch breakage which causes this error

We don't need 1.1.0 for Oneiric, as the cherrypicked patch in couchdb 1.0.1-0ubuntu17 has fixed the desktopcouch issue. But we should get the latest version from debian synced over as soon as the P archive opens up after 11.10 release.

Thanks Rodney, setting this for P then.

I've removed the duplicate from https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/825280. Since this is still a problem in Oneiric based on the above comments have occured after couchdb 1.0.1-0ubuntu17 landed on Sept. 13th, and issues were seen after that (see comment #18)

its still happening with a slightly different message, it says backend is opening now rather than missing, I included further info here: https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/825280/comments/14

Brad (primefalcon), your error seems to have nothing to do with couchdb, but rather is the same issue occuring within the gmail backend.

So how soon can this be merged into Precise? Is there anything I can do to help (as a person without upload rights)?

Someone just needs to do it. No upload rights required (wibblymat doesn't have upload rights either). Just prepare the merge and get it into the sponsorship queue https://wiki.ubuntu.com/SponsorshipProcess

Unsubscribing the release team. This FFe didn't happen.

https://code.launchpad.net/~wibblymat/ubuntu/oneiric/couchdb/update-to-1.1.0/+merge/74236 merges 1.1.0-1 from Debian, but as 1.1.0-2 is in testing now (and seems to resolve a number of regressions), it'd be great if somebody went ahead and updated the merge to 1.1.0-2.

Perhaps this bug should be closed? As there probably wont be as many Ubuntu specific changes going forward, I think a sync from Debian is more appropriate than a merge. Also up for debate is whether CouchDB should move from main to universe. Either way, the sooner this could happen the better. Also, Novacut says "pretty please" :P

I filled a sync-request bug here - https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/817656

And I personally think this bug here should be closed as Wont Fix.

Oops, never mind... didn't realize Debian didn't have the couchdb/couchdb-bin split.

Merge it is, but should be 1.1.1-1 from Debian testing.

Er, had the bug wrong too, my aim is way off tonight - https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/902417

That's the sync request. And to clearly correct the record, please don't sync from Debian... Novacut really needs the couchdb-bin package as we don't want to force our users to be running a system-wide CouchDB for no reason.

Okay, after more thought and research, here is my proposal for what to do about this in Precise, particularly considering that UbuntuOne must drop CouchDB sync:

https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/903098

We're gettting pretty late in Precise, is this going to be deferred for Q?

Kate, I just can't afford any more time on this bug personally, so looks like this wont happen for Precise.

Although this isn't really a regression as the current package in Precise is broken in the same way. The packaging stuff I could figure out, but I don't think I have the expertise needed with Erlang and CouchDB internals to fix the /etc/init.d script.

This bug was fixed in the package couchdb - 1.2.0-2ubuntu1

---------------
couchdb (1.2.0-2ubuntu1) quantal; urgency=low

  * Merge from Debian unstable (LP: #1022515, LP: #817656):
    - fixes ownership of /etc/couchdb. LP: #731272.
  * Remaining changes:
    - Use pkg-config instead of the js-config script shipped by libmozjs
    - debian/rules, debian/control: split couchdb and couchdb-bin
    - debian/postinst: renamed to couchdb-bin.postinst
    - debian/couchdb-bin.postrm: don't try to delete couchdb system
      user/group
  * Dropped changes, superseded upstream:
    - Backport svn r1039345 from 1.0.3
    - debian/patches/couchio-*: patchset from CouchIO to fix U1 replication
      over SSL. It's not clear if these were upstreamed, but most of the
      source files they applied to no longer exist at all upstream, so if
      these are still needed they will need to be redone.
    - debian/patches/fix-help2man.patch: dropped, no longer needed
    - debian/patches/jquery15-fix.patch: Make Futon work with jQuery 1.5
    - debian/patches/moz*.patch: Spidermonkey 1.8.54 patchset
  * Dropped changes, included in Debian:
    - debian/control - bump standards version
    - debian/control - fix lintian error not-binnmuable-all-depends-any
    - Build against a proper libmozjs
    - Update build-depends/depends/build config (we no longer need the
      --with-js build flags)
  * Dropped changes, superseded in Debian:
    - source/format - mark package as 1.0 format
    - change the default permissions on /etc/couchdb to 0775 and files to
      0664

  [ Jason Gerard DeRose ]
  * /etc/couchdb/default.ini and /etc/couchdb/default.d are delivered in the
    `couchdb-bin` package and are owned by root
  * /etc/couchdb/local.ini and /etc/couchdb/local.d are delivered in the
  * The `couchdb-bin` package does not create nor require the "couchdb" user
    (this is now done in the `couchdb` package instead). LP: #1007125.
  * Added a short sleep delay in couchdb.postrm so couchdb is more likely to
    have actually terminated by the time we `deluser couchdb`, which is needed
    for `sudo apt-get purge couchdb` to work when couchdb is running
 -- Steve Langasek <email address hidden> Tue, 31 Jul 2012 02:04:48 +0000

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release