python-bson-ext does not encode binary in Apache with mod_wsgi

Bug #1237615 reported by Gavin Panella
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Gavin Panella
maas (Ubuntu)
Fix Released
Undecided
Unassigned
pymongo (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

In a component of MAAS we're using python-bson to dump out structures containing binary data. Everything works fine in development, but breaks in production under Apache.

In Python 2.x (we're using 2.7) it's necessary to wrap str/bytes objects in bson.binary.Binary; the encoder then does an isinstance check and encodes these objects as binary data - or "\x05" e_name binary, from the spec. However, under Apache our binary data is being encoded as string data - or "\x02" e_name string, from the spec. When decoding this becomes a unicode object, which chokes the next part of the MAAS's processing.

I think this is due to the C extension module. Apache and/or mod_wsgi does some weird things with Python contexts (I don't know much about this, just enough to know it can be a problem), and I suspect the PyObject_IsInstance(value, state->Binary) condition in bson/_cbsonmodule.c is failing because state->Binary is referencing a different Binary type than MAAS's code is.

The following are the mod_wsgi directives we're using in Apache:

{{{
WSGIDaemonProcess maas user=maas group=maas processes=2 threads=1 display-name=%{GROUP}

# Without this, defining a tag as a malformed xpath expression will hang
# the region controller.
# See https://techknowhow.library.emory.edu/blogs/branker/2010/07/30/django-lxml-wsgi-and-python-sub-interpreter-magic
WSGIApplicationGroup %{GLOBAL}

WSGIScriptAlias /MAAS /usr/share/maas/wsgi.py
# Preload application when process starts. This will allow publishing
# the MAAS server existence over Avahi.
WSGIImportScript /usr/share/maas/wsgi.py process-group=maas application-group=maas
WSGIPassAuthorization On

<Directory /usr/share/maas/>
    WSGIProcessGroup maas
</Directory>
}}}

$ apt-cache policy apache2 libapache2-mod-wsgi python-bson python-bson-ext
apache2:
  Installed: 2.4.6-2ubuntu2
  Candidate: 2.4.6-2ubuntu2
  Version table:
 *** 2.4.6-2ubuntu2 0
        500 http://gb.archive.ubuntu.com/ubuntu/ saucy/main amd64 Packages
        100 /var/lib/dpkg/status
libapache2-mod-wsgi:
  Installed: 3.4-4
  Candidate: 3.4-4
  Version table:
 *** 3.4-4 0
        500 http://gb.archive.ubuntu.com/ubuntu/ saucy/main amd64 Packages
        100 /var/lib/dpkg/status
python-bson:
  Installed: 2.6-1
  Candidate: 2.6-1
  Version table:
 *** 2.6-1 0
        500 http://gb.archive.ubuntu.com/ubuntu/ saucy/main amd64 Packages
        100 /var/lib/dpkg/status
python-bson-ext:
  Installed: 2.6-1
  Candidate: 2.6-1
  Version table:
 *** 2.6-1 0
        500 http://gb.archive.ubuntu.com/ubuntu/ saucy/main amd64 Packages
        100 /var/lib/dpkg/status

Gavin Panella (allenap)
Changed in maas:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Julian Edwards (julian-edwards) wrote :

http://emptysqua.re/blog/python-c-extensions-and-mod-wsgi/ has some answers. I'm still not sure how to change our Apache config though as it has most of the hacks in place already.

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Don't know if this matters, but...

The blog article suggests WSGIProcessGroup %{GLOBAL}. We have WSGIApplicationGroup %{GLOBAL}, but WSGIProcessGroup maas.

Revision history for this message
Gavin Panella (allenap) wrote :

I can reproduce this - and demonstrate a fix - on a machine with apache2 and libapache2-mod-wsgi. See the attached archive; expand and run `make demo`.

Revision history for this message
Gavin Panella (allenap) wrote :

To summarise, WSGIImportScript was using application-group=maas. Changing it to application-group=%{GLOBAL} resolved the issue.

Changed in maas:
assignee: nobody → Gavin Panella (allenap)
status: Triaged → Fix Committed
milestone: none → 13.10
Changed in maas:
status: Fix Committed → Fix Released
Changed in maas (Ubuntu):
status: New → Fix Released
James Page (james-page)
Changed in pymongo (Ubuntu):
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.