Use gzip python library instead of sys call for compressing buildlogs

Bug #3111 reported by Celso Providelo
6
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Low
Celso Providelo

Bug Description

The original idea was to used the gzip python library to compress buildlog before storing in librarian, but it present a error decribed below:
"""
python.gzip presented weird errors at this point, most
related to incomplete file storage, the compressed file
was prematurely finished in a 0x00. Using system call for while
"""
this error needs to be further investigated to repair the current implementation, even if it works properly, the style is bad.

Revision history for this message
Celso Providelo (cprov) wrote :

it's a simple example I wrote to use gzip and it works fine.

import gzip
import tempfile
import os

CONTENT = 'Hello World !!!'

def gzipContent(content):
    """Return the filename of gzipped content."""
    out_file_fd, out_file_name = tempfile.mkstemp(suffix=".gz")
    gz_file = gzip.GzipFile(out_file_name, mode='wb')
    gz_file.write(content)
    gz_file.close()

    return out_file_name

def fileSize(filename):
    """Return the real file size."""
    _file = open(filename, 'r')
    _file.seek(0,2)
    size = _file.tell()
    _file.close()
    return size

def gunzipFile(filename):
    """Uncompress filename and return its content """
    gz_file = gzip.open(filename)
    raw = gz_file.read()
    gz_file.close()
    return raw

if __name__ == '__main__':

    filename = gzipContent(CONTENT)
    print 'Compressed: %s' % (filename)

    gz_size = fileSize(filename)
    print 'Gzip file size: %d' % gz_size

    content = gunzipFile(filename)
    print 'Raw content: %s' % content

    os.remove(filename)

Changed in launchpad-buildd:
assignee: nobody → malcolmcleaton
Changed in launchpad-buildd:
status: Unconfirmed → In Progress
Revision history for this message
Christian Reis (kiko) wrote :

Just so I know, is there any advantage to using the gzip library instead of os.system("gzip")?

Changed in soyuz:
assignee: malcolmcleaton → nobody
status: In Progress → Confirmed
Revision history for this message
Celso Providelo (cprov) wrote :

Well, it would just make the code more elegant, no other explicit advantage so far.

It is fine by me to keep it as 'wishlist' only, if you want to.

Revision history for this message
Christian Reis (kiko) wrote :

Agreed.

Changed in soyuz:
importance: Medium → Wishlist
Revision history for this message
Celso Providelo (cprov) wrote :

Barry, when reviewing some related code has suggested a short-term solution that might be worth to try:
{{{
    I read the bug and I agree it would be more elegant to use the gzip library,
    but barring that, what happens if the gzip command fails? Would it make more
    sense to use subprocess here to handle potential gzip failures?
}}}

Revision history for this message
Celso Providelo (cprov) wrote :

The barry's suggestion is easy to implement and will leave us in a more comfortable situation.

Changed in soyuz:
importance: Wishlist → Medium
Revision history for this message
Celso Providelo (cprov) wrote :

Optional for 1.1.7, may be left for 1.1.8

Changed in soyuz:
importance: Medium → Wishlist
Revision history for this message
Celso Providelo (cprov) wrote :

definitely 1.1.8

Celso Providelo (cprov)
Changed in soyuz:
assignee: nobody → cprov
Celso Providelo (cprov)
Changed in soyuz:
status: Confirmed → In Progress
Revision history for this message
Celso Providelo (cprov) wrote :

RF 4653

Changed in soyuz:
status: In Progress → Fix Committed
Celso Providelo (cprov)
Changed in soyuz:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.