Uploading large video data to swift storage

Bug #1664474 reported by dvigneshwer
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Invalid
Undecided
Unassigned

Bug Description

Hi,

The code snippet below works for uploading a small video file(15MB) to my swift storage service.

```
url = self.swift_public_url+'/'+self.swift_container+'/'+self.video_name
files = {'file': open(self.local_video_loc, 'rb')}
headers = {
 'x-auth-token': self.swift_auth_token,
         'content-type': "application/x-www-form-urlencoded",
}
response = requests.request("PUT",url,files=files,headers=headers)
```

The code crashes in my computer(becomes unresponsive) when I am trying to upload a video of say 2+ GB of size.

**_How to perform the following:_**
* Chunk the data to smaller units and upload to the service
* Is it possible to parallel run

**Attempts:**
* Opened the video file and started reading it in chunks
* Created a generator using the yield keyword
* read_as_gen should send me each chunk one by one
~~~~
url = self.swift_public_url+'/'+self.swift_container+'/'+self.video_name

def read_as_gen(filename, chunksize=-1):
   with open(filename, mode='rb') as f:
          while True:
     chunk = f.read(chunksize)
     if len(chunk) > 0:
               yield chunk
      else:
                raise StopIteration

files = {'file': read_as_gen(self.local_video_loc, 64*1024)}

# files = {'file': open(self.local_video_loc, 'rb')}
headers = {
           'x-auth-token': self.swift_auth_token,
           'content-type': "application/x-www-form-urlencoded",
  }
response = requests.request("PUT",url,files=files,headers=headers)
~~~~

Above attempt did not work for me, only a 70 bytes video file was created in my storage server.

~~~~
hdfs_client.upload('/'+self.hfds_video_loc+'/'+self.video_name,self.local_video_loc,overwrite=False,n_threads=1, temp_dir=None, chunk_size=65536, progress=None, cleanup=True)
~~~~
Above functionality can be implemented with the hdfs client.
Ref: https://hdfscli.readthedocs.io/en/latest/api.html#hdfs.client.Client.upload

I am basically trying to create the above client using the request module for the swift storage service.

Is there any client library that I can use for my swift storage which allows me to perform multithreading and chunk data uploading to my swift storage.

Revision history for this message
clayg (clay-gerrard) wrote :

Most swift clusters have a object size limit for regular uploads around ~2GB range. Like other object storage systems - for larger objects Swift has a multipart upload feature:

http://docs.openstack.org/developer/swift/overview_large_objects.html

Many clients like have large object upload support built in

e.g. https://github.com/openstack/python-swiftclient

Changed in swift:
status: New → Invalid
Revision history for this message
dvigneshwer (dvigneshwer) wrote :

Okay, thanks for the links. Will check them out.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.