Uploading large video data to swift storage
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Invalid
|
Undecided
|
Unassigned |
Bug Description
Hi,
The code snippet below works for uploading a small video file(15MB) to my swift storage service.
```
url = self.swift_
files = {'file': open(self.
headers = {
'x-auth-token': self.swift_
}
response = requests.
```
The code crashes in my computer(becomes unresponsive) when I am trying to upload a video of say 2+ GB of size.
**_How to perform the following:_**
* Chunk the data to smaller units and upload to the service
* Is it possible to parallel run
**Attempts:**
* Opened the video file and started reading it in chunks
* Created a generator using the yield keyword
* read_as_gen should send me each chunk one by one
~~~~
url = self.swift_
def read_as_
with open(filename, mode='rb') as f:
while True:
chunk = f.read(chunksize)
if len(chunk) > 0:
else:
files = {'file': read_as_
# files = {'file': open(self.
headers = {
}
response = requests.
~~~~
Above attempt did not work for me, only a 70 bytes video file was created in my storage server.
~~~~
hdfs_client.
~~~~
Above functionality can be implemented with the hdfs client.
Ref: https:/
I am basically trying to create the above client using the request module for the swift storage service.
Is there any client library that I can use for my swift storage which allows me to perform multithreading and chunk data uploading to my swift storage.
Most swift clusters have a object size limit for regular uploads around ~2GB range. Like other object storage systems - for larger objects Swift has a multipart upload feature:
http:// docs.openstack. org/developer/ swift/overview_ large_objects. html
Many clients like have large object upload support built in
e.g. https:/ /github. com/openstack/ python- swiftclient