- Assigned to you:
- No blueprints or bugs assigned to you.
- 3 Jason Gerard DeRose
- No blueprints are targeted to this milestone.
- 3 Fix Released
Download files for this release
Added a small C extension with a few functions back-ported from Python 3.3, plus binding to the Linux fallocate() system call. This extension allowed us to:
* Remove use of /usr/bin/fallocate check_call(), replaced instead with above fallocate() binding
* Improve read performance by using posix_fadvise()
* Reduce the queue depth (without really sacrificing throughput), so batch_iter() now has a max of 64 MiB in the queue, down from 128 MiB
On my system, this resulted in a 17% read performance improvement and I'm now getting 31.2 MB/s from a USB2 card reader (actually, it's a USB3 card reader, but plugged into USB2 port). This test was done with a 60 MB/s SanDisk Extreme card, so the limiting factor was definitely USB2.
This is probably about as fast as we're going to get in a read-IO limited situation, but we haven't really tuned things much yet when write-IO limited.
This release does not have a changelog.
0 blueprints and 3 bugs targeted
|824891||#824891||Replace hacky /usr/bin/fallocate subprocess call||3 High||Jason Gerard DeRose||10 Fix Released|
|847048||#847048||fallocate method in filestore assumes the executable is present in /usr/bin/fallocate||3 High||Jason Gerard DeRose||10 Fix Released|
|898957||#898957||Use Python 3.3 os.posix_fadvise()||3 High||Jason Gerard DeRose||10 Fix Released|