Add SmartQueue with get() that raises execptions in queue
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
FileStore |
Fix Released
|
High
|
Jason Gerard DeRose |
Bug Description
To keep read IO fully utilized (especially during imports from card readers), we read in a separate thread. This is all fine and good unless an exception occurs in the reader thread... the thread sort of eats the exception, and we don't have a well-defined condition in the main thread. It's difficult to debug and confusing to the programmer.
So instead of queue.Queue, we're going to use a new SmartQueue class something like this:
class SmartQueue(Queue):
def get(self, block=True, timeout=None):
item = super().get(block, timeout)
if isinstance(item, Exception):
raise item
return item
And then you'll update the two worker functions reader() and batch_reader() so they wrap everything in a try/except like this:
def reader(src_fp, queue):
try:
# Do stuff...
except Exception as e:
Which is all pretty easy... but then you also have to write the test for SmartQueue.get(), and update the tests for reader() and batch_reader() to make sure they handle exception scenarios correctly.
Related branches
- dmedia Dev: Pending requested
-
Diff: 329 lines (+159/-37)2 files modifiedfilestore.py (+55/-34)
test_filestore.py (+104/-3)
Changed in filestore: | |
importance: | Undecided → High |
summary: |
- Add SmartQueue with get() raises execptions in queue + Add SmartQueue with get() that raises execptions in queue |
Changed in filestore: | |
status: | New → In Progress |
assignee: | nobody → Jason Gerard DeRose (jderose) |
Changed in filestore: | |
status: | In Progress → Fix Committed |
Changed in filestore: | |
status: | Fix Committed → Fix Released |
Don't think I got time to fix this. Jason can you take it?