Add SmartQueue with get() that raises execptions in queue

Bug #847385 reported by Jason Gerard DeRose
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
FileStore
Fix Released
High
Jason Gerard DeRose

Bug Description

To keep read IO fully utilized (especially during imports from card readers), we read in a separate thread. This is all fine and good unless an exception occurs in the reader thread... the thread sort of eats the exception, and we don't have a well-defined condition in the main thread. It's difficult to debug and confusing to the programmer.

So instead of queue.Queue, we're going to use a new SmartQueue class something like this:

class SmartQueue(Queue):
    def get(self, block=True, timeout=None):
        item = super().get(block, timeout)
        if isinstance(item, Exception):
            raise item
        return item

And then you'll update the two worker functions reader() and batch_reader() so they wrap everything in a try/except like this:

def reader(src_fp, queue):
    try:
        # Do stuff...
    except Exception as e:
        queue.put(e)

Which is all pretty easy... but then you also have to write the test for SmartQueue.get(), and update the tests for reader() and batch_reader() to make sure they handle exception scenarios correctly.

Related branches

Changed in filestore:
importance: Undecided → High
summary: - Add SmartQueue with get() raises execptions in queue
+ Add SmartQueue with get() that raises execptions in queue
Revision history for this message
Manish Sinha (मनीष सिन्हा) (manishsinha) wrote :

Don't think I got time to fix this. Jason can you take it?

Changed in filestore:
assignee: Manish Sinha (मनीष सिन्हा) (manishsinha) → nobody
status: Triaged → New
Changed in filestore:
status: New → In Progress
assignee: nobody → Jason Gerard DeRose (jderose)
Changed in filestore:
status: In Progress → Fix Committed
Changed in filestore:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.