dispatcher sometimes stacks commands

Bug #796618 reported by Paul Larson
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
LAVA Dispatcher
Fix Released
High
Le Chi Thu

Bug Description

Normally, the dispatcher is supposed to wait for the command prompt before issuing a new command to the target. This way we ensure that the previous command has finished before trying to execute the next one. For some reason, however, I occasionally see a situation where it sends a line before the previous one has finished. For example:
http://validation.linaro.org/jenkins/job/Lava%20Daily%20Beaglexm01/imagetype=alip,target=beaglexm01%20omap3/17/

From this, you can see, that during the install of packages prior to installing lava-test, the bzr branch lp:lava-test command is sent. This doesn't end up running though, and the test fails because it sees that lava-test is not installed.

What I suspect is happening, is perhaps we are getting more than one match for the shell prompt at some point. expect can catch something in the buffer, even if it didn't get sent after the previous command. It may be worth considering trashing the buffer before sending a new command, but I'm not sure if this will be sufficient.

Revision history for this message
Paul Larson (pwlars) wrote :

I'd like to try to get this one fixed soon, I have a sneaking suspicion that it can sometimes lead to things failing when they shouldn't.

Changed in lava-dispatcher:
importance: Undecided → High
milestone: none → 2011.08
assignee: nobody → Paul Larson (pwlars)
Revision history for this message
Paul Larson (pwlars) wrote :

I'm not seeing a good way in pexpect to flush the buffer. I did some experimenting with using read_nonblocking() last night to try to clear it out, but it didn't seem to work the way I would have hoped. On the plus side, I did find at least one of the main places this is happening. When lava-test installs, the expect line looks to see that an line from lava-test help is printed, and the shell prompt is not searched for. Because of this, there is already a shell prompt in the buffer after running the next command. So if the next command takes a long time to run, and then it searches the buffer for the shell prompt, it will find it (from the previous command) even though the command is not complete yet, and carry on to the next command.

Changed in lava-dispatcher:
milestone: 2011.08 → 2011.09
status: New → Confirmed
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

So I think we've figured out the cause of the specific cases of this and fixed those. Do we want to develop a more generic solution? I think that more generic solution would be LavaClient.run_shell_command waiting until it sees a shell prompt before returning by default.

Paul Larson (pwlars)
Changed in lava-dispatcher:
assignee: Paul Larson (pwlars) → Le Chi Thu (le-chi-thu)
status: Confirmed → Fix Committed
Fathi Boudra (fboudra)
Changed in lava-dispatcher:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.