UEFI shell color codes + fastmodels emulated serial console makes it impossible to parse output with pexpect.

We could filter the data from the serial line using this regular expresssion:

'\x1b\[[0-9;]*m'

An example:

$ cat /tmp/remove-ansi-codes.py
import re
import sys

input_file = sys.argv[1]

with open(input_file, 'r') as data:
    for line in data.readlines():
        filtered = re.sub('\x1b\[[0-9;]*m', '', line)
        print(filtered.strip())

Awesome Antonio. Thank you very much.

On 1 August 2013 11:35, Antonio Terceiro <email address hidden> wrote:
> ** Changed in: lava-dispatcher
> Assignee: Georgy Redkozubov (gesha) => Antonio Terceiro (terceiro)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1197582
>
> Title:
> UEFI shell color codes makes it impossible to parse output with
> pexpect.
>
> Status in LAVA Dispatcher:
> Confirmed
>
> Bug description:
> For reference:
>
> http://www.nwdrone.com/scheduler/job/1178/log_file
>
> [a] Boot Manager
> [b] Shell
> [c] Reboot
> [d] Shutdown
> Start:
> Invalid input, please choose a menu option from the list above
> Start: b
> [2J [01;01H [01;01HUEFI Interactive Shell v2.0. UEFI v2.31 (ARM Versatile Express EFI Jun 19 2013 21:21:03, 0x00000000). Revision 1.02
> [1m [33m [40mMapping table [0m [37m [40m
> [1m [33m [40m FS1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40mF6:
> VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)
> [1m [33m [40m FS0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40mHD3b:;BLK2:
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(1,MBR,0x00000000,0x3F,0x19FC0)
> [1m [33m [40m BLK6: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(E7223039-5836-41E1-B542-D7EC736C5E59)
> [1m [33m [40m BLK0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(02118005-9DA7-443A-92D5-781F022AEDBB)
> [1m [33m [40m BLK4: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(1F15DA3C-37FF-4070-B471-BB4AF12A724A)
> [1m [33m [40m BLK5: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(CC2CBF29-1498-4CDD-8171-F8B6B41D0909)
> [1m [33m [40m BLK1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)
> [1m [33m [40m BLK3: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(2,MBR,0x00000000,0x1A000,0x3CE000)
> [22;01HPress [1m [37m [40mESC [0m [37m [40m in 5 seconds to skip [1m [33m [40mstartup.nsh [0m [37m [40m or any other key to continue.
>
> As you can see from the job above, LAVA is unable to parse the output
> due to the presense of color codes used in the UEFI shell. This is a
> major blocker as LAVA cannot be used to test UEFI properly, and should
> be fixed ASAP.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dispatcher/+bug/1197582/+subscriptions

ok the branch I linked does not fix the problem yet, but I have figured out what the problem is.

UEFI (or the fastmodel emulated serial port) is sending one character at a time, so that the regex filter won't be able to actually filter. I will have to implement a state machine to consume those characters until the entire escape sequence is consumed.

for a test case, it's not enough to send one character at a time. You also have to add some delay in the middle (time.sleep(0.1) once every 10 characters seem to do the trick).

The test case:

-----------------8<-----------------8<-----------------8<-----------------8<-----------------8<-----------------
import sys
import time

chars = 0
for char in "\033[23;01H2.0 \033[1m\033[33m\033[40mShell> \033[0m\033[37m\033[40m":
    sys.stdout.write(char)
    sys.stdout.flush()
    chars += 1
    if chars % 10 == 0:
        time.sleep(0.1)
-----------------8<-----------------8<-----------------8<-----------------8<-----------------8<-----------------

UGH... what a mess. Glad you got it pegged Antonio.

On 2 August 2013 13:04, Antonio Terceiro <email address hidden> wrote:
> for a test case, it's not enough to send one character at a time. You
> also have to add some delay in the middle (time.sleep(0.1) once every 10
> characters seem to do the trick).
>
> The test case:
>
> -----------------8<-----------------8<-----------------8<-----------------8<-----------------8<-----------------
> import sys
> import time
>
> chars = 0
> for char in "\033[23;01H2.0 \033[1m\033[33m\033[40mShell> \033[0m\033[37m\033[40m":
> sys.stdout.write(char)
> sys.stdout.flush()
> chars += 1
> if chars % 10 == 0:
> time.sleep(0.1)
> -----------------8<-----------------8<-----------------8<-----------------8<-----------------8<-----------------
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1197582
>
> Title:
> UEFI shell color codes makes it impossible to parse output with
> pexpect.
>
> Status in LAVA Dispatcher:
> Confirmed
>
> Bug description:
> For reference:
>
> http://www.nwdrone.com/scheduler/job/1178/log_file
>
> [a] Boot Manager
> [b] Shell
> [c] Reboot
> [d] Shutdown
> Start:
> Invalid input, please choose a menu option from the list above
> Start: b
> [2J [01;01H [01;01HUEFI Interactive Shell v2.0. UEFI v2.31 (ARM Versatile Express EFI Jun 19 2013 21:21:03, 0x00000000). Revision 1.02
> [1m [33m [40mMapping table [0m [37m [40m
> [1m [33m [40m FS1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40mF6:
> VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)
> [1m [33m [40m FS0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40mHD3b:;BLK2:
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(1,MBR,0x00000000,0x3F,0x19FC0)
> [1m [33m [40m BLK6: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(E7223039-5836-41E1-B542-D7EC736C5E59)
> [1m [33m [40m BLK0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(02118005-9DA7-443A-92D5-781F022AEDBB)
> [1m [33m [40m BLK4: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(1F15DA3C-37FF-4070-B471-BB4AF12A724A)
> [1m [33m [40m BLK5: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(CC2CBF29-1498-4CDD-8171-F8B6B41D0909)
> [1m [33m [40m BLK1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)
> [1m [33m [40m BLK3: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(2,MBR,0x00000000,0x1A000,0x3CE000)
> [22;01HPress [1m [37m [40mESC [0m [37m [40m in 5 seconds to skip [1m [33m [40mstartup.nsh [0m [37m [40m or any other key to continue.
>
> As you can see from the job above, LAVA is unable to parse the output
> due to the presense of color codes used in the UEFI shell. This is a
> major blocker as LAVA cannot be used to test UEFI properly, and should
> be fixed ASAP.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dispa...

I have just tested with both a Versatile Express TC2 and with the fastmodel, and the problem of receiving one character at a time could only be reproduced with the fast model.

It's pretty hard to make this work properly with fastmodels because of this crappy serial behavior.

Is it a hard requirement that UEFI LAVA testing is done on fastmodels, or can we get away with TC2 work?

On 7 August 2013 12:09, Antonio Terceiro <email address hidden> wrote:
> It's pretty hard to make this work properly with fastmodels because of
> this crappy serial behavior.
>
> Is it a hard requirement that UEFI LAVA testing is done on fastmodels,
> or can we get away with TC2 work?

For v7, we can target TC2. However, for v8, RTSM models is all we
have so we need it.

Have you tried the v8 model. I wonder if it has the same problem.

Is there v8 support yet for UEFI? Please let me know what hwpack this is
and I can give it a shot.

On 7 August 2013 09:50, Anmar Oueja <email address hidden> wrote:

> On 7 August 2013 12:09, Antonio Terceiro <email address hidden>
> wrote:
> > It's pretty hard to make this work properly with fastmodels because of
> > this crappy serial behavior.
> >
> > Is it a hard requirement that UEFI LAVA testing is done on fastmodels,
> > or can we get away with TC2 work?
>
> For v7, we can target TC2. However, for v8, RTSM models is all we
> have so we need it.
>
> Have you tried the v8 model. I wonder if it has the same problem.
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Dispatcher.
> https://bugs.launchpad.net/bugs/1197582
>
> Title:
> UEFI shell color codes makes it impossible to parse output with
> pexpect.
>
> Status in LAVA Dispatcher:
> Confirmed
>
> Bug description:
> For reference:
>
> http://www.nwdrone.com/scheduler/job/1178/log_file
>
> [a] Boot Manager
> [b] Shell
> [c] Reboot
> [d] Shutdown
> Start:
> Invalid input, please choose a menu option from the list above
> Start: b
> [2J [01;01H [01;01HUEFI Interactive Shell v2.0. UEFI v2.31 (ARM
> Versatile Express EFI Jun 19 2013 21:21:03, 0x00000000). Revision 1.02
> [1m [33m [40mMapping table [0m [37m [40m
> [1m [33m [40m FS1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40mF6:
> VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)
> [1m [33m [40m FS0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40mHD3b:;BLK2:
>
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(1,MBR,0x00000000,0x3F,0x19FC0)
> [1m [33m [40m BLK6: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(E7223039-5836-41E1-B542-D7EC736C5E59)
> [1m [33m [40m BLK0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(02118005-9DA7-443A-92D5-781F022AEDBB)
> [1m [33m [40m BLK4: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(1F15DA3C-37FF-4070-B471-BB4AF12A724A)
> [1m [33m [40m BLK5: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(CC2CBF29-1498-4CDD-8171-F8B6B41D0909)
> [1m [33m [40m BLK1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)
> [1m [33m [40m BLK3: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
>
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(2,MBR,0x00000000,0x1A000,0x3CE000)
> [22;01HPress [1m [37m [40mESC [0m [37m [40m in 5 seconds to skip [1m
> [33m [40mstartup.nsh [0m [37m [40m or any other key to continue.
>
> As you can see from the job above, LAVA is unable to parse the output
> due to the presense of color codes used in the UEFI shell. This is a
> major blocker as LAVA cannot be used to test UEFI properly, and should
> be fixed ASAP.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dispatcher/+bug/1197582/+subscriptions
>

--
Tyler Baker
Technical Architect, LAVA
Linaro.org | Open source sof...

Incremental pattern matching seemed like a fun distraction while waiting for a benchmark :) (I didn't go as far as this dude: http://jkff.info/articles/ire/)

I came up with the attached, it seems to work...

Wow, that is cool! :-)

Unfortunately the problem is a little harder; you have not only to feed the characters little by little, but also to be able to get the filtered output in parts. In terms of your implementation, we need to be able to flush() before a complete sequence is fed into the remover, and not get a partially-fed sequence sent to the sink.

we there we need separate input and output buffers, instead of modifying the input buffer ...

the current state of my branch passes all the unit tests that I wrote for this, but doesn't work against the real thing. there's still some detail I am missing

I think my code passes the output along as soon as it cannot possibly be a match to the pattern (the pattern could be tightened to more closely match actual ansi escape codes, I don't think you ever have more than 4 or 5 characters between the ESC [ and the letter).

Can I reproduce the problem on my machine somehow?

FTR Michael and I talked about this over IRC, and my usage of Michael's code was
not optimal. Calling flush() everytime would force outputting content
that could still contain the inicial bits of an incomplete escape
sequence. Just not calling flush() after every read does the trick.

I have just linked a branch that uses Michael's code and passes all my
unit tests. Next step is testing againt the fastmodel.

FWIW, I realised my code is a bit buggy with respect to some patterns (basically, I only build an NFA and then try to follow it deterministically, I think it would fail on patterns like "(ab)?ad" or "(ab)|(ad)") but I also think that it's fine for the vt100 escape code matching done here.

No luck . :-(

Tyler mentioned it was possible to build UEFI with the ANSI color codes disabled. Shall we do that?

Thanks Antonio. I think we definitely gave it a shot. I'll run it down
with Anmar/LEG/ARM.

Best regards,
Alan Bennett
On Aug 20, 2013 5:45 PM, "Antonio Terceiro" <email address hidden>
wrote:

> No luck . :-(
>
> Tyler mentioned it was possible to build UEFI with the ANSI color codes
> disabled. Shall we do that?
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Dispatcher.
> https://bugs.launchpad.net/bugs/1197582
>
> Title:
> UEFI shell color codes makes it impossible to parse output with
> pexpect.
>
> Status in LAVA Dispatcher:
> Confirmed
>
> Bug description:
> For reference:
>
> http://www.nwdrone.com/scheduler/job/1178/log_file
>
> [a] Boot Manager
> [b] Shell
> [c] Reboot
> [d] Shutdown
> Start:
> Invalid input, please choose a menu option from the list above
> Start: b
> [2J [01;01H [01;01HUEFI Interactive Shell v2.0. UEFI v2.31 (ARM
> Versatile Express EFI Jun 19 2013 21:21:03, 0x00000000). Revision 1.02
> [1m [33m [40mMapping table [0m [37m [40m
> [1m [33m [40m FS1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40mF6:
> VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)
> [1m [33m [40m FS0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40mHD3b:;BLK2:
>
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(1,MBR,0x00000000,0x3F,0x19FC0)
> [1m [33m [40m BLK6: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(E7223039-5836-41E1-B542-D7EC736C5E59)
> [1m [33m [40m BLK0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(02118005-9DA7-443A-92D5-781F022AEDBB)
> [1m [33m [40m BLK4: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(1F15DA3C-37FF-4070-B471-BB4AF12A724A)
> [1m [33m [40m BLK5: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(CC2CBF29-1498-4CDD-8171-F8B6B41D0909)
> [1m [33m [40m BLK1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)
> [1m [33m [40m BLK3: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
>
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(2,MBR,0x00000000,0x1A000,0x3CE000)
> [22;01HPress [1m [37m [40mESC [0m [37m [40m in 5 seconds to skip [1m
> [33m [40mstartup.nsh [0m [37m [40m or any other key to continue.
>
> As you can see from the job above, LAVA is unable to parse the output
> due to the presense of color codes used in the UEFI shell. This is a
> major blocker as LAVA cannot be used to test UEFI properly, and should
> be fixed ASAP.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dispatcher/+bug/1197582/+subscriptions
>

Looping in Steven to see if there is an option to turn off the colour
codes.

From my Linux powered device.
On Aug 20, 2013 8:01 PM, "Alan Bennett" <email address hidden> wrote:

> Thanks Antonio. I think we definitely gave it a shot. I'll run it down
> with Anmar/LEG/ARM.
>
> Best regards,
> Alan Bennett
> On Aug 20, 2013 5:45 PM, "Antonio Terceiro" <email address hidden>
> wrote:
>
> > No luck . :-(
> >
> > Tyler mentioned it was possible to build UEFI with the ANSI color codes
> > disabled. Shall we do that?
> >
> > --
> > You received this bug notification because you are a member of Linaro
> > Validation Team, which is subscribed to LAVA Dispatcher.
> > https://bugs.launchpad.net/bugs/1197582
> >
> > Title:
> > UEFI shell color codes makes it impossible to parse output with
> > pexpect.
> >
> > Status in LAVA Dispatcher:
> > Confirmed
> >
> > Bug description:
> > For reference:
> >
> > http://www.nwdrone.com/scheduler/job/1178/log_file
> >
> > [a] Boot Manager
> > [b] Shell
> > [c] Reboot
> > [d] Shutdown
> > Start:
> > Invalid input, please choose a menu option from the list above
> > Start: b
> > [2J [01;01H [01;01HUEFI Interactive Shell v2.0. UEFI v2.31 (ARM
> > Versatile Express EFI Jun 19 2013 21:21:03, 0x00000000). Revision 1.02
> > [1m [33m [40mMapping table [0m [37m [40m
> > [1m [33m [40m FS1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> > [40mF6:
> > VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)
> > [1m [33m [40m FS0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> > [40mHD3b:;BLK2:
> >
> >
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(1,MBR,0x00000000,0x3F,0x19FC0)
> > [1m [33m [40m BLK6: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> > [40m
> > VenHw(E7223039-5836-41E1-B542-D7EC736C5E59)
> > [1m [33m [40m BLK0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> > [40m
> > VenHw(02118005-9DA7-443A-92D5-781F022AEDBB)
> > [1m [33m [40m BLK4: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> > [40m
> > VenHw(1F15DA3C-37FF-4070-B471-BB4AF12A724A)
> > [1m [33m [40m BLK5: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> > [40m
> > VenHw(CC2CBF29-1498-4CDD-8171-F8B6B41D0909)
> > [1m [33m [40m BLK1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> > [40m
> > VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)
> > [1m [33m [40m BLK3: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> > [40m
> >
> >
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(2,MBR,0x00000000,0x1A000,0x3CE000)
> > [22;01HPress [1m [37m [40mESC [0m [37m [40m in 5 seconds to skip [1m
> > [33m [40mstartup.nsh [0m [37m [40m or any other key to continue.
> >
> > As you can see from the job above, LAVA is unable to parse the output
> > due to the presense of color codes used in the UEFI shell. This is a
> > major blocker as LAVA cannot be used to test UEFI properly, and should
> > be fixed ASAP.
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/lava-dispatcher/+bug/1197582/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed...

Will review in 8/23 LEG/LAVA sync meeting

Hi Anmar,

                    I can look into this while I manage the RC1.

BR,

Steve

On 20 August 2013 20:28, Anmar Oueja <email address hidden> wrote:

> Looping in Steven to see if there is an option to turn off the colour
> codes.
>
> From my Linux powered device.
> On Aug 20, 2013 8:01 PM, "Alan Bennett" <email address hidden> wrote:
>
>> Thanks Antonio. I think we definitely gave it a shot. I'll run it down
>> with Anmar/LEG/ARM.
>>
>> Best regards,
>> Alan Bennett
>> On Aug 20, 2013 5:45 PM, "Antonio Terceiro" <email address hidden>
>> wrote:
>>
>> > No luck . :-(
>> >
>> > Tyler mentioned it was possible to build UEFI with the ANSI color codes
>> > disabled. Shall we do that?
>> >
>> > --
>> > You received this bug notification because you are a member of Linaro
>> > Validation Team, which is subscribed to LAVA Dispatcher.
>> > https://bugs.launchpad.net/bugs/1197582
>> >
>> > Title:
>> > UEFI shell color codes makes it impossible to parse output with
>> > pexpect.
>> >
>> > Status in LAVA Dispatcher:
>> > Confirmed
>> >
>> > Bug description:
>> > For reference:
>> >
>> > http://www.nwdrone.com/scheduler/job/1178/log_file
>> >
>> > [a] Boot Manager
>> > [b] Shell
>> > [c] Reboot
>> > [d] Shutdown
>> > Start:
>> > Invalid input, please choose a menu option from the list above
>> > Start: b
>> > [2J [01;01H [01;01HUEFI Interactive Shell v2.0. UEFI v2.31 (ARM
>> > Versatile Express EFI Jun 19 2013 21:21:03, 0x00000000). Revision 1.02
>> > [1m [33m [40mMapping table [0m [37m [40m
>> > [1m [33m [40m FS1: [0m [37m [40m [1m [37m [40mAlias(s): [0m
>> [37m
>> > [40mF6:
>> > VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)
>> > [1m [33m [40m FS0: [0m [37m [40m [1m [37m [40mAlias(s): [0m
>> [37m
>> > [40mHD3b:;BLK2:
>> >
>> >
>> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(1,MBR,0x00000000,0x3F,0x19FC0)
>> > [1m [33m [40m BLK6: [0m [37m [40m [1m [37m [40mAlias(s): [0m
>> [37m
>> > [40m
>> > VenHw(E7223039-5836-41E1-B542-D7EC736C5E59)
>> > [1m [33m [40m BLK0: [0m [37m [40m [1m [37m [40mAlias(s): [0m
>> [37m
>> > [40m
>> > VenHw(02118005-9DA7-443A-92D5-781F022AEDBB)
>> > [1m [33m [40m BLK4: [0m [37m [40m [1m [37m [40mAlias(s): [0m
>> [37m
>> > [40m
>> > VenHw(1F15DA3C-37FF-4070-B471-BB4AF12A724A)
>> > [1m [33m [40m BLK5: [0m [37m [40m [1m [37m [40mAlias(s): [0m
>> [37m
>> > [40m
>> > VenHw(CC2CBF29-1498-4CDD-8171-F8B6B41D0909)
>> > [1m [33m [40m BLK1: [0m [37m [40m [1m [37m [40mAlias(s): [0m
>> [37m
>> > [40m
>> > VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)
>> > [1m [33m [40m BLK3: [0m [37m [40m [1m [37m [40mAlias(s): [0m
>> [37m
>> > [40m
>> >
>> >
>> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(2,MBR,0x00000000,0x1A000,0x3CE000)
>> > [22;01HPress [1m [37m [40mESC [0m [37m [40m in 5 seconds to skip
>> [1m
>> > [33m [40mstartup.nsh [0m [37m [40m or any other key to continue.
>> >
>> > As you can see from the job above, LAVA is unable to parse the output
>> > due to the presense of color codes used in the UEFI shell. This is a
>> > major blocker as LAVA ca...

So what is the final decision Alan?

Anmar,
  The EFI shell application when coupled with the fastmodel seems to create
a 'perfect storm' for parsing the characters in LAVA. We are unable to
accurately parse the resulting UEFI shell application when it is ran on the
fastmodel. UEFI + vexpress works.

We have yet to find a solution within LAVA to fix the UEFI shell +
fastmodel situation.

Possible next steps.
  1. Reduce the complexity of the UEFI shell output by disabling color
codes (would need to be tested)
  2. Investigate Fastmodel emulation when running UEFI shell. To root
cause this, we may need to understand why the fastmodel emulators are
causing delays which seem to break escape sequences and make parsing the
UEFI Shell very difficult.
  3. Blind-execute EFI applications
  4. Send more engineers after the parsing
  5. Semi-blind, maybe implement partial match capability?

On 3 September 2013 13:01, Anmar Oueja <email address hidden> wrote:

> So what is the final decision Alan?
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Dispatcher.
> https://bugs.launchpad.net/bugs/1197582
>
> Title:
> UEFI shell color codes makes it impossible to parse output with
> pexpect.
>
> Status in LAVA Dispatcher:
> Confirmed
>
> Bug description:
> For reference:
>
> http://www.nwdrone.com/scheduler/job/1178/log_file
>
> [a] Boot Manager
> [b] Shell
> [c] Reboot
> [d] Shutdown
> Start:
> Invalid input, please choose a menu option from the list above
> Start: b
> [2J [01;01H [01;01HUEFI Interactive Shell v2.0. UEFI v2.31 (ARM
> Versatile Express EFI Jun 19 2013 21:21:03, 0x00000000). Revision 1.02
> [1m [33m [40mMapping table [0m [37m [40m
> [1m [33m [40m FS1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40mF6:
> VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)
> [1m [33m [40m FS0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40mHD3b:;BLK2:
>
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(1,MBR,0x00000000,0x3F,0x19FC0)
> [1m [33m [40m BLK6: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(E7223039-5836-41E1-B542-D7EC736C5E59)
> [1m [33m [40m BLK0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(02118005-9DA7-443A-92D5-781F022AEDBB)
> [1m [33m [40m BLK4: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(1F15DA3C-37FF-4070-B471-BB4AF12A724A)
> [1m [33m [40m BLK5: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(CC2CBF29-1498-4CDD-8171-F8B6B41D0909)
> [1m [33m [40m BLK1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)
> [1m [33m [40m BLK3: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m
> [40m
>
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(2,MBR,0x00000000,0x1A000,0x3CE000)
> [22;01HPress [1m [37m [40mESC [0m [37m [40m in 5 seconds to skip [1m
> [33m [40mstartup.nsh [0m [37m [40m or any other key to continue.
>
> As you can see from the job above, LAVA is unable to parse the output
> due to the presense of color c...

I will discuss point 1 with Ryan and Leif and get back to you. I think we sunk enough time in trying to fix it. We can turn the colours back on when we switch away from models.

Hi Anmar: Any update?

Nothing. I am waiting on Ryan to finish the 2013.09 release before he
can take a look at it. I hope to have an answer next week.

On 27 September 2013 10:06, Dave Pigott <email address hidden> wrote:
> Hi Anmar: Any update?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1197582
>
> Title:
> UEFI shell color codes makes it impossible to parse output with
> pexpect.
>
> Status in LAVA Dispatcher:
> In Progress
>
> Bug description:
> For reference:
>
> http://www.nwdrone.com/scheduler/job/1178/log_file
>
> [a] Boot Manager
> [b] Shell
> [c] Reboot
> [d] Shutdown
> Start:
> Invalid input, please choose a menu option from the list above
> Start: b
> [2J [01;01H [01;01HUEFI Interactive Shell v2.0. UEFI v2.31 (ARM Versatile Express EFI Jun 19 2013 21:21:03, 0x00000000). Revision 1.02
> [1m [33m [40mMapping table [0m [37m [40m
> [1m [33m [40m FS1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40mF6:
> VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)
> [1m [33m [40m FS0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40mHD3b:;BLK2:
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(1,MBR,0x00000000,0x3F,0x19FC0)
> [1m [33m [40m BLK6: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(E7223039-5836-41E1-B542-D7EC736C5E59)
> [1m [33m [40m BLK0: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(02118005-9DA7-443A-92D5-781F022AEDBB)
> [1m [33m [40m BLK4: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(1F15DA3C-37FF-4070-B471-BB4AF12A724A)
> [1m [33m [40m BLK5: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(CC2CBF29-1498-4CDD-8171-F8B6B41D0909)
> [1m [33m [40m BLK1: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)
> [1m [33m [40m BLK3: [0m [37m [40m [1m [37m [40mAlias(s): [0m [37m [40m
> VenHw(09831032-6FA3-4484-AF4F-0A000A8D3A82)/HD(2,MBR,0x00000000,0x1A000,0x3CE000)
> [22;01HPress [1m [37m [40mESC [0m [37m [40m in 5 seconds to skip [1m [33m [40mstartup.nsh [0m [37m [40m or any other key to continue.
>
> As you can see from the job above, LAVA is unable to parse the output
> due to the presense of color codes used in the UEFI shell. This is a
> major blocker as LAVA cannot be used to test UEFI properly, and should
> be fixed ASAP.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dispatcher/+bug/1197582/+subscriptions

I've hacked up the UEFI binary for RTSM A15 to remove the colour information.

There may still be some/many control codes in there, so please try them and see if they fix your problems.

Hi Ryan,

I was able to boot an image with this UEFI binary but it still has color codes in the UEFI Shell part, and still has quite come control characters to send the cursor all over the place. the control characters are mostly fine; the ANSI color escapes are the actual problem.

Hello guys, I think we have beaten this horse enough and we should stop trying to fix this issue. Instead, we should put it on hold until we get real hardware we can test against.