Firmware Test Suite

Regression: FTWS version V20.09.00 prints incorrect JSON files

Bug #1902249 reported by Klaus Heinrich Kiwi on 2020-10-30

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Firmware Test Suite	Fix Released	High	Colin Ian King

Bug Description

FWTS as of V20.09.00 incorrectly places a trailing comma at the end of collections:
./fwts -q -r stdout --log-type=json

{
  "fwts":[
    {
      "heading":[
        {
          "line_num":0,
          "date":"30/10/20",
          "time":"08:51:50",
          "field_type":"Info",
          "level":"None",
          "status":"None",
          "failure_label":"None",
          "log_text":"Results generated by fwts: Version V20.09.00 (2020-09-17 18:08:13).",
        },
        {
          "line_num":1,
          "date":"30/10/20",
          "time":"08:51:50",
          "field_type":"Info",
          "level":"None",
          "status":"None",
          "failure_label":"None",
          "log_text":"Some of this work - Copyright (c) 1999 - 2020, Intel Corp. All rights reserved.",
        },
<...>

The trailing comma in "log_text":"Results generated by fwts: Version V20.09.00 (2020-09-17 18:08:13).",}" is incorrect. It doesn't pass jsonlint and breaks python json parsers.

This was not the case with V20.03.00:
{
  "fwts":[
    {
      "heading":[
        {
          "line_num":0,
          "date":"30\/10\/20",
          "time":"08:59:08",
          "field_type":"Info",
          "level":"None",
          "status":"None",
          "failure_label":"None",
          "log_text":"Results generated by fwts: Version V20.03.00 (2020-03-23 18:42:42)."
        },
        {
          "line_num":1,
          "date":"30\/10\/20",
          "time":"08:59:08",
          "field_type":"Info",
          "level":"None",
          "status":"None",
          "failure_label":"None",
          "log_text":"Some of this work - Copyright (c) 1999 - 2020, Intel Corp. All rights reserved."
        },

Revision history for this message

Colin Ian King (colin-king) wrote on 2020-10-30:

This regression occurred with the json back-end changes. I'll sort this out for the next release.

Changed in fwts:
status:	New → In Progress
importance:	Undecided → High
assignee:	nobody → Colin Ian King (colin-king)

Revision history for this message

Colin Ian King (colin-king) wrote on 2020-10-30:

Fix submitted: https://lists.ubuntu.com/archives/fwts-devel/2020-October/012307.html

Revision history for this message

Colin Ian King (colin-king) wrote on 2020-11-04:

Commit applied: https://kernel.ubuntu.com/git/hwe/fwts.git/commit/?id=b241bd8fc5d1d0906a43649559901a35f2a8b0d6

Changed in fwts:
status:	In Progress → Fix Committed

Revision history for this message

Klaus Heinrich Kiwi (klauskiwi) wrote on 2020-11-20:

I'm afraid we'll have to reopen this, since the master version still seems to be generating invalid json files, albeit for another detail now:

The python 3.6 json parser says this:
"Exit unexpectedly with Exception=Invalid \escape: line 418 column 166 (char 15377)"

the Line in question on the master fwts (line 418 is the second-to-last line below)
{
                          "line_num":25,
                          "date":"20/11/20",
                          "time":"14:59:56",
                          "field_type":"Info",
                          "level":"None",
                          "status":"None",
                          "failure_label":"None",
                          "log_text":"Region name ibm,firmware-code@30000000 start: 0x30000000, len: 0x00500000\
"
                        }

on fwts V19.12.00, the same entry is shown as:
                        {
                          "line_num":25,
                          "date":"20\/11\/20",
                          "time":"15:08:58",
                          "field_type":"Info",
                          "level":"None",
                          "status":"None",
                          "failure_label":"None",
                          "log_text":"Region name ibm,firmware-code@30000000 start: 0x30000000, len: 0x00500000\n"
                        },

So looks like we need to escape the carriage-returns for '\n', not simply escape them with '\'.

Revision history for this message

Colin Ian King (colin-king) wrote on 2020-11-20: Re: [Bug 1902249] Re: Regression: FTWS version V20.09.00 prints incorrect JSON files

On 20/11/2020 21:18, Klaus Heinrich Kiwi wrote:
> I'm afraid we'll have to reopen this, since the master version still
> seems to be generating invalid json files, albeit for another detail
> now:
>
> The python 3.6 json parser says this:
> "Exit unexpectedly with Exception=Invalid \escape: line 418 column 166 (char 15377)"
>
> the Line in question on the master fwts (line 418 is the second-to-last line below)
> {
> "line_num":25,
> "date":"20/11/20",
> "time":"14:59:56",
> "field_type":"Info",
> "level":"None",
> "status":"None",
> "failure_label":"None",
> "log_text":"Region name ibm,firmware-code@30000000 start: 0x30000000, len: 0x00500000\
> "
> }
>
> on fwts V19.12.00, the same entry is shown as:
> {
> "line_num":25,
> "date":"20\/11\/20",
> "time":"15:08:58",
> "field_type":"Info",
> "level":"None",
> "status":"None",
> "failure_label":"None",
> "log_text":"Region name ibm,firmware-code@30000000 start: 0x30000000, len: 0x00500000\n"
> },
>
> So looks like we need to escape the carriage-returns for '\n', not
> simply escape them with '\'.
>
My bad, I'll fix that up before next week's release