scan-for-operator script fails: org.ofono.Error.Failed: Operation failed

@Ian

First, the expected results for this script should be an AccessDenied shown in a backtrace, as we currently only support automatic network registration, and thus scanning is forbidden.

From the error message in your description, it appears ofono can't allocate memory for an instance of an ofono_network_operator struct. As there hasn't been a release of ofono in the past week, I'm suspecting this was caused by some other process. Also, there were quite a few failures reported by QA for build #160, thus it was never promoted.

I have not been able to reproduce the problem with build #161 ( which is the latest to be promoted ), nor has Oliver Grawert.

1. Did you install any custom packages?

2. What were you doing before the problem occurred ( ie. playing videos, some kind of testing, ... )?

3. Can you reliably reproduce the problem on image #161 ( which was promoted ) or greater?

4. If you can't reproduce it on #161 and want dig deeper into the problem on #160, further questions

5. Can you reliably reproduce the problem on #160??

6. Does it go away when you reboot?

Also, if you get it into this condition again, it'd be very helpful to try and grab some memory utilization #s. The "free" command and/or the contents of /proc/meminfo would both be helpful. If you want to get more detailed in order to determine the memory hog, you could also install "smem" and run the command "smem -s uss".

1. Can you easily reproduce this on image #160? What happens if you reboot? Can you still make it happen?

If the answer is yes, can you

!@#$, after all the proof-reading... please disregard the the last two lines ( starting with "1. Can you easily..." ).

I restarted my phone and I can now use the script to get results.

Does this square with the assertion that manual selection is disabled?

The original bug I'm investigating is that the 'manual' screen in system-settings (which uses Ofono's C++ library) doesn't list any networks any more—not even when scan-for-operators is working.

I have been testing this with different SIM cards in mako. scan-for-operators seems to work. Sometimes I get a timeout, but re-trying with a bigger timeout works in the end (although you need to wait for the first request to finish, otherwise the script will return an "operation in progress" error).

Using the GUI (Cellular -> Carrier -> Refresh), I see that a call to 'org.ofono.NetworkRegistration.Scan' is made. A spinning wheel appears on the screen. A reply with the list of networks is received on the ofono side, and the spinning wheel stops, so I assume that the reply to the call is received in System Settings. Also, a call to set selection to manual is made to ofono right in that moment, which I guess is made by system settings. However, no operator is shown on the "Carrier" screen.

There are issues with network selection GUI at the moment: bugs #1273610 and #1274618.

I have not been able to reproduce the original problem. It can be a parsing problem in case a malformed parcel is received in ofono from rild, but we need a way to reproduce it.

I think the original problem was caused by memory starvation, not a parcel parsing problem. Ian mentions that when run manually from the command-line, ofono output the following message when he ran the script:

ofonod[23701]: ril_cops_list_cb: can't allocate ofono_network_operator

I've updated bug #1273610, and am no updating my maguro to the latest image to see if there's any difference in results.

Okay, I'm getting it again.

I'm doing some debugging now. Setting a bp in gdb where the error is generated:

Breakpoint 1, ril_cops_list_cb (message=<optimized out>, user_data=0xf4e98)
    at drivers/rilmodem/network-registration.c:231
231 drivers/rilmodem/network-registration.c: No such file or directory.
(gdb) print ops
$1 = (struct ofono_network_operator *) 0x0
(gdb) print reply
$2 = (struct reply_avail_ops *) 0xf5c10
(gdb) print reply->num_ops
$3 = 0

From the glib documentation for g_try_new0: The function returns NULL when n_structs is 0 of if an overflow occurs.

So the error here is that reply->num_ops is 0.

205 in gril/grilreply.c
(gdb) print num_ops
$11 = 0

At this point I'm in too deep, someone's going to have to help me out. :-)

@Ian

Thanks for debugging this :-)

I was able to reproduce the problem you saw and found the same thing: the number of operators was 0. Which in fact is what is happening: the query for available operators sent to RILd is returning an empty list. That triggered the (misleading) error message about a memory allocation problem. This probably needs to be corrected to make oFono return an empty list instead of an operation failed error.

Interestingly, this does not happen for all SIMs and phones. I have done some research about this, using 2 SIMs, one that is roaming (foreign SIM) and a local SIM. Result are as follows:

maguro, roaming SIM -> scan-for-operators works
maguro, local SIM -> scan-for-operators works
mako, roaming SIM -> scan-for-operators works
mako, local SIM -> scan-for-operators does not work*
OEM phone**, roaming SIM -> scan-for-operators works
OEM phone, local SIM -> scan-for-operators works

* Occasionally I have seen it returning a list of operators.
**[OEM phone brand cannot be revealed yet to the public :-)

Then I switched to Android 4.2.2 for both mako and maguro. I saw that the results were the same, including that mako returns an empty operator list for the local SIM, although sometimes it returns a valid list, exactly as with Ubuntu. This can be either:

1.- The modem decides to return an empty list in case all networks but the one you are currently registered to are forbidden (usually, for local operators you can only register to your home network). Of course, choosing a network can be done only when you have more than one option, so it might be reasonable to return an empty list. But why does it occasionally return a valid list?

2.- A FW bug. It probably would be interesting to see if newer mako Android has this behaviour.

Therefore, I suggest to try with a roaming SIM in mako or test in maguro with any SIM to see if you get the same results as me.

Ah, sorry - I only have mako and one SIM, so I can't test elsewhere. :(

What I don't understand is why this problem is intermittent. Your 1. is never sometimes the case for the same SIM, so I think the question there is important. I think there's a real bug here somewhere, and that bug is that it sometimes/most of the time returns empty erroneously.

Trying on 4.4 now and I haven't got the script to fail yet after a few reboots.

@Iain

Is that 4.4 patches + Ubuntu image? That would suggest that it is a rild or modem FW bug in 4.2.2.

Yeah, I followed this: https://wiki.ubuntu.com/Touch/Install_UT_on_android4.4.2

My findings aren't conclusive; it was always intermittent.

Could you try too?

I also have this behavior with another Android phone (either fails, or an empty list, or the proper list), so it might just be a firmware bug indeed.

Based on the comments above ( thanks everyone for testing ), I'm changing the status of this bug to Invalid, as all opinions seem to point to this being a bug in the modem fw and/or the rild implementation.

If anyone objects to this, please feel free to re-open.