ValueError raised getting length of batch results

I dug into this a bit. I do not believe that this is a bug in lazr.batchnavigator, or at least not in lazr.batchnavigator.z3batching.batch. The error comes when the len *of the underlying result object* (not the batch itself) is less than zero.

The OOPS is for the search page (lp.app.browser.root.LaunchpadSearchView), when rendering the expression view/pages/@@+navigation-links-upper.

"view/pages" in this case is a GoogleBatchNavigator. It's created in this code:

        if len(page_matches) == 0:
            return None
        navigator = GoogleBatchNavigator(
            page_matches, self.request, start=start)

The page_matches in that code is the underlying result object. It is defined in lib/canonical/launchpad/utilities/searchservice.py . The value that is a negative number appears to be "total = int(results.find('M').text)" in that file. I'd like to test if it is less than zero, and, if so, either raise an error there, or create an informational oops and set total to zero.

Isn't the "negative" number simply the None? After all, in python, None < 0.

On Wed, Dec 1, 2010 at 8:35 PM, Jeroen T. Vermeulen
<email address hidden> wrote:
> Isn't the "negative" number simply the None?  After all, in python, None
> < 0.

nit: in Python 2, None is < everything. In Python 3 that expression
will raise a TypeError:

>>> None < 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: NoneType() < int()

@jtv: I don't think so. As noted in the original bug report, we are getting a ValueError. if __len__ returns None, that generates a TypeError, even in 2.6.

$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo(object):
... def __len__(self): return None
...
>>> foo = Foo()
>>> len(foo)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: an integer is required

Fixed in stable r12013 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12013>.

It looks like my diagnosis was right. There are not instances of the original OOPS since the deployment. However, the OOPS report does include this error, raised from code in my branch:

  1 GoogleResponseError: The reported total (-1, from '-1') was less than zero. See bug 683115.
   GET: 1 Robots: 0 Local: 1
      1 https://launchpad.net/+search (RootObject:+search)
       OOPS-1807O1888

We've had four in the last four days since this change was rolled out.

Therefore, to fix the root cause of the error, we would need to look into why the Google XML response was giving us a "total results" count of -1. To paste over the error, in Launchpad code we can convert the -1 (or anything lower than 0) to 0.

Looking into the first approach, the "total" is the "M" value in the Google response (http://code.google.com/apis/searchappliance/documentation/68/xml_reference.html#results_xml_tag_m). This is an approximation. Curtis, who coded our Google integration, reports that "Google does not guarantee the number in the index matches the number in the cache. -1 can happen when they are out of sync". He also confirms that we can treat -1 as 0. I will make this change, removing the custom exception I raised, and then close this bug.

Fixed in stable r12058 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12058>.