For-purchase apps not showing up on Oneiric

>> The issue is that we have 3 applications (all PDFs) that show up fine in
>> apps.ubuntu.com.
>>
>> https://apps.ubuntu.com/cat/applications/oneiric/fullcircle-issue-57/
>> https://apps.ubuntu.com/cat/applications/oneiric/fullcircle-it-spec-3/
>> https://apps.ubuntu.com/cat/applications/oneiric/fullcircle-it-issue-56/
>>
>> However all three of these cannot be found in the Software Center client
>> (5.04) in Oneiric.
>>
>> Michael V is not able to reproduce and I believe that Anthony is unable
>> to reproduce the problem either. However the developer, Dave M and I can
>> reproduce it.
> There is a error when the client tries to update the data from the
> software-center agent server. With the help of you and Dave I found
> that the cache of httplib2 contains the follwing in e.g.:
> ~/.cache/software-center/scaclient/software-center.ubuntu.com,api,2.0,applications,en,ubuntu,oneiric,i386,,d8cf7fa67c82edc14789f34f01dad0df
>
> """
> content-location: https://software-center.ubuntu.com/api/2.0/applications/en/ubuntu/oneiric/i386/
> -x-permanent-redirect-url:
> http://software-center.ubuntu.com/api/2.0/applications/en/ubuntu/oneiric/i386/
> """
> This causes the httplib2 code to immediately raise a error.
>
> As a workaround we need to remove the "-x-permanent-redirect-url" from
> the servers response and then figure out if/how to fix httplib2.
- At first I thought it was httplib2 that was storing the https and
  http responses under the same file, but then I noticed that the cache
  filename includes an md5 sum of the complete url, including the
  scheme.
- Confirmed by looking at a dump of davmor's cache, that there were
  *two* files starting with
  software-center.ubuntu.com,api,2.0,applications,en,ubuntu,oneiric,i386
  (jpugh, could you confirm that you have two as well?), one for http
  and one for https, and they're redirecting the browser between each
  other, causing the infinite loop.
- As mentioned in my previous email, requesting available apps via https
  bounces you over to http, but requesting them via http returns the
  actual list of apps, so this didn't add up with the contents of
  davmor's cache.
- With RT #50628 we added a squid cache in front of our 2.0 API. Until
  then, it seems the list of apps was being served over http*s*, and
  requesting over http would bounce you over (exactly opposite to how
  it is now. This was a bug that was fixed during this RT.
- This RT was set up on production on February 2. Checking the dates in
  the responses in davmor's cache, the redirect from http->https is from
  December (before the RT), and the redirect for https->http is from
  February 8 (after the RT), so that makes sense.
- The thing with 301 responses is that they *never* get expired by
  httplib2's cache. They're permanent redirects, so that's technically
  the correct thing to do.

So I don't know how many users will be affected by this, but it seems
like the way to work around this would be to either:
a) Modify httplib2 a bit to expire permanent redirect responses from the
cache every once in a while, or
b) zap the entire cache and start over every once in a while.

Option a) would be tidier I think. As for a short term ...

Status changed to 'Confirmed' because the bug affects multiple users.

This is fixed in precise with the following commits:
...
revno: 2830
committer: Gary Lasker <email address hidden>
branch nick: trunk
timestamp: Thu 2012-03-08 12:35:19 -0500
message:
  * lp:~mvo/software-center/top-rated-refresh:
  * lp:~mvo/software-center/apptreeview-tweaks:
    - fix 'load_range" errors when navigating the installed view, and
      do some nice needed refactoring/cleanup
...
revno: 2597 [merge]
fixes bug(s): https://launchpad.net/bugs/803028 https://launchpad.net/bugs/88669
8
committer: Michael Vogt <email address hidden>
branch nick: software-center
timestamp: Tue 2011-12-13 17:41:50 +0100
message:
  * lp:~gary-lasker/software-center/catalog-published-date-lp803028:
    - add date_published value from the software-center-agent server
      for use with cataloged time so that for-purchase items appear
      correctly and in the proper order in What's New; include unit
      tests for the new functionality (LP: #803028, LP: #886698
...

Hello Michael, or anyone else affected,

Accepted software-center into oneiric-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

I had the caching bug, and opening s-c twice with the fixed version allows
me to see the apps again.

My understanding is that having to open twice is expected.

Thanks,

James

Thanks for the verification James. You are correct the fix as we have it currently will only work on the second open of software-center.

This bug was fixed in the package software-center - 5.0.6

---------------
software-center (5.0.6) oneiric-proposed; urgency=low

  [ Gary Lasker ]
  * lp:~gary-lasker/software-center/catalog-published-date-lp803028i-for-5.0:
    - use date_published value from the software-center-agent server
      so that for-purchase items appear correctly and in the proper order
      in What's New; include corresponding unit test (LP: #803028)

  [ Michael Vogt ]
  * backport fix from lp:~mvo/software-center/fix-refresh-of-whats-new
    to ensure that new apps from e.g. the software-center-agent are
    immediately displayed in the "Whats new" section of the lobby
    (LP: #949725)

software-center (5.0.5) oneiric-proposed; urgency=low

  [ Gary Lasker ]
  * lp:~gary-lasker/software-center/fix-lp913756-for-5.0:
    - do not offer to add an icon to the Unity launcher for packages
      that do not have an Exec entry in their corresponding desktop
      file, e.g. ubuntu-restricted-extras, wine (LP: #913756)
  * lp:~gary-lasker/software-center/staging-certs-2-for-5.0:
    - add SOFTWARE_CENTER_FORCE_DISABLE_CERTS_CHECK to allow QA easier
      testing (LP: #918746)

  [ Michael Vogt ]
  * lp:~mvo/software-center/fix-server-pagination:
    - reset server side review "page" when switching to a different
      app
  * lp:~mvo/software-center/expunge-cache-5.0:
    - fix invalid httplib cache entries (LP: #938736)

  [ Kiwinote ]
  * grab exhibits for the current series only (LP: #899257)
 -- Michael Vogt <email address hidden> Thu, 08 Mar 2012 09:07:20 +0100