[FFe] OperationError when large object greater than 2gb

The patch needs some work to be released. More details in the upstream issue.

Here is the updated pull request for this.

https://github.com/psycopg/psycopg2/pull/260

?Before I let this into Utopic, what testing of the proposed Ubuntu package has been done?

It is currently being used in the MAAS project, and will be released requiring psycopg2 to support this.

That's not a responsive answer to the question.

The question was about the Ubuntu package. Rejecting until we find out if this has been tested properly.

Also, this is a design change, not a bug fix. It needs to go through the feature freeze exception process. See https://wiki.ubuntu.com/FreezeExceptionProcess

The ubuntu package in -proposed had been tested alongside a large MAAS Cluster in real hardware, and verified that there are not regressions in what we are using it for.

Why would this be a design change ?

Because it uses a new libpq api. Specifically the new api is only available from libpq 9.3: If the package is built against libpq 9.3 and is used on a client that has an older libpq version it will fail import. More details at <https://github.com/psycopg/psycopg2/pull/259#issuecomment-55209844>. Probably your "large MAAS cluster" has verified it with its own setup but not with older libpq versions.

From the psycopg PoV the LO64 support is a new feature that will only be introduced in psycopg 2.6. IMO MAAS should install psycopg from pip or from upstream instead of using the Ubuntu one, which is painfully out of date.

On Monday, September 29, 2014 14:43:44 you wrote:
> Because it uses a new libpq api. Specifically the new api is only
> available from libpq 9.3: If the package is built against libpq 9.3 and
> is used on a client that has an older libpq version it will fail import.
> More details at
> <https://github.com/psycopg/psycopg2/pull/259#issuecomment-55209844>.
> Probably your "large MAAS cluster" has verified it with its own setup
> but not with older libpq versions.
>
> From the psycopg PoV the LO64 support is a new feature that will only be
> introduced in psycopg 2.6. IMO MAAS should install psycopg from pip or
> from upstream instead of using the Ubuntu one, which is painfully out of
> date.

The one we're talking about is 2.5.3, so it's not too horribly out of date,
but since MAAS has to be updated for older releases which are still on 2.4.5,
that's a relevant point. It doesn't solve MAAS' problem just to patch this in
Utopic. 12.04 only has libpq 9.1, so unless 12.04 is no longer a deployment
target for MAAS, using the LO64 doesn't solve your problem anyway.

This does solve our problem because we are not SRU'ing MAAS, nor this change to 12.04. This only affects where MAAS is hosted. In this case, it only affects MAAS running in Utopic.

If MAAS 1.7 (Utopic's version) is SRU'd into Trusty, then we will continue to see this issue. However, we have tested 2.4.5 in trusty, with this patch and the latest version of MAAS and proven that this works as expected:

In Trusty with psycopg 2.4.5 with the patch:

https://launchpad.net/~maas-maintainers/+archive/ubuntu/experimental?field.series_filter=trusty

In Utopic with psycopg 2.5.3 with the patch:

https://launchpad.net/~maas-maintainers/+archive/ubuntu/experimental?field.series_filter=utopic

Right, except this patch is totally unsuitable for an SRU.

Which is something will be discussed when / if we SRU MAAS intro Trusty. At this point, I'm interested on getting this released in Utopic.

Does this require an FFe?

On Tuesday, September 30, 2014 22:12:08 you wrote:
> Which is something will be discussed when / if we SRU MAAS intro Trusty.
> At this point, I'm interested on getting this released in Utopic.
>
> Does this require an FFe?

Yes.

> Attached is the upstream issue and pull request.
>
> https://github.com/psycopg/psycopg2/pull/259

Commentary from upstream in the pull request:

> I have fixed several issues in the lo64 branch. Haven't run the entire test
> grid yet (all supported Py and PG versions, 32 and 64 bits) but modulo
> small tweaks the branch should be ready for integration. However there is
> no chance it can be released in 2.4 or in 2.5 either. What added here is
> the support for an entirely new API, not a bugfix, and its integration has
> added new modes of failure.

Given upstream was unwilling to release this change as part of 2.5, I don't
think we should override them. pyscopg2 is widely used outside of MAAS with a
diverse set of workloads that I don't think we can test.

Not marking the bug invalid, since it's an actual limitation of the current pyscopg2 we have, but FFe not approved.

Fixed in 2.6-1 in Vivid.