Kill fti.py

Flagging this WONTFIX, at least for now. We could extract this now, but it would make our sampledata suck much more since we would need to include the tsvector columns rather than just NULL like we do at the moment.

I'm going to put this back to triaged: sample data doesn't need to contain the vectors: we just need to separate out 'bootstrap a db' and 'apply a change' here.

I think we should leave stripping functionality out of fti.py until later, when we know more about the future of text search in Launchpad and how we are going to maintain the existing text search indexes in the database.

On Thu, Nov 24, 2011 at 8:35 PM, Stuart Bishop
<email address hidden> wrote:
> I think we should leave stripping functionality out of fti.py until
> later, when we know more about the future of text search in Launchpad
> and how we are going to maintain the existing text search indexes in the
> database.

So the short term goal here is to continue to reduce fastdowntime
overheads; if thats been done - if we are not calling fti.py during
fastdowntime, then this bug is about 90% done : the remaining bits
would simply be to make sure that its only called by initial setup, no
?

We are not calling fti.py during fastdowntime. It is not called on staging, as staging uses the same full-update.py that production does. It was being called on qastaging which seems to have a more manual rebuild process, but I've fixed that with the losas.

So the remaining places it is being called are dev environments, and possibly dogfood. It doesn't need to be called on dogfood.

On Thu, Nov 24, 2011 at 10:50 PM, Stuart Bishop
<email address hidden> wrote:
> We are not calling fti.py during fastdowntime. It is not called on
> staging, as staging uses the same full-update.py that production does.
> It was being called on qastaging which seems to have a more manual
> rebuild process, but I've fixed that with the losas.
>
> So the remaining places it is being called are dev environments, and
> possibly dogfood. It doesn't need to be called on dogfood.

Whats the use for it in a dev environment?

- It is called when building the sampledata to set all the fti columns to NULL
- It is also called after populating our databases with sampledata to revert this, changing all the NULLs to real data. This shrinks the size of our sample data scripts, and makes things more robust with regards to manual edits of the sample data file.
- It is called to install tsearch2 into empty databases, before installing the 'baseline' schema dump. This also includes the helpers we have written, such as ftq().

I'm advocating leaving things as they are until we know how we are going to be maintaining changes to production. At the moment, to change full text indexes on production we would either roll a custom script to apply (probably as a database patch to run as part of a fastdowntime deploy), or we would integrate fti.py into the fastdowntime deployment script somehow (perhaps modifying it to generate .sql that is then included in the big script executed by update.py and avoiding the overheads that caused us to remove it from fastdowntime deployments earlier).

The fate of this script will be resolved as we organize how to maintain the text indexes (which will be needed for at least the next 12 months) and transition to PostgreSQL 9.1.