Mahara

Better way to disable elasticsearch triggers during upgrade

Bug #1720912 reported by Robert Lyon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mahara
Fix Released
High
Robert Lyon
Mahara 17.10.0
17.04
Fix Released
High
Unassigned
Mahara 17.04.4
17.10
Fix Released
High
Robert Lyon
Mahara 17.10.0

Bug Description

Currently if we want to upgrade some of our tables like view, artefact, view_atefact they can take up lots of time/memory if elasticsearch is being used as that has triggers that add info to the search_elasticsearch_queue table as well.

And with elasticsearch queue triggers it checks the queue for the item about to be added and only add it if not already there. So if the number of rows to update is 100,000+ each row add/check is slightly slower than the last.

Instead we should just drop the triggers when we do the change and add them back in after and to make this easier we should have a
   drop_elasticsearch_trigger($table);
and
   create_elasticsearch_trigger($table);
pair that we can wrap around specific updates that will do the check to see if elasticsearch is being used and do the correct dropping/adding of triggers

Revision history for this message
Robert Lyon (robertl-9) wrote :

Need to backport this to 17.04 as there is an update on the view table that causes problems in large sites

Revision history for this message
Robert Lyon (robertl-9) wrote :

Actually as certain updates, eg updating block instances effect artefacts, it will be best to drop all triggers and add them in again to avoid having dev guess which to drop/add.

Revision history for this message
Robert Lyon (robertl-9) wrote :

The patch for this is https://reviews.mahara.org/#/c/8084/

Revision history for this message
Mahara Bot (dev-mahara) wrote : A change has been merged

Reviewed: https://reviews.mahara.org/8084
Committed: https://git.mahara.org/mahara/mahara/commit/6e9321c77d7ca6fe47066c6c28e6887aa1363971
Submitter: Robert Lyon (<email address hidden>)
Branch: master

commit 6e9321c77d7ca6fe47066c6c28e6887aa1363971
Author: Robert Lyon <email address hidden>
Date: Tue Oct 3 11:35:10 2017 +1300

Bug 1720912: Allow easier turning off/on of elasticsearch triggers

As part of the db upgrade process. This allows us to turn the triggers
off while updating a certain table and turning them on again afterwards.

behatnotneeded

Change-Id: I5d6f2f7fb7075fca5fe8c811820abe87d92790b3
Signed-off-by: Robert Lyon <email address hidden>

Revision history for this message
Mahara Bot (dev-mahara) wrote : A patch has been submitted for review

Patch for "17.10_STABLE" branch: https://reviews.mahara.org/8134

Revision history for this message
Mahara Bot (dev-mahara) wrote : A change has been merged

Reviewed: https://reviews.mahara.org/8134
Committed: https://git.mahara.org/mahara/mahara/commit/87c2f7ede1fa921f4b7fd282dccdc570f8e30b4c
Submitter: Robert Lyon (<email address hidden>)
Branch: 17.10_STABLE

commit 87c2f7ede1fa921f4b7fd282dccdc570f8e30b4c
Author: Robert Lyon <email address hidden>
Date: Tue Oct 3 11:35:10 2017 +1300

Bug 1720912: Allow easier turning off/on of elasticsearch triggers

As part of the db upgrade process. This allows us to turn the triggers
off while updating a certain table and turning them on again afterwards.

behatnotneeded

Change-Id: I5d6f2f7fb7075fca5fe8c811820abe87d92790b3
Signed-off-by: Robert Lyon <email address hidden>
(cherry picked from commit 6e9321c77d7ca6fe47066c6c28e6887aa1363971)

Revision history for this message
Mahara Bot (dev-mahara) wrote : A patch has been submitted for review

Patch for "17.04_STABLE" branch: https://reviews.mahara.org/8135

Revision history for this message
Mahara Bot (dev-mahara) wrote : A change has been merged

Reviewed: https://reviews.mahara.org/8135
Committed: https://git.mahara.org/mahara/mahara/commit/52eb6f4cd8049be1b49edde8d07acbbb7d90f29c
Submitter: Robert Lyon (<email address hidden>)
Branch: 17.04_STABLE

commit 52eb6f4cd8049be1b49edde8d07acbbb7d90f29c
Author: Robert Lyon <email address hidden>
Date: Tue Oct 3 11:35:10 2017 +1300

Bug 1720912: Allow easier turning off/on of elasticsearch triggers

As part of the db upgrade process. This allows us to turn the triggers
off while updating a certain table and turning them on again afterwards.

behatnotneeded

Change-Id: I5d6f2f7fb7075fca5fe8c811820abe87d92790b3
Signed-off-by: Robert Lyon <email address hidden>
(cherry picked from commit 6e9321c77d7ca6fe47066c6c28e6887aa1363971)
(cherry picked from commit 87c2f7ede1fa921f4b7fd282dccdc570f8e30b4c)

Robert Lyon (robertl-9)
no longer affects: mahara/18.04
Revision history for this message
Mahara Bot (dev-mahara) wrote : A patch has been submitted for review

Patch for "master" branch: https://reviews.mahara.org/8137

Revision history for this message
Mahara Bot (dev-mahara) wrote : A change has been merged

Reviewed: https://reviews.mahara.org/8137
Committed: https://git.mahara.org/mahara/mahara/commit/b737ffe83f4ebf9a108b9aecda2639372b74a8a2
Submitter: Robert Lyon (<email address hidden>)
Branch: master

commit b737ffe83f4ebf9a108b9aecda2639372b74a8a2
Author: Robert Lyon <email address hidden>
Date: Fri Oct 13 10:50:01 2017 +1300

Bug 1720912: Drop triggers correctly with drop_elasticsearch_triggers()

behatnotneeded

Change-Id: I439d1161e2d3ae12920ae24bba8af47c62a7ce07
Signed-off-by: Robert Lyon <email address hidden>

Revision history for this message
Mahara Bot (dev-mahara) wrote : A patch has been submitted for review

Patch for "17.10_STABLE" branch: https://reviews.mahara.org/8140

Revision history for this message
Mahara Bot (dev-mahara) wrote : A change has been merged

Reviewed: https://reviews.mahara.org/8140
Committed: https://git.mahara.org/mahara/mahara/commit/a411332972b55a1e18f3f276d39e92c1a0f60839
Submitter: Robert Lyon (<email address hidden>)
Branch: 17.10_STABLE

commit a411332972b55a1e18f3f276d39e92c1a0f60839
Author: Robert Lyon <email address hidden>
Date: Fri Oct 13 10:50:01 2017 +1300

Bug 1720912: Drop triggers correctly with drop_elasticsearch_triggers()

behatnotneeded

Change-Id: I439d1161e2d3ae12920ae24bba8af47c62a7ce07
Signed-off-by: Robert Lyon <email address hidden>
(cherry picked from commit b737ffe83f4ebf9a108b9aecda2639372b74a8a2)

Revision history for this message
Mahara Bot (dev-mahara) wrote : A patch has been submitted for review

Patch for "17.04_STABLE" branch: https://reviews.mahara.org/8141

Revision history for this message
Mahara Bot (dev-mahara) wrote : A change has been merged

Reviewed: https://reviews.mahara.org/8141
Committed: https://git.mahara.org/mahara/mahara/commit/a5282d746190ee14ade738807fa53b7cf7acaa5b
Submitter: Robert Lyon (<email address hidden>)
Branch: 17.04_STABLE

commit a5282d746190ee14ade738807fa53b7cf7acaa5b
Author: Robert Lyon <email address hidden>
Date: Fri Oct 13 10:50:01 2017 +1300

Bug 1720912: Drop triggers correctly with drop_elasticsearch_triggers()

behatnotneeded

Change-Id: I439d1161e2d3ae12920ae24bba8af47c62a7ce07
Signed-off-by: Robert Lyon <email address hidden>
(cherry picked from commit b737ffe83f4ebf9a108b9aecda2639372b74a8a2)
(cherry picked from commit a411332972b55a1e18f3f276d39e92c1a0f60839)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.