Allow NULLable columns to be converted to NOT NULL

Bug #1336734 reported by Simon J Mudd
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Percona Toolkit moved to https://jira.percona.com/projects/PT
Fix Released
Medium
Frank Cizmich

Bug Description

A recent OSC I did required me to change an existing column from being nullable to being NOT NULL.
The new definition included a ... NOT NULL DEFAULT ... clause which should mean that the OSC should be able to run cleanly yet pt-online-schema-change does not see this and complains.

Jul 02 12:30:44 my-wrapper-script[26818]: Starting run...
pt-online-schema-change --no-version-check --critical-load="Threads_running:296" --max-load "Threads_running:196" --recurse=0 F=...,D=mydb,t=mytable --alter "MODIFY title varchar(120) CHARACTER SET utf8 not null default '' ..." --check-slave-lag h=some-slave,u=someuser,p='somepass' --max-lag=0.1 --chunk-time=0.1 --execute
No slaves found. See --recursion-method if host ... has slaves.
Will check slave lag on:
  some-slave
Operation, tries, wait:
  copy_rows, 10, 0.25
  create_triggers, 10, 1
  drop_triggers, 10, 1
  swap_tables, 10, 1
  update_foreign_keys, 10, 1
Altering `mydb`.`mytable`...
Creating new table...
Created new table mydb._mytable_new OK.
Altering new table...
Altered `mydb`.`_mytable_new` OK.
2014-07-02T12:30:45 Creating triggers...
2014-07-02T12:30:45 Created triggers OK.
2014-07-02T12:30:45 Copying approximately 28552223 rows...
2014-07-02T12:30:45 Dropping triggers...
2014-07-02T12:30:45 Dropped triggers OK.
2014-07-02T12:30:45 Dropping new table...
2014-07-02T12:30:45 Dropped new table OK.
`mydb`.`mytable` was not altered.
2014-07-02T12:30:45 Error copying rows from `mydb`.`mytable` to `mydb`.`_mytable_new`: 2014-07-02T12:30:45 Copying rows caused a MySQL error 1048:
    Level: Warning
     Code: 1048
  Message: Column 'title' cannot be null
    Query: INSERT LOW_PRIORITY IGNORE INTO `mydb`.`_mytable_new` (...

It looks to me as if this should be checked and if the now NOT NULL column was NULLable before the inserted entry should be something like ISNULL(column,<default value of new column>). That would then prevent the error occurring and allow a column to be changed from being NULLable to being NOT NULL (assuming a DEFAULT value is provided).

Please consider adding this extra functionality.

Seen on: percona-toolkit-2.2.8-1.noarch (CentOS)

Revision history for this message
Nilnandan Joshi (nilnandan-joshi) wrote :

Verified.

[root@centos65 mysql]# pt-online-schema-change --alter "MODIFY mobile varchar(10) NOT NULL DEFAULT ''" D=test,t=t1 --execute
No slaves found. See --recursion-method if host centos65 has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
  copy_rows, 10, 0.25
  create_triggers, 10, 1
  drop_triggers, 10, 1
  swap_tables, 10, 1
  update_foreign_keys, 10, 1
Altering `test`.`t1`...
Creating new table...
Created new table test._t1_new OK.
Altering new table...
Altered `test`.`_t1_new` OK.
2014-08-05T13:22:21 Creating triggers...
2014-08-05T13:22:21 Created triggers OK.
2014-08-05T13:22:21 Copying approximately 4 rows...
2014-08-05T13:22:21 Dropping triggers...
2014-08-05T13:22:21 Dropped triggers OK.
2014-08-05T13:22:21 Dropping new table...
2014-08-05T13:22:21 Dropped new table OK.
`test`.`t1` was not altered.
2014-08-05T13:22:21 Error copying rows from `test`.`t1` to `test`.`_t1_new`: 2014-08-05T13:22:21 Copying rows caused a MySQL error 1048:
    Level: Warning
     Code: 1048
  Message: Column 'mobile' cannot be null
    Query: INSERT LOW_PRIORITY IGNORE INTO `test`.`_t1_new` (`a`, `b`, `who`, `rep_count`, `trx_count`, `trx_started`, `mobile`) SELECT `a`, `b`, `who`, `rep_count`, `trx_count`, `trx_started`, `mobile` FROM `test`.`t1` LOCK IN SHARE MODE /*pt-online-schema-change 2515 copy table*/

[root@centos65 mysql]#

Changed in percona-toolkit:
status: New → Confirmed
Revision history for this message
Giuseppe L (g-liguori) wrote :

Hi, i encountered the same problem.
Fixed with this change in this array

  my %ignore_code = (
      # Error: 1592 SQLSTATE: HY000 (ER_BINLOG_UNSAFE_STATEMENT)
      # Message: Statement may not be safe to log in statement format.
      # Ignore this warning because we have purposely set statement-based
      # replication.
      1592 => 1,
      # Error: 1062 SQLSTATE: 23000 ( ER_DUP_ENTRY )
      # Message: Duplicate entry '%ld' for key '%s'
      # MariaDB 5.5.28+ has this as a warning; See https://bugs.launchpad.net/percona-toolkit/+bug/1099836
      1062 => 1,

#added by Giuseppe
      1048 => 1,
   );

because 1048 it's just a warning to ignore.

Revision history for this message
Adam Johnson (adamchainz) wrote :

+1 I've run into this. Giuseppe's patch there does make it go through, at least for the single-column low-risk schema change I had to do.

tags: added: i64194
Revision history for this message
Frank Cizmich (frank-cizmich) wrote :

Giuseppe's patch works like a charm. Although I think we should consider adding this as an option.

e.g:

--ignore-error-codes 1048,...

although that may be a bit obscure.

maybe, since this is the only error that has cropped up as possibly legitimate to ignore:

--allow-null-to-non-null (or something clearer)

Perhaps the tool could suggest using that option if it detects the error or parses the modify clause correctly.

Changed in percona-toolkit:
status: Confirmed → In Progress
importance: Undecided → Medium
assignee: nobody → Frank Cizmich (frank-cizmich)
milestone: none → 2.2.17
Revision history for this message
Frank Cizmich (frank-cizmich) wrote :

... hardcoding it is not of the table though

Revision history for this message
Sveta Smirnova (svetasmirnova) wrote :

I agree it would be good to have this optional. Some of users would like to fix NULL issues manually before altering table.

Changed in percona-toolkit:
status: In Progress → Fix Committed
tags: added: pt-online-schema-change
Revision history for this message
Frank Cizmich (frank-cizmich) wrote :

Current fix is optional "--null-to-non-null" flag
Default: off

Changed in percona-toolkit:
status: Fix Committed → Fix Released
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PT-653

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.