-
Sven Sandberg authored
WL#7592 step 13. GTIDs: Generate Gtid_log_event and Previous_gtids_log_event always. Fix binlogging of strange SQL statements. A few SQL statements have strange semantics that causes trouble for GTIDs. This includes DROP TABLE with multiple tables, CREATE TABLE ... SELECT, DROP DATABASE that fails on rmdir, OPTIMIZE/REPAIR/ANALYZE/CHECKSUM TABLE, CREATE TEMPORARY/DROP TEMPORARY, DROP TEMPORARY generated by client disconnect, and statements/transactions that mix both transactional and non-transactional updates. 1. Background: When DROP TABLE is used with multiple tables, and the tables are of different types (transactional/non-transactional or temporary/non-temporary), tables of the same type get grouped together and each group is logged as a separate statement. For example: DROP TABLE temporary, non_temporary gets logged as DROP TABLE temporary; DROP TABLE non_temporary. When GTID_MODE = ON, each such statement is assigned its own GTID. In order to generate the GTID, mysql_rm_table_no_locks calls mysql_bin_log.commit for each group of tables. Problem: 1. mysql_bin_log.commit is only called when gtid_mode != OFF. When gtid_mode == OFF, all the statements are written to the binary log in one operation. So after WL#7592 there is only one Anonymous_gtids_log_event, instead of one for each DROP TABLE. 2. If GTID_NEXT='ANONYMOUS', Gtid_state::update_on_commit releases anonymous ownership. But the statement should hold anonymous ownership until it completes. Fix: 1. Call mysql_bin_log.commit unconditionally. Note: inside a transaction, if only temporary tables are dropped, we should not call mysql_bin_log.commit, since the transactional context must remain open in this case. 2. Set thd->is_commit_in_middle_of_statement before calling mysql_bin_log.commit. This tells Gtid_state::update_on_rollback to not release anonymous ownership. 2. Background: Prior to this patch, when binlog_format=row, CREATE...SELECT gets written to the binary log as BEGIN CREATE row events COMMIT CREATE...SELECT is not allowed when gtid_mode=on (in fact, not when enforce_gtid_consistency=1). Problem: 1. Although CREATE without SELECT has an implicit commit, it appears in the middle of a transaction on the slave. Thus, after this worklog and prior to this patch, it gets logged as: Anonymous_gtids_log_event BEGIN CREATE row events COMMIT This causes problems on an MTS slave. 2. If GTID_NEXT='ANONYMOUS', Gtid_state::update_on_commit releases anonymous ownership. But the statement should hold anonymous ownership until it completes. Fix: 1. Call mysql_bin_log.commit after writing the CREATE statement. This causes CREATE...SELECT to be logged like: Anonymous_gtids_log_event CREATE Anonymous_gtids_log_event BEGIN row events COMMIT 2. Set thd->is_commit_in_middle_of_statement before calling mysql_bin_log.commit. This tells Gtid_state::update_on_rollback to not release anonymous ownership. A side-effect of this is that a CREATE...SELECT statement that fails in the SELECT part is already logged when the error happens, but the error causes the statement to be rolled back. To make this case work correctly, we log a compensatory DROP statement if the SELECT part fails. 3. Background: If DROP DATABASE fails after dropping some tables (e.g., if there are extra files in the database directory), then it writes a DROP TABLE statement that lists all the tables that it dropped. If there are many tables, this statement gets long. In this case, the server splits the statement into multiple DROP TABLE statements. Problem: 1. If the statement fails when GTID_NEXT='UUID:NUMBER', then there is no way to log this correctly. So we must generate an error, log nothing, and not add GTID to GTID_EXECUTED. 2. If the statement fails and generates multiple transactions when GTID_NEXT='AUTOMATIC' or 'ANONYMOUS', then we must generate multiple transactions, and each should have its own Gtid_log_event or Anonymous_gtid_log_event. 3. If GTID_NEXT='ANONYMOUS', we must hold anonymous ownership until all transactions have been written to the binary log. Fix: 1. Introduce a new error code, ER_CANNOT_LOG_PARTIAL_DROP_DATABASE_WITH_GTID, and generate the error if GTID_NEXT='UUID:NUMBER' and DROP DATABASE fails. 2. Call mysql_bin_log.commit after writing DROP TABLE statements. 3. Set thd->is_commit_in_middle_of_statement before calling mysql_bin_log.commit. This tells Gtid_state::update_on_rollback to not release anonymous ownership. 4. Background: OPTIMIZE/REPAIR/ANALYZE/CHECKSUM TABLE are written to the binary log even if they fail, after having called trans_rollback. Problem: trans_rollback calls gtid_state::update_on_rollback, which normally releases GTID ownership. But we must not release ownership before writing to the binary log. Fix: This was already fixed for the case gtid_mode=on; for that case we set a special flag in the THD object which tells gtid_state::update_on_rollback to not release ownership. Now we need to fix the case gtid_mode=off, so we set the flag in this case too. 5. Background: CREATE TEMPORARY and DROP TEMPORARY behave very strange. If executed outside transactional context, they behave as DDL: they get logged without BEGIN...COMMIT and cannot be rolled back. If executed in transactional context, they behave as non-transactional DML: they get logged inside BEGIN...COMMIT, leave the transactional context open, but cannot be rolled back. Before this patch, CREATE TEMPORARY and DROP TEMPORARY call gtid_end_transaction unconditionally. Problem: gtid_end_transaction ends the transactional context and releases ownership. This was not a problem before WL#7592 since gtid_end_transaction could only be called when gtid_mode=on, and when gtid_mode=on we disallow CREATE TEMPORARY and DROP TEMPORARY inside transactional context. However, after WL#7592, we call gtid_end_transaction also when gtid_mode=off, and gtid_end_transaction releases anonymous ownership. Fix: Do not call gtid_end_transaction for CREATE TEMPORARY and DROP TEMPORARY inside transaction context. 6. Background: When a client that has open temporary tables disconnects, the temporary tables are dropped and DROP TEMPORARY is written to the binary log. Problem: After WL#7592 and before this patch, if a client disconnects when GTID_NEXT='ANONYMOUS', the client would not hold anonymous ownership when writing to the binary log, which would trigger an assertion in write_gtid. There was no problem when GTID_NEXT='UUID:NUMBER', since this case was taken care of already before WL#7592. In this case, we set GTID_NEXT='AUTOMATIC' before dropping any tables. Fix: Set GTID_NEXT='AUTOMATIC' regardless of GTID_MODE. 7. Background: Anything that is written to the binary log is first written to a thread-specific IO_CACHE. There are two such IO_CACHES: the transaction cache and the statement cache. Transactional updates are written to the transaction cache and non-transactional updates are written to the statement cache. (Under certain conditions, such as when a non-transactional update appears after a transactional update within the same transaction and binlog_direct_non_transactional_updates=0, non-transactional updates are written to the transaction cache, but that is not relevant to the current discussion.) The statement cache is flushed when the statement ends and the transaction cache is flushed when the transaction ends. It is possible that both caches are non-empty if a single transaction or a single statement updates both transactional and non-transactional tables. This is not allowed when GTID_MODE=ON. If both caches are non-empty, an autocommitted transaction flushes first the statement cache and then the transaction cache (see binlog.cc:binlog_cache_mngr::flush). Both happen in the BGC flush stage. Problems: 7.1. When flushing both caches in an autocommitted transaction, and GTID_NEXT=AUTOMATIC, Gtid_state::generate_automatic_gtid is called once for each cache. Gtid_state::generate_automatic_gtid acquires anonymous ownership, but it also assumes (and asserts) that no ownership is held when entering the function. Thus, before this patch the assert would be raised when flushing the transaction cache, since the previous flush of the statement cache did acquire ownership. 7.2. When the statement cache is flushed in the middle of a transaction (due to a non-transactional update happening in the middle of the transaction), and GTID_NEXT=ANONYMOUS, before this patch ownership was released. This breaks the protocol that anonymous ownership must be held for the duration of the transaction when GTID_NEXT=ANONYMOUS. 7.3. Any statement calls gtid_reacquire_ownership_if_anonymous before it executes (mysql_execute_statement calls gtid_pre_statement_checks which calls gtid_reacquire_ownership_if_anonymous). This is important because when GTID_NEXT=ANONYMOUS, anonymous ownership must be held from when the transaction begins to execute. Therefore, when the transaction commits it also asserts that if GTID_NEXT=ANONYMOUS, it holds anonymous ownership. For the same reason, Rows_log_event::do_apply_event calls gtid_pre_statement_checks which calls gtid_reacquire_ownership_if_anonymous. However, before this patch the call to gtid_reacquire_ownership_if_anonymous was missing in Xid_log_event::do_apply_event. There is no existing case when GTID_NEXT=ANONYMOUS and anonymous ownership is not held when Xid_log_event::do_apply_event is called. However, it could potentially happen if a future version has a bug that sends a lonely Xid_log_event to the slave, or if the relay log is generated by something else than mysql, which has a bug. So we should call gtid_reacquire_ownership_if_anonymous at the beginning of Xid_log_event::do_apply_event too (just like we would do in a Query_log_event that contains a COMMIT query). Fix: 7.1. When both caches are non-empty, and GTID_NEXT='AUTOMATIC', call thd->clear_owned_gtids between the two cache flushes. This releases anonymous ownership so that the call to Gtid_cache::generate_automatic_gtid for the second flush is done without holding any ownership. 7.2. In Gtid_state::update_gtids_impl, do not release ownership if the transaction cache is nonempty. 7.3. Call gtid_reacquire_ownership_if_anonymous from Xid_log_event::do_apply_event. @mysql-test/extra/binlog_tests/drop_tables_logical_timestamp.inc - Remove this file. See binlog_mts_logical_clock.test for explanation. @mysql-test/extra/binlog_tests/logical_timestamping.inc - Use assert_logical_timestamps.inc instead of grep_pattern.inc See binlog_mts_logical_clock.test for explanation. @mysql-test/extra/rpl_tests/rpl_gtid_drop_multiple_tables.inc - New auxiliary file used by rpl_gtid_drop_multiple_tables_in_multiple_ways.inc. @mysql-test/extra/rpl_tests/ rpl_gtid_drop_multiple_tables_in_multiple_ways.inc - New auxiliary test file used by rpl_split_statements.test. @mysql-test/extra/rpl_tests/rpl_split_statements.test - New test case. @mysql-test/include/assert_binlog_events.inc - New test utility. This is needed by rpl_split_statements.inc. @mysql-test/include/assert_grep.inc - New test utility. This is needed by assert_logical_timestamps.inc. @mysql-test/include/assert_logical_timestamps.inc - New auxiliary test utility used by binlog_mts_logical_clock.test. @mysql-test/include/grep_pattern.inc - Add comment suggesting to use assert_grep.inc instead. @mysql-test/include/gtid_step_assert.inc - Add $gtid_step_gtid_mode_agnostic, needed by rpl_split_statements.test - Update test utility due to changes in gtid_utilities.inc. @mysql-test/include/gtid_utils.inc - Add new utility functions. @mysql-test/include/gtid_utils_end.inc - Drop the new functions introduced in gtid_utilities.inc. @mysql-test/include/rpl_connect.inc - Make it easier to map mysqltest connections to thread numbers in server debug traces. @mysql-test/include/rpl_connection.inc - Add $rpl_connection_silent_if_same. @mysql-test/include/rpl_get_end_of_relay_log.inc - New auxiliary test utility. This is needed by rpl_skip_to_end_of_relay_log.inc. @mysql-test/include/rpl_init.inc - Set some more mtr variables for convenience. @mysql-test/include/rpl_skip_to_end_of_relay_log.inc - New auxiliary test utility. This is needed by rpl_gtid_split_statements.test. @mysql-test/include/rpl_stop_slaves.inc - Correct a typo. @mysql-test/include/save_binlog_position.inc - New auxiliary test script, useful in combination with include/assert_binlog_events.inc. @mysql-test/include/set_gtid_next_gtid_mode_agnostic.inc - New auxiliary test utility. This is needed by rpl_split_statements.test. @mysql-test/include/wait_for_status_var.inc - Fix broken implementation of $status_fail_query. @mysql-test/suite/binlog/r/binlog_gtid_utils.result - Update result file for modified test. @mysql-test/suite/binlog/r/binlog_mts_logical_clock.result - Update result file due to change in test. @mysql-test/suite/binlog/r/binlog_mts_logical_clock_gtid.result - Update result file due to change in test. @mysql-test/suite/binlog/t/binlog_gtid_utils.test - Add tests for new utility function. @mysql-test/suite/binlog/t/binlog_mts_logical_clock.test - This test started failing because DROP TABLE is now logged differently. The failure was in 'grep_pattern.inc' executed just after drop_tables_logical_timestamp.inc. This was fixed by changing the test assertion. - The test assertion following drop_tables_logical_timestamp.inc belongs inside drop_tables_logical_timestamp.inc. Moved the test assertion there. - The tests in drop_tables_logical_timestamp.inc were located in this file because they differed between gtid_mode=on and gtid_mode=off. But because of the change in logging of DROP TABLE, the test works in the same way regardless of gtid_mode. Therefore, the contents of drop_tables_logical_timestamp.inc was moved into logical_timestamping.inc and drop_tables_logical_timestamp.inc as removed. - This file used grep_pattern.inc to check assertions. This made the test very verbose and hard to read and understand. Also, it was imprecise since any future server bug that introduces a logical timestamp that does not match the given pattern would go unnoticed by the test. Therefore, replaced all grep_pattern.inc by assert_logical_timestamps.inc. @mysql-test/suite/binlog/t/binlog_mts_logical_clock_gtid.test - See binlog_mts_logical_clock.test. @mysql-test/suite/rpl/r/rpl_gtid_create_select.result - Result file for new test. @mysql-test/suite/rpl/r/rpl_gtid_disconnect_drop_temporary_table.result - Result file for new test. @mysql-test/suite/rpl/r/rpl_gtid_split_statements.result - Result file for new test. @mysql-test/suite/rpl/r/rpl_no_gtid_split_statements.result - Result file for new test. @mysql-test/suite/rpl/r/rpl_semi_sync.result - When semisync is enabled, there will be one ack for each transaction. Since we now have more 'transactions' due to the extra Anonymous_log_event preceding each DROP statement, this changes the result file. @mysql-test/suite/rpl/t/rpl_drop_db.test - This test can now run regardless of GTID_MODE. @mysql-test/suite/rpl/t/rpl_gtid_create_select.test - New test to verify that CREATE ... SELECT is logged as expected. @mysql-test/suite/rpl/t/rpl_gtid_disconnect_drop_temporary_table.test - New test file to verify that DROP TEMPORARY generated by client disconnect is logged correctly. @mysql-test/suite/rpl/t/rpl_gtid_split_statements.test - New test file. @mysql-test/suite/rpl/t/rpl_invoked_features.test - Explain why not_gtid_enabled is used. @mysql-test/suite/rpl/t/rpl_mixed_drop_create_temp_table.test - Explain why not_gtid_enabled is used. @mysql-test/suite/rpl/t/rpl_no_gtid_split_statements.test - New test file. @mysql-test/suite/rpl/t/rpl_stop_slave.test - Explain why not_gtid_enabled is used. @sql/rpl_gtid.h - Add need_lock parameter to Sid_map::sidno_to_sid, Gtid::to_string, Gtid::dbug_print, Gtid_specification::to_string, and Gtid_specification::dbug_print, to simplify the use of these functions. - Simplify the definition of Gtid_specification::MAX_TEXT_LENGTH. - Make gtid_reacquire_ownership_if_anonymous extern so that it can be called from log_event.cc. @sql/rpl_gtid_execution.cc - Make gtid_reacquire_ownership_if_anonymous extern so that it can be called from log_event.cc. - Move parts of gtid_pre_statement_checks into an own function, gtid_reacquire_ownership_if_anonymous. This is just to make the logic more easy to follow. - Make gtid_reacquire_ownership acquire anonymous ownership in case GTID_NEXT='ANONYMOUS'. This is needed for cases like: SET AUTOCOMMIT=1; SET GTID_NEXT='ANONYMOUS'; INSERT; BEGIN; The first INSERT will commit the transaction and therefore release anonymous ownership. Then the BEGIN has to re-acquire anonymous ownership so that it can execute correctly. (The logic to reacquire ownership in this case is needed because we allow user to set GTID_NEXT='ANONYMOUS' and then execute multiple transactions. This differs from the case of GTID_NEXT='UUID:NUMBER', where we generate an error if user tries to execute a second transaction without setting GTID_NEXT again. The reason we want to generate an error in this case is that otherwise the GTID autoskip feature would silently skip the transaction and user would not notice it.) - Move the call to gtid_reacquire_ownership_if_anonymous earlier in gtid_pre_statemen_checks. This is needed for the statements BEGIN/ROLLBACK/COMMIT in two cases. Case 1: User executes: SET GTID_NEXT='ANONYMOUS'; BEGIN; INSERT; COMMIT; BEGIN; Here the second BEGIN needs to acquire anonymous ownership so that anonymous ownership is held during the entire transaction. Case 2: The relay log comes from an old master that does not generate Anonymous_gtids_log_event, so it begins with: Format_desc Previous_gtids BEGIN; Then the Format_desc will set GTID_NEXT=NOT_YET_DETERMINED, and the BEGIN needs to acquire anonyous ownership. The only case where we should not acquire anonymous ownership is for a SET statement that does not invoke a stored function. This is important in case a client processes output from mysqlbinlog, for a binary log generated with GTID_MODE=ON. Then, the server will first execute a Format_description_log_event, which will set GTID_NEXT='NOT_YET_DETERMINED', followed by a SET GTID_NEXT='UUID:NUMBER' statement. If it would try to reacquire anonymous ownership in this case, and gtid_mode=on, an error would be generated since anonymous transactions are not allowed when gtid_mode=on. @sql/rpl_gtid_misc.cc - Add need_lock parameter to Gtid::to_string, to simplify the use of this function. - Allow Gtid::to_string to take sid_map==NULL in debug mode, so that debug printouts can show the sidno. @sql/rpl_gtid_specification.cc - Add need_lock to Gtid_specification::to_string. @sql/rpl_gtid_state.cc - Print any change of thd->owned_gtid to the debug trace. - Implement thd->is_commit_in_middle_of_statement. - Return early from update_gtids_impl to avoid releasing ownership, if the transaction cache is nonempty and GTID_NEXT=ANONYMOUS. @sql/share/errmsg-utf8.txt - New error message. @sql/binlog.cc - Release anonymous ownership between flushing the statement cache and flushing the transaction cache, if GTID_NEXT=AUTOMATIC. @sql/log_event.cc - Call gtid_reacquire_ownership_if_anonymous in Xid_log_event::do_apply_event. @sql/sql_admin.cc - Administrational statements (OPTIMIZE TABLE, REPAIR TABLE, CHECKSUM TABLE, ANALYZE TABLE) behave strange if they fail: they first call trans_rollback and then write the statement to the binary log. This causes problems for GTIDs, since ha_rollback eventually calls gtid_state::update_on_rollback, which releases GTID ownership. Then no GTID is owned when it comes to writing the statements to the binary log. This was handled when GTID_NEXT='UUID:NUMBER' by setting the flag thd->skip_gtid_rollback before calling ha_rollback. gtid_state::update_on_rollback checks the flag and if the flag is set, it does not release ownership. After WL#7592 we need to hold anonymous ownership in case GTID_NEXT='ANONYMOUS'. Therefore we set the flag also in this case. @sql/sql_base.cc - When a client which has open tables disconnects, DROP TEMPORARY TABLE is written to the binary log. Before doing that, we must set GTID_NEXT='AUTOMATIC', so that it generates new GTIDs correctly. This was done only in the case of GTID_MODE=ON. Now we need to do it regardless of GTID_MODE. Moreover, in case a GTID is owned already, we must release ownership before setting GTID_NEXT='AUTOMATIC'. Thus we call gtid_state->update_on_rollback() first. @sql/sql_class.cc - Print any change of thd->owned_gtid to the debug trace. - Initialize new THD member variable. @sql/sql_class.h - Clarify life cycle of THD::owned_gtid. - Print any change of thd->owned_gtid to the debug trace. - Add THD::is_commit_in_middle_of_statement. @sql/sql_db.cc - If DROP TABLE fails, so that it generates DROP TABLE statements in the binary log, then we need to: - Call mysql_bin_log.commit after each statement. If we did not do this, all DROP TABLE statements would be written to the same statement cache, so there would just be one Anonymous_gtid_log_event. We need each DROP TABLE to have its own - If GTID_NEXT='UUID:NUMBER', and multiple DROP TABLE are needed, then there is no way to log this correctly. Thus, we generate ER_CANNOT_LOG_FAILED_DROP_DATABASE_WITH_MULTIPLE_STATEMENTS. This is a new error code. To handle this case correctly, we must also postpone the generation of ER_DB_DROP_RMDIR. If we would not move this error until later, then ER_DB_DROP_RMDIR would be generated before ER_CANNOT_LOG_FAILED_DROP_DATABASE_WITH_MULTIPLE_STATEMENTS, so then the user would never see ER_CANNOT_LOG_FAILED_DROP_DATABASE_WITH_MULTIPLE_STATEMENTS. - If GTID_NEXT='ANONYMOUS', it is not a problem that we generate multiple transactions. However, we must set thd->is_commit_in_middle_of_statement in order to hold anonymous ownership for the duration of the statement. @sql/sql_insert.cc - For CREATE ... SELECT, write the CREATE as a non-transactional statement, directly to the binlog. This prevents BEGIN and COMMIT statements from being generated around the CREATE statement. - Call mysql_bin_log.commit after writing the CREATE statement. This causes the row events to be written as a separate transaction, so they will be preceded by an Anonymous_gtid_log_event. - Do not release anonymous ownership in the middle of the statement. - Log a compensatory DROP TABLE statement if the CREATE...SELECT fails on the SELECT part. @sql/sql_parse.cc - CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE are very strange statements. When executed in transactional context, they behave like MyISAM: they leave the transaction context open and get logged between BEGIN and COMMIT, but cannot be rolled back. When executed outside transactional context, they get logged as DDL, without BEGIN/COMMIT. If we would call gtid_end_transaction without ending the transaction, then ownership would be released too early. This would cause an assertion in write_gtid, where it is expected that the thread holds ownership of whatever GTID_NEXT is set to (unless GTID_NEXT='AUTOMATIC'). This was not a concern before WL#7592, since gtid_end_transaction was only called if gtid_mode=ON, and CREATE TEMPORARY/DROP TEMPORARY are disallowed in transactional context when gtid_mode=ON. Now that we can call gtid_end_transaction also when gtid_mode=OFF, CREATE TEMPORARY/DROP TEMPORARY must only invoke gtid_end_transaction if executed outside transaction context. @sql/sql_table.cc - Call mysql_bin_log.commit regardless of gtid_mode, only avoid it between temporary tables inside a transaction. The code has three blocks: 1. nontransactional temporary tables 2. transactional temporary tables 3. non-temporary tables Before, the commit was at the end of block 1 and 2. We moved it to the beginning of block 2 and 3 instead, to simplify the condition. This does not change the logic for writing to the log: in both cases the point is that we do the commit *between* two calls to thd->binlog_query. - Fix some comments and re-wrap some long lines. - In parameters to mysql_bin_log.commit, use true/false instead of TRUE/FALSE, and add comments to clarify the meaning of the parameters. @mysql-test/suite/binlog/r/binlog_row_binlog.result - Update result file due to design change for logging CREATE ... SELECT in RBR. @mysql-test/suite/binlog/r/binlog_row_insert_select.result - Update result file due to design change for logging CREATE ... SELECT in RBR. @mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result - Update result file due to design change for logging CREATE ... SELECT in RBR. @mysql-test/suite/rpl/r/rpl_non_direct_row_mixing_engines.result - Update result file due to design change for logging CREATE ... SELECT in RBR. @mysql-test/suite/rpl/r/rpl_row_mixing_engines.result - Update result file due to design change for logging CREATE ... SELECT in RBR. @mysql-test/suite/rpl/t/rpl_stm_start_stop_slave.test - Suppress a warning, which is generated from its included file rpl_start_stop_slave.test @mysql-test/suite/rpl/r/rpl_stm_start_stop_slave.result - Update result file to remove a redundent suppression warning.
Sven Sandberg authoredWL#7592 step 13. GTIDs: Generate Gtid_log_event and Previous_gtids_log_event always. Fix binlogging of strange SQL statements. A few SQL statements have strange semantics that causes trouble for GTIDs. This includes DROP TABLE with multiple tables, CREATE TABLE ... SELECT, DROP DATABASE that fails on rmdir, OPTIMIZE/REPAIR/ANALYZE/CHECKSUM TABLE, CREATE TEMPORARY/DROP TEMPORARY, DROP TEMPORARY generated by client disconnect, and statements/transactions that mix both transactional and non-transactional updates. 1. Background: When DROP TABLE is used with multiple tables, and the tables are of different types (transactional/non-transactional or temporary/non-temporary), tables of the same type get grouped together and each group is logged as a separate statement. For example: DROP TABLE temporary, non_temporary gets logged as DROP TABLE temporary; DROP TABLE non_temporary. When GTID_MODE = ON, each such statement is assigned its own GTID. In order to generate the GTID, mysql_rm_table_no_locks calls mysql_bin_log.commit for each group of tables. Problem: 1. mysql_bin_log.commit is only called when gtid_mode != OFF. When gtid_mode == OFF, all the statements are written to the binary log in one operation. So after WL#7592 there is only one Anonymous_gtids_log_event, instead of one for each DROP TABLE. 2. If GTID_NEXT='ANONYMOUS', Gtid_state::update_on_commit releases anonymous ownership. But the statement should hold anonymous ownership until it completes. Fix: 1. Call mysql_bin_log.commit unconditionally. Note: inside a transaction, if only temporary tables are dropped, we should not call mysql_bin_log.commit, since the transactional context must remain open in this case. 2. Set thd->is_commit_in_middle_of_statement before calling mysql_bin_log.commit. This tells Gtid_state::update_on_rollback to not release anonymous ownership. 2. Background: Prior to this patch, when binlog_format=row, CREATE...SELECT gets written to the binary log as BEGIN CREATE row events COMMIT CREATE...SELECT is not allowed when gtid_mode=on (in fact, not when enforce_gtid_consistency=1). Problem: 1. Although CREATE without SELECT has an implicit commit, it appears in the middle of a transaction on the slave. Thus, after this worklog and prior to this patch, it gets logged as: Anonymous_gtids_log_event BEGIN CREATE row events COMMIT This causes problems on an MTS slave. 2. If GTID_NEXT='ANONYMOUS', Gtid_state::update_on_commit releases anonymous ownership. But the statement should hold anonymous ownership until it completes. Fix: 1. Call mysql_bin_log.commit after writing the CREATE statement. This causes CREATE...SELECT to be logged like: Anonymous_gtids_log_event CREATE Anonymous_gtids_log_event BEGIN row events COMMIT 2. Set thd->is_commit_in_middle_of_statement before calling mysql_bin_log.commit. This tells Gtid_state::update_on_rollback to not release anonymous ownership. A side-effect of this is that a CREATE...SELECT statement that fails in the SELECT part is already logged when the error happens, but the error causes the statement to be rolled back. To make this case work correctly, we log a compensatory DROP statement if the SELECT part fails. 3. Background: If DROP DATABASE fails after dropping some tables (e.g., if there are extra files in the database directory), then it writes a DROP TABLE statement that lists all the tables that it dropped. If there are many tables, this statement gets long. In this case, the server splits the statement into multiple DROP TABLE statements. Problem: 1. If the statement fails when GTID_NEXT='UUID:NUMBER', then there is no way to log this correctly. So we must generate an error, log nothing, and not add GTID to GTID_EXECUTED. 2. If the statement fails and generates multiple transactions when GTID_NEXT='AUTOMATIC' or 'ANONYMOUS', then we must generate multiple transactions, and each should have its own Gtid_log_event or Anonymous_gtid_log_event. 3. If GTID_NEXT='ANONYMOUS', we must hold anonymous ownership until all transactions have been written to the binary log. Fix: 1. Introduce a new error code, ER_CANNOT_LOG_PARTIAL_DROP_DATABASE_WITH_GTID, and generate the error if GTID_NEXT='UUID:NUMBER' and DROP DATABASE fails. 2. Call mysql_bin_log.commit after writing DROP TABLE statements. 3. Set thd->is_commit_in_middle_of_statement before calling mysql_bin_log.commit. This tells Gtid_state::update_on_rollback to not release anonymous ownership. 4. Background: OPTIMIZE/REPAIR/ANALYZE/CHECKSUM TABLE are written to the binary log even if they fail, after having called trans_rollback. Problem: trans_rollback calls gtid_state::update_on_rollback, which normally releases GTID ownership. But we must not release ownership before writing to the binary log. Fix: This was already fixed for the case gtid_mode=on; for that case we set a special flag in the THD object which tells gtid_state::update_on_rollback to not release ownership. Now we need to fix the case gtid_mode=off, so we set the flag in this case too. 5. Background: CREATE TEMPORARY and DROP TEMPORARY behave very strange. If executed outside transactional context, they behave as DDL: they get logged without BEGIN...COMMIT and cannot be rolled back. If executed in transactional context, they behave as non-transactional DML: they get logged inside BEGIN...COMMIT, leave the transactional context open, but cannot be rolled back. Before this patch, CREATE TEMPORARY and DROP TEMPORARY call gtid_end_transaction unconditionally. Problem: gtid_end_transaction ends the transactional context and releases ownership. This was not a problem before WL#7592 since gtid_end_transaction could only be called when gtid_mode=on, and when gtid_mode=on we disallow CREATE TEMPORARY and DROP TEMPORARY inside transactional context. However, after WL#7592, we call gtid_end_transaction also when gtid_mode=off, and gtid_end_transaction releases anonymous ownership. Fix: Do not call gtid_end_transaction for CREATE TEMPORARY and DROP TEMPORARY inside transaction context. 6. Background: When a client that has open temporary tables disconnects, the temporary tables are dropped and DROP TEMPORARY is written to the binary log. Problem: After WL#7592 and before this patch, if a client disconnects when GTID_NEXT='ANONYMOUS', the client would not hold anonymous ownership when writing to the binary log, which would trigger an assertion in write_gtid. There was no problem when GTID_NEXT='UUID:NUMBER', since this case was taken care of already before WL#7592. In this case, we set GTID_NEXT='AUTOMATIC' before dropping any tables. Fix: Set GTID_NEXT='AUTOMATIC' regardless of GTID_MODE. 7. Background: Anything that is written to the binary log is first written to a thread-specific IO_CACHE. There are two such IO_CACHES: the transaction cache and the statement cache. Transactional updates are written to the transaction cache and non-transactional updates are written to the statement cache. (Under certain conditions, such as when a non-transactional update appears after a transactional update within the same transaction and binlog_direct_non_transactional_updates=0, non-transactional updates are written to the transaction cache, but that is not relevant to the current discussion.) The statement cache is flushed when the statement ends and the transaction cache is flushed when the transaction ends. It is possible that both caches are non-empty if a single transaction or a single statement updates both transactional and non-transactional tables. This is not allowed when GTID_MODE=ON. If both caches are non-empty, an autocommitted transaction flushes first the statement cache and then the transaction cache (see binlog.cc:binlog_cache_mngr::flush). Both happen in the BGC flush stage. Problems: 7.1. When flushing both caches in an autocommitted transaction, and GTID_NEXT=AUTOMATIC, Gtid_state::generate_automatic_gtid is called once for each cache. Gtid_state::generate_automatic_gtid acquires anonymous ownership, but it also assumes (and asserts) that no ownership is held when entering the function. Thus, before this patch the assert would be raised when flushing the transaction cache, since the previous flush of the statement cache did acquire ownership. 7.2. When the statement cache is flushed in the middle of a transaction (due to a non-transactional update happening in the middle of the transaction), and GTID_NEXT=ANONYMOUS, before this patch ownership was released. This breaks the protocol that anonymous ownership must be held for the duration of the transaction when GTID_NEXT=ANONYMOUS. 7.3. Any statement calls gtid_reacquire_ownership_if_anonymous before it executes (mysql_execute_statement calls gtid_pre_statement_checks which calls gtid_reacquire_ownership_if_anonymous). This is important because when GTID_NEXT=ANONYMOUS, anonymous ownership must be held from when the transaction begins to execute. Therefore, when the transaction commits it also asserts that if GTID_NEXT=ANONYMOUS, it holds anonymous ownership. For the same reason, Rows_log_event::do_apply_event calls gtid_pre_statement_checks which calls gtid_reacquire_ownership_if_anonymous. However, before this patch the call to gtid_reacquire_ownership_if_anonymous was missing in Xid_log_event::do_apply_event. There is no existing case when GTID_NEXT=ANONYMOUS and anonymous ownership is not held when Xid_log_event::do_apply_event is called. However, it could potentially happen if a future version has a bug that sends a lonely Xid_log_event to the slave, or if the relay log is generated by something else than mysql, which has a bug. So we should call gtid_reacquire_ownership_if_anonymous at the beginning of Xid_log_event::do_apply_event too (just like we would do in a Query_log_event that contains a COMMIT query). Fix: 7.1. When both caches are non-empty, and GTID_NEXT='AUTOMATIC', call thd->clear_owned_gtids between the two cache flushes. This releases anonymous ownership so that the call to Gtid_cache::generate_automatic_gtid for the second flush is done without holding any ownership. 7.2. In Gtid_state::update_gtids_impl, do not release ownership if the transaction cache is nonempty. 7.3. Call gtid_reacquire_ownership_if_anonymous from Xid_log_event::do_apply_event. @mysql-test/extra/binlog_tests/drop_tables_logical_timestamp.inc - Remove this file. See binlog_mts_logical_clock.test for explanation. @mysql-test/extra/binlog_tests/logical_timestamping.inc - Use assert_logical_timestamps.inc instead of grep_pattern.inc See binlog_mts_logical_clock.test for explanation. @mysql-test/extra/rpl_tests/rpl_gtid_drop_multiple_tables.inc - New auxiliary file used by rpl_gtid_drop_multiple_tables_in_multiple_ways.inc. @mysql-test/extra/rpl_tests/ rpl_gtid_drop_multiple_tables_in_multiple_ways.inc - New auxiliary test file used by rpl_split_statements.test. @mysql-test/extra/rpl_tests/rpl_split_statements.test - New test case. @mysql-test/include/assert_binlog_events.inc - New test utility. This is needed by rpl_split_statements.inc. @mysql-test/include/assert_grep.inc - New test utility. This is needed by assert_logical_timestamps.inc. @mysql-test/include/assert_logical_timestamps.inc - New auxiliary test utility used by binlog_mts_logical_clock.test. @mysql-test/include/grep_pattern.inc - Add comment suggesting to use assert_grep.inc instead. @mysql-test/include/gtid_step_assert.inc - Add $gtid_step_gtid_mode_agnostic, needed by rpl_split_statements.test - Update test utility due to changes in gtid_utilities.inc. @mysql-test/include/gtid_utils.inc - Add new utility functions. @mysql-test/include/gtid_utils_end.inc - Drop the new functions introduced in gtid_utilities.inc. @mysql-test/include/rpl_connect.inc - Make it easier to map mysqltest connections to thread numbers in server debug traces. @mysql-test/include/rpl_connection.inc - Add $rpl_connection_silent_if_same. @mysql-test/include/rpl_get_end_of_relay_log.inc - New auxiliary test utility. This is needed by rpl_skip_to_end_of_relay_log.inc. @mysql-test/include/rpl_init.inc - Set some more mtr variables for convenience. @mysql-test/include/rpl_skip_to_end_of_relay_log.inc - New auxiliary test utility. This is needed by rpl_gtid_split_statements.test. @mysql-test/include/rpl_stop_slaves.inc - Correct a typo. @mysql-test/include/save_binlog_position.inc - New auxiliary test script, useful in combination with include/assert_binlog_events.inc. @mysql-test/include/set_gtid_next_gtid_mode_agnostic.inc - New auxiliary test utility. This is needed by rpl_split_statements.test. @mysql-test/include/wait_for_status_var.inc - Fix broken implementation of $status_fail_query. @mysql-test/suite/binlog/r/binlog_gtid_utils.result - Update result file for modified test. @mysql-test/suite/binlog/r/binlog_mts_logical_clock.result - Update result file due to change in test. @mysql-test/suite/binlog/r/binlog_mts_logical_clock_gtid.result - Update result file due to change in test. @mysql-test/suite/binlog/t/binlog_gtid_utils.test - Add tests for new utility function. @mysql-test/suite/binlog/t/binlog_mts_logical_clock.test - This test started failing because DROP TABLE is now logged differently. The failure was in 'grep_pattern.inc' executed just after drop_tables_logical_timestamp.inc. This was fixed by changing the test assertion. - The test assertion following drop_tables_logical_timestamp.inc belongs inside drop_tables_logical_timestamp.inc. Moved the test assertion there. - The tests in drop_tables_logical_timestamp.inc were located in this file because they differed between gtid_mode=on and gtid_mode=off. But because of the change in logging of DROP TABLE, the test works in the same way regardless of gtid_mode. Therefore, the contents of drop_tables_logical_timestamp.inc was moved into logical_timestamping.inc and drop_tables_logical_timestamp.inc as removed. - This file used grep_pattern.inc to check assertions. This made the test very verbose and hard to read and understand. Also, it was imprecise since any future server bug that introduces a logical timestamp that does not match the given pattern would go unnoticed by the test. Therefore, replaced all grep_pattern.inc by assert_logical_timestamps.inc. @mysql-test/suite/binlog/t/binlog_mts_logical_clock_gtid.test - See binlog_mts_logical_clock.test. @mysql-test/suite/rpl/r/rpl_gtid_create_select.result - Result file for new test. @mysql-test/suite/rpl/r/rpl_gtid_disconnect_drop_temporary_table.result - Result file for new test. @mysql-test/suite/rpl/r/rpl_gtid_split_statements.result - Result file for new test. @mysql-test/suite/rpl/r/rpl_no_gtid_split_statements.result - Result file for new test. @mysql-test/suite/rpl/r/rpl_semi_sync.result - When semisync is enabled, there will be one ack for each transaction. Since we now have more 'transactions' due to the extra Anonymous_log_event preceding each DROP statement, this changes the result file. @mysql-test/suite/rpl/t/rpl_drop_db.test - This test can now run regardless of GTID_MODE. @mysql-test/suite/rpl/t/rpl_gtid_create_select.test - New test to verify that CREATE ... SELECT is logged as expected. @mysql-test/suite/rpl/t/rpl_gtid_disconnect_drop_temporary_table.test - New test file to verify that DROP TEMPORARY generated by client disconnect is logged correctly. @mysql-test/suite/rpl/t/rpl_gtid_split_statements.test - New test file. @mysql-test/suite/rpl/t/rpl_invoked_features.test - Explain why not_gtid_enabled is used. @mysql-test/suite/rpl/t/rpl_mixed_drop_create_temp_table.test - Explain why not_gtid_enabled is used. @mysql-test/suite/rpl/t/rpl_no_gtid_split_statements.test - New test file. @mysql-test/suite/rpl/t/rpl_stop_slave.test - Explain why not_gtid_enabled is used. @sql/rpl_gtid.h - Add need_lock parameter to Sid_map::sidno_to_sid, Gtid::to_string, Gtid::dbug_print, Gtid_specification::to_string, and Gtid_specification::dbug_print, to simplify the use of these functions. - Simplify the definition of Gtid_specification::MAX_TEXT_LENGTH. - Make gtid_reacquire_ownership_if_anonymous extern so that it can be called from log_event.cc. @sql/rpl_gtid_execution.cc - Make gtid_reacquire_ownership_if_anonymous extern so that it can be called from log_event.cc. - Move parts of gtid_pre_statement_checks into an own function, gtid_reacquire_ownership_if_anonymous. This is just to make the logic more easy to follow. - Make gtid_reacquire_ownership acquire anonymous ownership in case GTID_NEXT='ANONYMOUS'. This is needed for cases like: SET AUTOCOMMIT=1; SET GTID_NEXT='ANONYMOUS'; INSERT; BEGIN; The first INSERT will commit the transaction and therefore release anonymous ownership. Then the BEGIN has to re-acquire anonymous ownership so that it can execute correctly. (The logic to reacquire ownership in this case is needed because we allow user to set GTID_NEXT='ANONYMOUS' and then execute multiple transactions. This differs from the case of GTID_NEXT='UUID:NUMBER', where we generate an error if user tries to execute a second transaction without setting GTID_NEXT again. The reason we want to generate an error in this case is that otherwise the GTID autoskip feature would silently skip the transaction and user would not notice it.) - Move the call to gtid_reacquire_ownership_if_anonymous earlier in gtid_pre_statemen_checks. This is needed for the statements BEGIN/ROLLBACK/COMMIT in two cases. Case 1: User executes: SET GTID_NEXT='ANONYMOUS'; BEGIN; INSERT; COMMIT; BEGIN; Here the second BEGIN needs to acquire anonymous ownership so that anonymous ownership is held during the entire transaction. Case 2: The relay log comes from an old master that does not generate Anonymous_gtids_log_event, so it begins with: Format_desc Previous_gtids BEGIN; Then the Format_desc will set GTID_NEXT=NOT_YET_DETERMINED, and the BEGIN needs to acquire anonyous ownership. The only case where we should not acquire anonymous ownership is for a SET statement that does not invoke a stored function. This is important in case a client processes output from mysqlbinlog, for a binary log generated with GTID_MODE=ON. Then, the server will first execute a Format_description_log_event, which will set GTID_NEXT='NOT_YET_DETERMINED', followed by a SET GTID_NEXT='UUID:NUMBER' statement. If it would try to reacquire anonymous ownership in this case, and gtid_mode=on, an error would be generated since anonymous transactions are not allowed when gtid_mode=on. @sql/rpl_gtid_misc.cc - Add need_lock parameter to Gtid::to_string, to simplify the use of this function. - Allow Gtid::to_string to take sid_map==NULL in debug mode, so that debug printouts can show the sidno. @sql/rpl_gtid_specification.cc - Add need_lock to Gtid_specification::to_string. @sql/rpl_gtid_state.cc - Print any change of thd->owned_gtid to the debug trace. - Implement thd->is_commit_in_middle_of_statement. - Return early from update_gtids_impl to avoid releasing ownership, if the transaction cache is nonempty and GTID_NEXT=ANONYMOUS. @sql/share/errmsg-utf8.txt - New error message. @sql/binlog.cc - Release anonymous ownership between flushing the statement cache and flushing the transaction cache, if GTID_NEXT=AUTOMATIC. @sql/log_event.cc - Call gtid_reacquire_ownership_if_anonymous in Xid_log_event::do_apply_event. @sql/sql_admin.cc - Administrational statements (OPTIMIZE TABLE, REPAIR TABLE, CHECKSUM TABLE, ANALYZE TABLE) behave strange if they fail: they first call trans_rollback and then write the statement to the binary log. This causes problems for GTIDs, since ha_rollback eventually calls gtid_state::update_on_rollback, which releases GTID ownership. Then no GTID is owned when it comes to writing the statements to the binary log. This was handled when GTID_NEXT='UUID:NUMBER' by setting the flag thd->skip_gtid_rollback before calling ha_rollback. gtid_state::update_on_rollback checks the flag and if the flag is set, it does not release ownership. After WL#7592 we need to hold anonymous ownership in case GTID_NEXT='ANONYMOUS'. Therefore we set the flag also in this case. @sql/sql_base.cc - When a client which has open tables disconnects, DROP TEMPORARY TABLE is written to the binary log. Before doing that, we must set GTID_NEXT='AUTOMATIC', so that it generates new GTIDs correctly. This was done only in the case of GTID_MODE=ON. Now we need to do it regardless of GTID_MODE. Moreover, in case a GTID is owned already, we must release ownership before setting GTID_NEXT='AUTOMATIC'. Thus we call gtid_state->update_on_rollback() first. @sql/sql_class.cc - Print any change of thd->owned_gtid to the debug trace. - Initialize new THD member variable. @sql/sql_class.h - Clarify life cycle of THD::owned_gtid. - Print any change of thd->owned_gtid to the debug trace. - Add THD::is_commit_in_middle_of_statement. @sql/sql_db.cc - If DROP TABLE fails, so that it generates DROP TABLE statements in the binary log, then we need to: - Call mysql_bin_log.commit after each statement. If we did not do this, all DROP TABLE statements would be written to the same statement cache, so there would just be one Anonymous_gtid_log_event. We need each DROP TABLE to have its own - If GTID_NEXT='UUID:NUMBER', and multiple DROP TABLE are needed, then there is no way to log this correctly. Thus, we generate ER_CANNOT_LOG_FAILED_DROP_DATABASE_WITH_MULTIPLE_STATEMENTS. This is a new error code. To handle this case correctly, we must also postpone the generation of ER_DB_DROP_RMDIR. If we would not move this error until later, then ER_DB_DROP_RMDIR would be generated before ER_CANNOT_LOG_FAILED_DROP_DATABASE_WITH_MULTIPLE_STATEMENTS, so then the user would never see ER_CANNOT_LOG_FAILED_DROP_DATABASE_WITH_MULTIPLE_STATEMENTS. - If GTID_NEXT='ANONYMOUS', it is not a problem that we generate multiple transactions. However, we must set thd->is_commit_in_middle_of_statement in order to hold anonymous ownership for the duration of the statement. @sql/sql_insert.cc - For CREATE ... SELECT, write the CREATE as a non-transactional statement, directly to the binlog. This prevents BEGIN and COMMIT statements from being generated around the CREATE statement. - Call mysql_bin_log.commit after writing the CREATE statement. This causes the row events to be written as a separate transaction, so they will be preceded by an Anonymous_gtid_log_event. - Do not release anonymous ownership in the middle of the statement. - Log a compensatory DROP TABLE statement if the CREATE...SELECT fails on the SELECT part. @sql/sql_parse.cc - CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE are very strange statements. When executed in transactional context, they behave like MyISAM: they leave the transaction context open and get logged between BEGIN and COMMIT, but cannot be rolled back. When executed outside transactional context, they get logged as DDL, without BEGIN/COMMIT. If we would call gtid_end_transaction without ending the transaction, then ownership would be released too early. This would cause an assertion in write_gtid, where it is expected that the thread holds ownership of whatever GTID_NEXT is set to (unless GTID_NEXT='AUTOMATIC'). This was not a concern before WL#7592, since gtid_end_transaction was only called if gtid_mode=ON, and CREATE TEMPORARY/DROP TEMPORARY are disallowed in transactional context when gtid_mode=ON. Now that we can call gtid_end_transaction also when gtid_mode=OFF, CREATE TEMPORARY/DROP TEMPORARY must only invoke gtid_end_transaction if executed outside transaction context. @sql/sql_table.cc - Call mysql_bin_log.commit regardless of gtid_mode, only avoid it between temporary tables inside a transaction. The code has three blocks: 1. nontransactional temporary tables 2. transactional temporary tables 3. non-temporary tables Before, the commit was at the end of block 1 and 2. We moved it to the beginning of block 2 and 3 instead, to simplify the condition. This does not change the logic for writing to the log: in both cases the point is that we do the commit *between* two calls to thd->binlog_query. - Fix some comments and re-wrap some long lines. - In parameters to mysql_bin_log.commit, use true/false instead of TRUE/FALSE, and add comments to clarify the meaning of the parameters. @mysql-test/suite/binlog/r/binlog_row_binlog.result - Update result file due to design change for logging CREATE ... SELECT in RBR. @mysql-test/suite/binlog/r/binlog_row_insert_select.result - Update result file due to design change for logging CREATE ... SELECT in RBR. @mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result - Update result file due to design change for logging CREATE ... SELECT in RBR. @mysql-test/suite/rpl/r/rpl_non_direct_row_mixing_engines.result - Update result file due to design change for logging CREATE ... SELECT in RBR. @mysql-test/suite/rpl/r/rpl_row_mixing_engines.result - Update result file due to design change for logging CREATE ... SELECT in RBR. @mysql-test/suite/rpl/t/rpl_stm_start_stop_slave.test - Suppress a warning, which is generated from its included file rpl_start_stop_slave.test @mysql-test/suite/rpl/r/rpl_stm_start_stop_slave.result - Update result file to remove a redundent suppression warning.
Loading