-
Sujatha Sivakumar authored
ABORT Problem: ======== Step 1. enable semisync and set a big value for rpl_semi_sync_master_timeout. Step 2. kill io thread of slave, so the threads on master will keep on WAITING ACK FROM SLAVE. Step 3. shutdown the master ---hang Analysis: ======== On master when semisync is enabled the transaction thread will wait for an ack from the slave in a conditional timed wait. Since the IO thread is already killed and on master a huge timeout value is set transaction thread will continue to wait for an ack for a long time. At this time if shutdown command is issued dump thread will try to go down. But dump thread when goes down it has a check if user has enabled rpl_semi_sync_master_wait_no_slave or not. If it is enabled dump thread will go down without switching off semi sync. Transaction thread will continue to wait for an ack even when slave count drops to zero. Because of this shutdown also will wait for the transaction thread to go down and shutdown hangs. Fix: === At the time of shutdown if dump thread finds that there no semi sync slaves that are connected to master it will switch off semisync even when rpl_semi_sync_master_wait_no_slave is set to on. i.e if shutdown finds no slaves are there to provide ack it will not honour rpl_semi_sync_master_wait_no_slave and it will switch off semisync. The transaction thread when woken up realises the semi sync is switched off it will not wait any more for ack it will continue and shutdown will be complete. When semi sync is forcefully shutdown at the time of waiting for an ack a warning is written to error log to convey to users that it was forceful shutdown. There can be a scenario where dump thread is already gone and transaction thread still waits for an ack. Hence a similar fix is implemented for the waiting transaction thread. I.e at the time of shutdown the thread gets a wakeup signal and it checks if shutdown is in progress if no slaves are connected then it will switch off the semi sync.
Sujatha Sivakumar authoredABORT Problem: ======== Step 1. enable semisync and set a big value for rpl_semi_sync_master_timeout. Step 2. kill io thread of slave, so the threads on master will keep on WAITING ACK FROM SLAVE. Step 3. shutdown the master ---hang Analysis: ======== On master when semisync is enabled the transaction thread will wait for an ack from the slave in a conditional timed wait. Since the IO thread is already killed and on master a huge timeout value is set transaction thread will continue to wait for an ack for a long time. At this time if shutdown command is issued dump thread will try to go down. But dump thread when goes down it has a check if user has enabled rpl_semi_sync_master_wait_no_slave or not. If it is enabled dump thread will go down without switching off semi sync. Transaction thread will continue to wait for an ack even when slave count drops to zero. Because of this shutdown also will wait for the transaction thread to go down and shutdown hangs. Fix: === At the time of shutdown if dump thread finds that there no semi sync slaves that are connected to master it will switch off semisync even when rpl_semi_sync_master_wait_no_slave is set to on. i.e if shutdown finds no slaves are there to provide ack it will not honour rpl_semi_sync_master_wait_no_slave and it will switch off semisync. The transaction thread when woken up realises the semi sync is switched off it will not wait any more for ack it will continue and shutdown will be complete. When semi sync is forcefully shutdown at the time of waiting for an ack a warning is written to error log to convey to users that it was forceful shutdown. There can be a scenario where dump thread is already gone and transaction thread still waits for an ack. Hence a similar fix is implemented for the waiting transaction thread. I.e at the time of shutdown the thread gets a wakeup signal and it checks if shutdown is in progress if no slaves are connected then it will switch off the semi sync.
Loading