Skip to content
  • Sujatha Sivakumar's avatar
    cd010cb6
    Bug#17879675:SHUTDOWN HANG IF SEMISYNC IS ENABLED AND SLAVE · cd010cb6
    Sujatha Sivakumar authored
    ABORT
    
    Problem:
    ========
    Step 1. enable semisync and set a big value for
    rpl_semi_sync_master_timeout.
    
    Step 2. kill io thread of slave, so the threads on master
    will keep on WAITING ACK FROM SLAVE.
    
    Step 3. shutdown the master ---hang
    
    Analysis:
    ========
    On master when semisync is enabled the transaction thread
    will wait for an ack from the slave in a conditional timed
    wait. Since the IO thread is already killed and on master
    a huge timeout value is set transaction thread will continue
    to wait for an ack for a long time. At this time if shutdown
    command is issued dump thread will try to go down. But dump
    thread when goes down it has a check if user has enabled
    rpl_semi_sync_master_wait_no_slave or not. If it is enabled
    dump thread will go down without switching off semi sync.
    Transaction thread will continue to wait for an ack even
    when slave count drops to zero. Because of this shutdown
    also will wait for the transaction thread to go down and
    shutdown hangs.
    
    Fix:
    ===
    At the time of shutdown if dump thread finds that there no
    semi sync slaves that are connected to master it will switch
    off semisync even when rpl_semi_sync_master_wait_no_slave
    is set to on. i.e if shutdown finds no slaves are there to
    provide ack it will not honour
    rpl_semi_sync_master_wait_no_slave and it will switch off
    semisync. The transaction thread when woken up realises the
    semi sync is switched off it will not wait any more for ack
    it will continue and shutdown will be complete. When semi
    sync is forcefully shutdown at the time of waiting for an
    ack a warning is written to error log to convey to users
    that it was forceful shutdown.
    
    There can be a scenario where dump thread is already gone
    and transaction thread still waits for an ack. Hence a
    similar fix is implemented for the waiting transaction
    thread. I.e at the time of shutdown the thread gets a
    wakeup signal and it checks if shutdown is in progress
    if no slaves are connected then it will switch off the
    semi sync.
    cd010cb6
    Bug#17879675:SHUTDOWN HANG IF SEMISYNC IS ENABLED AND SLAVE
    Sujatha Sivakumar authored
    ABORT
    
    Problem:
    ========
    Step 1. enable semisync and set a big value for
    rpl_semi_sync_master_timeout.
    
    Step 2. kill io thread of slave, so the threads on master
    will keep on WAITING ACK FROM SLAVE.
    
    Step 3. shutdown the master ---hang
    
    Analysis:
    ========
    On master when semisync is enabled the transaction thread
    will wait for an ack from the slave in a conditional timed
    wait. Since the IO thread is already killed and on master
    a huge timeout value is set transaction thread will continue
    to wait for an ack for a long time. At this time if shutdown
    command is issued dump thread will try to go down. But dump
    thread when goes down it has a check if user has enabled
    rpl_semi_sync_master_wait_no_slave or not. If it is enabled
    dump thread will go down without switching off semi sync.
    Transaction thread will continue to wait for an ack even
    when slave count drops to zero. Because of this shutdown
    also will wait for the transaction thread to go down and
    shutdown hangs.
    
    Fix:
    ===
    At the time of shutdown if dump thread finds that there no
    semi sync slaves that are connected to master it will switch
    off semisync even when rpl_semi_sync_master_wait_no_slave
    is set to on. i.e if shutdown finds no slaves are there to
    provide ack it will not honour
    rpl_semi_sync_master_wait_no_slave and it will switch off
    semisync. The transaction thread when woken up realises the
    semi sync is switched off it will not wait any more for ack
    it will continue and shutdown will be complete. When semi
    sync is forcefully shutdown at the time of waiting for an
    ack a warning is written to error log to convey to users
    that it was forceful shutdown.
    
    There can be a scenario where dump thread is already gone
    and transaction thread still waits for an ack. Hence a
    similar fix is implemented for the waiting transaction
    thread. I.e at the time of shutdown the thread gets a
    wakeup signal and it checks if shutdown is in progress
    if no slaves are connected then it will switch off the
    semi sync.
Loading