Skip to content
  • Venkatesh Duggirala's avatar
    56bcc9c8
    Bug#17283409 4-WAY DEADLOCK: ZOMBIES, PURGING BINLOGS, · 56bcc9c8
    Venkatesh Duggirala authored
          SHOW PROCESSLIST, SHOW BINLOGS
          
          Merge from mysql-5.5 ( Fix is changed from
          mysql-5.5's patch)
          
          Problem:  A deadlock was occurring when 4 threads were
          involved in acquiring locks in the following way
          Thread 1: Dump thread ( Slave is reconnecting, so on
                        Master, a new dump thread is trying kill
                        zombie dump threads. It acquired thread's
                        LOCK_thd_data and it is about to acquire
                        mysys_var->current_mutex ( which LOCK_log)
          Thread 2: Application thread is executing show binlogs and
                         acquired LOCK_log and it is about to acquire
                         LOCK_index.
          Thread 3: Application thread is executing Purge binary logs
                         and acquired LOCK_index and it is about to
                         acquire LOCK_thread_count.
          Thread 4: Application thread is executing show processlist
                         and acquired LOCK_thread_count and it is
                         about to acquire zombie dump thread's
                         LOCK_thd_data.
          Deadlock Cycle:
               Thread 1 -> Thread 2 -> Thread 3-> Thread 4 ->Thread 1
          
          The same above deadlock was observed even when thread 4 is
          executing 'SELECT * FROM information_schema.processlist' command and
          acquired LOCK_thread_count and it is about to acquire zombie
          dump thread's LOCK_thd_data.
          
          Analysis:
          There are four locks involved in the deadlock.  LOCK_log,
          LOCK_thread_count, LOCK_index and LOCK_thd_data.
          LOCK_log, LOCK_thread_count, LOCK_index are global mutexes
          where as LOCK_thd_data is local to a thread.
          We can divide these four locks in two groups.
          Group 1 consists of LOCK_log and LOCK_index and the order
          should be LOCK_log followed by LOCK_index.
          Group 2 consists of other two mutexes
          LOCK_thread_count, LOCK_thd_data and the order should
          be LOCK_thread_count followed by LOCK_thd_data.
          Unfortunately, there is no specific predefined lock order defined
          to follow in the MySQL system when it comes to locks across these
          two groups. In the above problematic example,
          there is no problem in the way we are acquiring the locks
          if you see each thread individually.
          But If you combine all 4 threads, they end up in a deadlock.
          
          Fix:
          Since everything seems to be fine in the way threads are taking locks,
          In this patch We are changing the duration of the locks in Thread 4
          to break the deadlock. i.e., before the patch, Thread 4
          ('show processlist' command) mysqld_list_processes()
          function acquires LOCK_thread_count for the complete duration
          of the function and it also acquires/releases
          each thread's LOCK_thd_data. Instead of it, Now it will take
          a copy of THDs from global_thread_list and perform traversal
          (on copied THDs) only after releasing  LOCK on LOCK_thread_count.
          During traversal(on copied THDs), removal from global_thread_list is
          blocked using another mutex LOCK_thd_remove such that
          THD copied are valid during traversal(otherwise remove destroys THD).
          
          Now the new locking order after this patch is:
          LOCK_thd_remove -> LOCK_thd_data -> LOCK_log ->
          LOCK_index -> LOCK_thread_count
    56bcc9c8
    Bug#17283409 4-WAY DEADLOCK: ZOMBIES, PURGING BINLOGS,
    Venkatesh Duggirala authored
          SHOW PROCESSLIST, SHOW BINLOGS
          
          Merge from mysql-5.5 ( Fix is changed from
          mysql-5.5's patch)
          
          Problem:  A deadlock was occurring when 4 threads were
          involved in acquiring locks in the following way
          Thread 1: Dump thread ( Slave is reconnecting, so on
                        Master, a new dump thread is trying kill
                        zombie dump threads. It acquired thread's
                        LOCK_thd_data and it is about to acquire
                        mysys_var->current_mutex ( which LOCK_log)
          Thread 2: Application thread is executing show binlogs and
                         acquired LOCK_log and it is about to acquire
                         LOCK_index.
          Thread 3: Application thread is executing Purge binary logs
                         and acquired LOCK_index and it is about to
                         acquire LOCK_thread_count.
          Thread 4: Application thread is executing show processlist
                         and acquired LOCK_thread_count and it is
                         about to acquire zombie dump thread's
                         LOCK_thd_data.
          Deadlock Cycle:
               Thread 1 -> Thread 2 -> Thread 3-> Thread 4 ->Thread 1
          
          The same above deadlock was observed even when thread 4 is
          executing 'SELECT * FROM information_schema.processlist' command and
          acquired LOCK_thread_count and it is about to acquire zombie
          dump thread's LOCK_thd_data.
          
          Analysis:
          There are four locks involved in the deadlock.  LOCK_log,
          LOCK_thread_count, LOCK_index and LOCK_thd_data.
          LOCK_log, LOCK_thread_count, LOCK_index are global mutexes
          where as LOCK_thd_data is local to a thread.
          We can divide these four locks in two groups.
          Group 1 consists of LOCK_log and LOCK_index and the order
          should be LOCK_log followed by LOCK_index.
          Group 2 consists of other two mutexes
          LOCK_thread_count, LOCK_thd_data and the order should
          be LOCK_thread_count followed by LOCK_thd_data.
          Unfortunately, there is no specific predefined lock order defined
          to follow in the MySQL system when it comes to locks across these
          two groups. In the above problematic example,
          there is no problem in the way we are acquiring the locks
          if you see each thread individually.
          But If you combine all 4 threads, they end up in a deadlock.
          
          Fix:
          Since everything seems to be fine in the way threads are taking locks,
          In this patch We are changing the duration of the locks in Thread 4
          to break the deadlock. i.e., before the patch, Thread 4
          ('show processlist' command) mysqld_list_processes()
          function acquires LOCK_thread_count for the complete duration
          of the function and it also acquires/releases
          each thread's LOCK_thd_data. Instead of it, Now it will take
          a copy of THDs from global_thread_list and perform traversal
          (on copied THDs) only after releasing  LOCK on LOCK_thread_count.
          During traversal(on copied THDs), removal from global_thread_list is
          blocked using another mutex LOCK_thd_remove such that
          THD copied are valid during traversal(otherwise remove destroys THD).
          
          Now the new locking order after this patch is:
          LOCK_thd_remove -> LOCK_thd_data -> LOCK_log ->
          LOCK_index -> LOCK_thread_count
Loading