Bug#17283409  4-WAY DEADLOCK: ZOMBIES, PURGING BINLOGS, SHOW PROCESSLIST, SHOW BINLOGS

Reverted revisions are:

------------------------------------------------------------
revno: 2876.552.146
revision-id: venkatesh.duggirala@oracle.com-20140718110044-ivyb4xygx4q59puv
parent: ashish.y.agarwal@oracle.com-20140717135640-3m6s2n57s1th3vom
committer: Venkatesh Duggirala<venkatesh.duggirala@oracle.com>
branch nick: mysql-5.6.20-release
timestamp: Fri 2014-07-18 16:30:44 +0530
message:
  Bug#17283409  4-WAY DEADLOCK: ZOMBIES, PURGING BINLOGS,
  SHOW PROCESSLIST, SHOW BINLOGS
  
  Post push fix (removing try-catch)
------------------------------------------------------------
revno: 2876.552.16 [merge]
revision-id: venkatesh.duggirala@oracle.com-20140509042350-ni6xx9khsej94hl9
parent: venkatesh.duggirala@oracle.com-20140508124746-8de4t4utydx15x2k
parent: venkatesh.duggirala@oracle.com-20140509042215-kukwf135nfyy9yy2
committer: Venkatesh Duggirala<venkatesh.duggirala@oracle.com>
branch nick: mysql-5.6
timestamp: Fri 2014-05-09 09:53:50 +0530
message:
  Bug#17283409  4-WAY DEADLOCK: ZOMBIES, PURGING BINLOGS,
  SHOW PROCESSLIST, SHOW BINLOGS
  
  Fixing post push test failure (MTR does not like giving
  127.0.0.1 for localhost incase of --embedded run, it thinks
  it is an external ip address)
    ------------------------------------------------------------
    revno: 2875.596.13
    revision-id: venkatesh.duggirala@oracle.com-20140509042215-kukwf135nfyy9yy2
    parent: venkatesh.duggirala@oracle.com-20140508124301-9kqzomeujeucnxcu
    committer: Venkatesh Duggirala<venkatesh.duggirala@oracle.com>
    branch nick: mysql-5.5
    timestamp: Fri 2014-05-09 09:52:15 +0530
    message:
      Bug#17283409  4-WAY DEADLOCK: ZOMBIES, PURGING BINLOGS,
      SHOW PROCESSLIST, SHOW BINLOGS
      
      Fixing post push test failure (MTR does not like giving
      127.0.0.1 for localhost incase of --embedded run, it thinks
      it is an external ip address)
------------------------------------------------------------
revno: 2876.552.15 [merge]
revision-id: venkatesh.duggirala@oracle.com-20140508124746-8de4t4utydx15x2k
parent: mithun.c.y@oracle.com-20140508092217-2vyrjxxtzf18r0ho
parent: venkatesh.duggirala@oracle.com-20140508124301-9kqzomeujeucnxcu
committer: Venkatesh Duggirala<venkatesh.duggirala@oracle.com>
branch nick: mysql-5.6
timestamp: Thu 2014-05-08 18:17:46 +0530
message:
        Bug#17283409  4-WAY DEADLOCK: ZOMBIES, PURGING BINLOGS,
        SHOW PROCESSLIST, SHOW BINLOGS
        
        Merge from mysql-5.5 ( Fix is changed from
        mysql-5.5's patch)
        
        Problem:  A deadlock was occurring when 4 threads were
        involved in acquiring locks in the following way
        Thread 1: Dump thread ( Slave is reconnecting, so on
                      Master, a new dump thread is trying kill
                      zombie dump threads. It acquired thread's
                      LOCK_thd_data and it is about to acquire
                      mysys_var->current_mutex ( which LOCK_log)
        Thread 2: Application thread is executing show binlogs and
                       acquired LOCK_log and it is about to acquire
                       LOCK_index.
        Thread 3: Application thread is executing Purge binary logs
                       and acquired LOCK_index and it is about to
                       acquire LOCK_thread_count.
        Thread 4: Application thread is executing show processlist
                       and acquired LOCK_thread_count and it is
                       about to acquire zombie dump thread's
                       LOCK_thd_data.
        Deadlock Cycle:
             Thread 1 -> Thread 2 -> Thread 3-> Thread 4 ->Thread 1
        
        The same above deadlock was observed even when thread 4 is
        executing 'SELECT * FROM information_schema.processlist' command and
        acquired LOCK_thread_count and it is about to acquire zombie
        dump thread's LOCK_thd_data.
        
        Analysis:
        There are four locks involved in the deadlock.  LOCK_log,
        LOCK_thread_count, LOCK_index and LOCK_thd_data.
        LOCK_log, LOCK_thread_count, LOCK_index are global mutexes
        where as LOCK_thd_data is local to a thread.
        We can divide these four locks in two groups.
        Group 1 consists of LOCK_log and LOCK_index and the order
        should be LOCK_log followed by LOCK_index.
        Group 2 consists of other two mutexes
        LOCK_thread_count, LOCK_thd_data and the order should
        be LOCK_thread_count followed by LOCK_thd_data.
        Unfortunately, there is no specific predefined lock order defined
        to follow in the MySQL system when it comes to locks across these
        two groups. In the above problematic example,
        there is no problem in the way we are acquiring the locks
        if you see each thread individually.
        But If you combine all 4 threads, they end up in a deadlock.
        
        Fix:
        Since everything seems to be fine in the way threads are taking locks,
        In this patch We are changing the duration of the locks in Thread 4
        to break the deadlock. i.e., before the patch, Thread 4
        ('show processlist' command) mysqld_list_processes()
        function acquires LOCK_thread_count for the complete duration
        of the function and it also acquires/releases
        each thread's LOCK_thd_data. Instead of it, Now it will take
        a copy of THDs from global_thread_list and perform traversal
        (on copied THDs) only after releasing  LOCK on LOCK_thread_count.
        During traversal(on copied THDs), removal from global_thread_list is
        blocked using another mutex LOCK_thd_remove such that
        THD copied are valid during traversal(otherwise remove destroys THD).
        
        Now the new locking order after this patch is:
        LOCK_thd_remove -> LOCK_thd_data -> LOCK_log ->
        LOCK_index -> LOCK_thread_count
    ------------------------------------------------------------
    revno: 2875.596.12
    revision-id: venkatesh.duggirala@oracle.com-20140508124301-9kqzomeujeucnxcu
    parent: mithun.c.y@oracle.com-20140508091953-qafkm6r1dhycdjyh
    committer: Venkatesh Duggirala<venkatesh.duggirala@oracle.com>
    branch nick: mysql-5.5
    timestamp: Thu 2014-05-08 18:13:01 +0530
    message:
      Bug#17283409  4-WAY DEADLOCK: ZOMBIES, PURGING BINLOGS,
      SHOW PROCESSLIST, SHOW BINLOGS
      
      Problem:  A deadlock was occurring when 4 threads were
      involved in acquiring locks in the following way
      Thread 1: Dump thread ( Slave is reconnecting, so on
                    Master, a new dump thread is trying kill
                    zombie dump threads. It acquired thread's
                    LOCK_thd_data and it is about to acquire
                    mysys_var->current_mutex ( which LOCK_log)
      Thread 2: Application thread is executing show binlogs and
                     acquired LOCK_log and it is about to acquire
                     LOCK_index.
      Thread 3: Application thread is executing Purge binary logs
                     and acquired LOCK_index and it is about to
                     acquire LOCK_thread_count.
      Thread 4: Application thread is executing show processlist
                     and acquired LOCK_thread_count and it is
                     about to acquire zombie dump thread's
                     LOCK_thd_data.
      Deadlock Cycle:
           Thread 1 -> Thread 2 -> Thread 3-> Thread 4 ->Thread 1
      
      The same above deadlock was observed even when thread 4 is
      executing 'SELECT * FROM information_schema.processlist' command and
      acquired LOCK_thread_count and it is about to acquire zombie
      dump thread's LOCK_thd_data.
      
      Analysis:
      There are four locks involved in the deadlock.  LOCK_log,
      LOCK_thread_count, LOCK_index and LOCK_thd_data.
      LOCK_log, LOCK_thread_count, LOCK_index are global mutexes
      where as LOCK_thd_data is local to a thread.
      We can divide these four locks in two groups.
      Group 1 consists of LOCK_log and LOCK_index and the order
      should be LOCK_log followed by LOCK_index.
      Group 2 consists of other two mutexes
      LOCK_thread_count, LOCK_thd_data and the order should
      be LOCK_thread_count followed by LOCK_thd_data.
      Unfortunately, there is no specific predefined lock order defined
      to follow in the MySQL system when it comes to locks across these
      two groups. In the above problematic example,
      there is no problem in the way we are acquiring the locks
      if you see each thread individually.
      But If you combine all 4 threads, they end up in a deadlock.
      
      Fix: 
      Since everything seems to be fine in the way threads are taking locks,
      In this patch We are changing the duration of the locks in Thread 4
      to break the deadlock. i.e., before the patch, Thread 4
      ('show processlist' command) mysqld_list_processes()
      function acquires LOCK_thread_count for the complete duration
      of the function and it also acquires/releases
      each thread's LOCK_thd_data.
      
      LOCK_thread_count is used to protect addition and
      deletion of threads in global threads list. While show
      process list is looping through all the existing threads,
      it will be a problem if a thread is exited but there is no problem
      if a new thread is added to the system. Hence a new mutex is
      introduced "LOCK_thd_remove" which will protect deletion
      of a thread from global threads list. All threads which are
      getting exited should acquire LOCK_thd_remove
      followed by LOCK_thread_count. (It should take LOCK_thread_count
      also because other places of the code still thinks that exit thread
      is protected with LOCK_thread_count. In this fix, we are changing
      only 'show process list' query logic )
      (Eg: unlink_thd logic will be protected with
      LOCK_thd_remove).
      
      Logic of mysqld_list_processes(or file_schema_processlist)
      will now be protected with 'LOCK_thd_remove' instead of
      'LOCK_thread_count'.
      
      Now the new locking order after this patch is:
      LOCK_thd_remove -> LOCK_thd_data -> LOCK_log ->
      LOCK_index -> LOCK_thread_count