Description
-----------
When there is a large amount of slaves to a master using the semi-sync plugin, a
degradation in performance is observed due to high contention on mutexes.

Analysis
--------
For each plugin event that is triggered, semi-sync callbacks will iterate over
the observers list, acquiring both `LOCK_plugin` and `Delegate::lock` in the
process.

As we add semi-sync slaves, the number of threads invoking the above callbacks
--the dump threads-- increases, generating higher contention over the said
locks.

Fix
---
We can relax the contention over the said locks by:

1) Since the plugin lock acquisition occurring within the semi-sync observer
   callbacks is already protected by a local rw-lock that protects changes in
   the semi-sync plugin itself, we can use the internal semi-sync plugin
   reference instead of looking for it in the plugin list. To prevent eventual
   and unforeseen behavior changes, a global variable
   `replication_optimize_for_static_plugin_config` is introduced. When enabled,
   will defer uninstall of transaction state change consumers by acquiring a
   hard-reference to the plugin and make the semi-sync observer callback
   processor to use the internal plugin reference instead of looking it up in
   the plugin list and, as a result, avoiding the plugin lock acquisition. When
   the option is disabled, the behavior observed prior to the patch is kept.

2) Since the addition and removal of observer plugins aren't, in most cases,
   frequently executed operations, we can replace the current `Delegate` class
   rw-locking with a more read-friendly lock. Therefore, we can replace the
   currently used read-write lock, with a shared-exclusive spin-lock that will
   speed up read access by requiring that acquiring threads check only an atomic
   flag, in the best case scenario. In the worst case, when an exclusive lock
   has been acquired, threads will be made to spin until such lock is
   released. This change in the type of lock is controlled by enabling the
   `replication_optimize_for_static_plugin_config` global variable.

3) For each transaction and binlog changes observer that is registered, a large
   list of callbacks is executed. In the semi-sync case, we only need to be
   notified upon commit. Therefore, an operation mode is added for which we only
   enable event callback on commit, relaxing the stress on the locks. This
   behavior is controlled by enabling the
   `replication_sender_observe_commit_only` global variable.

How to use it
-------------
Both `replication_optimize_for_static_plugin_config` and
`replication_sender_observe_commit_only` are dynamic global variables that can
be set after server startup.

Both can be enabled --set to `1`-- either before or after installing the plugins
and no need to stop replication for changing either of them.

In the case of `replication_optimize_for_static_plugin_config`, semi-sync plugin
will only be uninstalled when the option is disabled. If `UNINSTALL` of the
semi-sync plugin is executed while the
`replication_optimize_for_static_plugin_config` is enabled, the plugin will be
unloaded when the option is set to `0`.

For enabling the new behavior:

```
     mysql> SET GLOBAL replication_optimize_for_static_plugin_config = 1;
     mysql> SET GLOBAL replication_sender_observe_commit_only = 1;
```

For disabling the new behavior:

```
     mysql> SET GLOBAL replication_optimize_for_static_plugin_config = 0;
     mysql> SET GLOBAL replication_sender_observe_commit_only = 0;
```

The new behavior may be enabled both in the master and in the slaves, because
both `semi-master` and `semi-slave` plugins use the same infra-structure and
both benefit from the relaxed mutex access behavior.

Reviewed-by: Joro <georgi.kodinov@oracle.com>
Reviewed-by: Luís Soares <luis.soares@oracle.com>
Reviewed-by: Vitor Oliveira <vitor.s.p.oliveira@oracle.com>
RB: 23598