A. Overview:
------------
This big patch introduces data dictionary (DD) schema in MySQL
server. Some of the main benefits of this is,

* The .FRM files are gone, and all dictionary changes will
  now happen in the dictionary schema stored in InnoDB.

* Builds a base platform enabling us to improve performance of
  INFORMATION_SCHEMA queries, which can be implemented as a view.

* This patch enables a forthcoming incremental patch that would
  remove internal dictionary table of InnoDB. Enables,
   - There by eliminate inconsistencies between .FRM files and
     InnoDB tables.
   - Remove file system dependencies (lower-case-table-names).
   - Single repository of meta data for Server, SE and Plugins.
   - Eliminate meta data redundancy.

* Remove db.opt and rely on dictionary tables.

B. Work logs:
-------------
The WL#6378 is the umbrella WL tracking all dependent WL's. This
patch contains implementation of following dependent WL's,

* Worklogs that deal with just DD API framework:

  WL#6379 - Schema definitions for new DD.
            This important WL defines the central data dictionary
            table definitions.

  WL#6380 - Formulate framework for API for DD.

  WL#7284 - Implement common code for different DD APIs.
  WL#6385 - Define and Implement API for Schema.
  WL#6382 - Define and Implement API for Table objects.
  WL#6389 - Implementation of common View API.
  WL#6387 - Define and Implement API for Tablespaces.
  WL#7630 - Define and Implement API for Table Partition Info
  WL#8150 - Dictionary object cache.
  WL#7770 - Develop GUNIT test framework and guidelines for DD API.
  WL#7771 - Make sure errors are properly handled in DD API.

* Worklogs that change server code invoking DD API's:

  WL#6390 - Use new DD API for handling non-partitioned tables.
  WL#7836 - Use new DD API for handling partitioned tables.
  WL#6394 - Bootstrapping the new data dictionary.
  WL#7784 - Store temporary table meta data in memory.
  WL#8542 - Server shutdown procedure with new data dictionary
  WL#7593 - New data dictionary: don't hold LOCK_open while
            reading table definition.
  WL#7464 - InnoDB: provide a way to do non-locking reads
            This is pre-requisite for WL#6599.

C. Tips to DD API users:
------------------------
* Refer code in sql/dd/dd_schema.* to quickly learn how to write
  code using DD API and update dictionary tables.

  Refer sql/dd/cache/dictionary_client.h to get overview on what
  is Dictionary_client and Auto_release interface.

  Interested can refer to sql/dd_table_share.{h,cc} which
  implements mapping between the TABLE_SHARE and the dd::Table DD
  object.

* Overview of the directory structure and source files introduced
  by this patch is as following,

  DD user exposed files:-

  sql/dd/        - Top level directory containing most of DD code.
  sql/dd/*.h     - Headers that you need to operate on DD.
  sql/dd/types/  - Contains headers to operate on single
                   dictionary object.
  sql/dd/dd_*.cc - Contains MySQL server and DD API framework
                   glue code. E.g., Code that updates to DD tables
                   upon table creation/alter table/drop table and
                   so on.
  sql/dd/cache/  - Contains implementation of DD Object cache.

  Implementation that is hidden to DD user:-

  sql/dd/impl/   - Contains implementation of DD API's.
  sql/dd/impl/cache/  - Contains implementation of DD object cache
                        operations.
                         E.g., dd::cache::Dictionary_client
  sql/dd/impl/types/  - Contains implementation of DD user object
                        operations. E.g., dd:Table, dd::View etc.
  sql/dd/impl/tables/ - Contains implementation of DD table
                        operations. E.g., dd::tables::* classes
                        that abstracts operations on a DD table.
  sql/dd/impl/raw/    - Contains implementation of generic
                        operations on all DD tables.

* The code related to .FRM handle is removed. So, the following
  files are removed,
    sql/datadict.h
    sql/datadict.cc
    sql/discover.h
    sql/discover.cc
    sql/unireg.h
    sql/unireg.cc

D. New configuration variables and startup options:
---------------------------------------------------
* Introduced a new option "--install-server", behaving like
  "--bootstrap", but additionally creating the dictionary tables.
  Note that the --install-server is a temporary workaround until
  MTR use --initialize.

* The existing "--bootstrap" option behaves like before as far
  as possible, but note that the changes in plugin initialization
  means that e.g. InnoDB must be able to start, the DD must be
  able to start, etc., for bootstrap to enter the stage where SQL
  commands are actually executed.

* The mysql_install_db and the mysql_test_run scripts are modified
  to use the new "--install-server" option.

* The cmake script create_initial_db.cmake.in is also changed to
  use --install-server rather than --bootstrap.

* WL#8150 adds server options schema_definition_cache,
  tablespace_definition_cache and stored_program_definition_cache
  size. These define the DD object cache size for respective DD
  objects.

E. Upcoming improvements
-----------------------
* WL#6599 New Data Dictionary and I_S integration
  Information schema queries will be execute faster as this WL
  would make information schema tables as a view on top of DD
  tables.

* WL#7743 New data dictionary: changes to DDL-related parts of SE API
  - Support atomic/crash-safe DDL.
  - Support auxiliary columns (InnoDB-specific)
  - Support auxiliary tables (needed for InnoDB FTS)

* WL#6394 - Bootstrapping the new data dictionary.
  - Improvements in bootstrap procedure supporting removal of
    InnoDB dictionary.

* WL#7896 Use DD API to work with triggers.
  - Will remove use of .TRN and .TRG files and start using
    mysql.triggers DD table.

* Additional system tables are to be moved from MyISAM
  WL#7897 Use DD API to work with stored routines
  - Will remove mysql.proc MyISAM table and use mysql.routines
    InnoDB dictionary table.

  WL#7898 Use DD API to work with events.
  - Will remove mysql.event MyISAM table and use mysql.events
    InnoDB dictionary table.

* WL#6391: Hide DD system tables from user.
  - Will hide direct access to DD system tables from user.

F. Permanent changes in behavior.
---------------------------------------------
* Since the .frm files are gone, you will not be able to copy
  .FRMs/data files and move them around. We will address this
  when we finish the implementation of new import, based on
  serialized dictionary information, see WL#7069.

* Related to the above is the fact that --innodb-read-only
  option does not currently work. In 5.7, meta-data would be
  written to .FRM files and not be affected by the option, but
  now as meta-data is written to InnoDB tables, we must
  reconsider the semantics of this option.

* The setting of the system variable 'lower_case_table_names'
  determines the collation of e.g. the 'name' column in the table
  storing the meta data for tables. This collation is decided once
  and for all when the dictionary table is created. Thus, we do
  not support changing 'lower_case_table_names' afterwards.

* Character sets and collation tables are populated at initial
  boot, and for every subsequent startup, unless the server or
  the dictionary storage engine (InnoDB) is started in read
  only mode.

G. Intermediate limitations in functionality:
---------------------------------------------
* Tests on upgrade/downgrade will not work, and will be
  temporarily disabled. QA will also have to temporarily postpone
  any upgrade tests, until we have completed the upgrade tool
  (WL#6392) from 5.7 to 5.8. We are working towards building a
  intermediate solution for system QA.

* Some InnoDB tests related to recovery of TRUNCATE are
  disabled. But n-test can be run as long as you are not trying to
  recover TRUNCATE. The tests will be enabled when we push the
  bundle of WLs(7141/7016/7743), which will move the storing of
  InnoDB DD to the common data dictionary, and make DDL atomic.
  Until which recovery after killing the server will also not
  work.

* MySQL Cluster cannot be used with this version of the server.

H. Disabled tests:
------------------
Apart from test that are disabled due to above functionality that
does not work now. Additionally we have temporarily disabled a
few other tests, which might be a bit challenging for bug fixing
in trunk in very restricted areas of the code. Refer
sql/dd/mtr_readme.txt to see which exact test cases are disabled.