1. Establish a maintenance window (one hour should be sufficient). During this time the Kudu cluster will be unavailable.
2. Decide how many masters to use. The number of masters should be odd. Three or five node master configurations are recommended; they can tolerate one or two failures respectively.
3. Perform the following preparatory steps for the existing master:

   • Identify and record the directories where the master’s write-ahead log (WAL) and data live. If using Kudu system packages, their default locations are /var/lib/kudu/master, but they may be customized using the fs_wal_dir and fs_data_dirs configuration parameters. The command below assume that fs_wal_dir is /data/kudu/master/wal and fs_data_dirs is /data/kudu/master/data. Your configuration may differ. For more information on configuring these directories, see the Kudu Configuration docs.

   • Identify and record the port the master is using for RPCs. The default port value is 7051, but it may have been customized using the rpc_bind_addresses configuration parameter.

   • Identify the master’s UUID. It can be fetched using the following command:

     $ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null

     master_data_dir

     The location of the existing master’s previously recorded data directory.

     For example:

     $ sudo -u kudu kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
     4aab798a69e94fab8d77069edff28ce0

   • (Optional) Configure a DNS alias for the master. The alias could be a DNS cname (if the machine already has an A record in DNS), an A record (if the machine is only known by its IP address), or an alias in /etc/hosts. The alias should be an abstract representation of the master (e.g. master-1).

       Important: Without DNS aliases, it is not possible to recover from permanent master failures without bringing the cluster down for maintenance. t is highly recommended that you use DNS aliases.
4. If you have Kudu tables that are accessed from Impala, you must update the master addresses in the Apache Hive Metastore (HMS) database.
   • If you set up the DNS aliases, run the following statement in impala-shell, replacing master-1, master-2, and master-3 with your actual aliases.

     ALTER TABLE table_name
     SET TBLPROPERTIES
     ('kudu.master_addresses' = 'master-1,master-2,master-3');

   • If you do not have DNS aliases set up, see Step #11 in the Performing the migration section for updating HMS.
5. Perform the following preparatory steps for each new master:

   • Choose an unused machine in the cluster. The master generates very little load so it can be collocated with other data services or load-generating processes, though not with another Kudu master from the same configuration.

   • Ensure Kudu is installed on the machine, either using system packages (in which case the kudu and kudu-master packages should be installed), or some other means.

   • Choose and record the directory where the master’s data will live.

   • Choose and record the port the master should use for RPCs.

   • (Optional) Configure a DNS alias for the master (e.g. master-2, master-3, etc).

1. Stop all the Kudu processes in the entire cluster.
2. Format the data directory on each new master machine, and record the generated UUID. Use the following commands:

   $ sudo -u kudu kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
   $ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null

   master_data_dir

   The new master’s previously recorded data directory.

   For example:

   $ sudo -u kudu kudu fs format --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data
   $ sudo -u kudu kudu fs dump uuid --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 2>/dev/null
   f5624e05f40649b79a757629a69d061e

3. If you are using Cloudera Manager, add the new Kudu master roles now, but do not start them.

   • If using DNS aliases, override the empty value of the Master Address parameter for each role (including the existing master role) with that master’s alias.

   • Add the port number (separated by a colon) if using a non-default RPC port value.

4. Rewrite the master’s Raft configuration with the following command, executed on the existing master:

   $ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <all_masters>

   master_data_dir

   The existing master’s previously recorded data directory

   tablet_id

   This must be set to the string, 00000000000000000000000000000000.

   all_masters

   A space-separated list of masters, both new and existing. Each entry in the list must be a string of the form <uuid>:<hostname>:<port>.

   uuid

   The master’s previously recorded UUID.

   hostname

   The master’s previously recorded hostname or alias.

   port

   The master’s previously recorded RPC port number.

   For example:

   $ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:master-1:7051 f5624e05f40649b79a757629a69d061e:master-2:7051 988d8ac6530f426cbe180be5ba52033d:master-3:7051

     Important: If you are using Cloudera Manager, skip the next step.
5. Modify the value of the master_addresses configuration parameter for both existing master and new masters. The new value must be a comma-separated list of all of the masters. Each entry is a string of the form, <hostname>:<port>.

   hostname

   The master's previously recorded hostname or alias.

   port

   The master's previously recorded RPC port number.

6. Start the existing master.
7. Copy the master data to each new master with the following command, executed on each new master machine.
     Important: If your Kudu cluster is secure, in addition to running as the Kudu UNIX user, you must authenticate as the Kudu service user prior to running this command.

   $ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=<master_data_dir> <tablet_id> <existing_master>

   master_data_dir

   The new master's previously recorded data directory.

   tablet_id

   Must be set to the string, 00000000000000000000000000000000.

   existing_master

   RPC address of the existing master. It must be a string of the form <hostname>:<port>.

   hostname

   The existing master's previously recorded hostname or alias.

   port

   The existing master's previously recorded RPC port number.

   Example

   $ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data  00000000000000000000000000000000 master-1:7051

8. Start all the new masters.
     Important: If you are using Cloudera Manager, skip the next step.
9. Modify the value of the tserver_master_addrs configuration parameter for each tablet server. The new value must be a comma-separated list of masters where each entry is a string of the form <hostname>:<port>

   hostname

   The master's previously recorded hostname or alias

   port

   The master's previously recorded RPC port number

10. Start all the tablet servers.
11. If you have Kudu tables that are accessed from Impala and you didn’t set up DNS aliases, update the HMS database manually in the underlying database that provides the storage for HMS.
    • The following is an example SQL statement you would run in the HMS database:

      UPDATE TABLE_PARAMS
      SET PARAM_VALUE =
        'master-1.example.com,master-2.example.com,master-3.example.com'
      WHERE PARAM_KEY = 'kudu.master_addresses' AND PARAM_VALUE = 'old-master';

    • Invalidate the metadata by running the command in impala-shell:

      INVALIDATE METADATA;

1. Format the data directory on the replacement master machine using the previously recorded UUID of the dead master. Use the following command sequence:

   $ sudo -u kudu kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] --uuid=<uuid>

   master_data_dir

   The replacement master’s previously recorded data directory.

   uuid

   The dead master’s previously recorded UUID.

   For example:

   $ sudo -u kudu kudu fs format --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data --uuid=80a82c4b8a9f4c819bab744927ad765c

2. Copy the master data to the replacement master with the following command.
     Important: If your Kudu cluster is secure, in addition to running as the Kudu UNIX user, you must authenticate as the Kudu service user prior to running this command.

   $ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <reference_master>

   master_data_dir

   The replacement master’s previously recorded data directory.

   tablet_id

   Must be set to the string, 00000000000000000000000000000000.

   reference_master

   The RPC address of the reference master. It must be a string of the form <hostname>:<port>.

   hostname

   The reference master’s previously recorded hostname or alias.

   port

   The reference master’s previously recorded RPC port number.

   For example:

   $ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 master-2:7051

3. If you are using Cloudera Manager, add the replacement Kudu master role now, but do not start it.

   • Override the empty value of the Master Address parameter for the new role with the replacement master’s alias.

   • If you are using a non-default RPC port, add the port number (separated by a colon) as well.

4. If the cluster was set up with DNS aliases, reconfigure the DNS alias for the dead master to point at the replacement master.
5. If the cluster was set up without DNS aliases, perform the following steps:
   1. Stop the remaining live masters.
   2. Rewrite the Raft configurations on these masters to include the replacement master. See Step 4 of Perform the Migration for more details.
6. Start the replacement master.
7. Restart the remaining masters in the new multi-master deployment. While the masters are shut down, there will be an availability outage, but it should last only as long as it takes for the masters to come back up.

1. Establish a maintenance window (one hour should be sufficient). During this time the Kudu cluster will be unavailable.
2. Identify the UUID and RPC address current leader of the multi-master deployment by visiting the /masters page of any master’s web UI. This master must not be removed during this process; its removal may result in severe data loss.
3. Stop all the Kudu processes in the entire cluster.
4. If you are using Cloudera Manager, remove the unwanted Kudu master from your cluster's Kudu service.

1. Rewrite the Raft configuration on the remaining masters to include only the remaining masters. See Step 4 of Perform the Migration for more details.
2. Remove the data directories and WAL directory on the unwanted masters. This is a precaution to ensure that they cannot start up again and interfere with the new multi-master deployment.
3. Modify the value of the master_addresses configuration parameter for the masters of the new multi-master deployment. See Kudu Configuration docs for the steps to modify a configuration parameter. If migrating to a single-master deployment, the master_addresses flag should be omitted entirely.
4. Start all of the masters that were not removed.
     Important: If you are using Cloudera Manager, skip the next step.
5. Modify the value of the tserver_master_addrs configuration parameter for the tablet servers to remove any unwanted masters. See Kudu Configuration docs for the steps to modify a configuration parameter.
6. Start all of the tablet servers.

You can add data directories to an existing master or tablet server using the kudu fs update_dirs tool. Data is striped across data directories, and when a new data directory is added, new data will be striped across the union of the old and new directories.

  Note: Only new tablet replicas, i.e. brand new tablets' replicas and replicas that are copied to the server for high availability, will use the new directory. Existing tablet replicas on the server will not be rebalanced across the new directory.

  Attention: The first configured data directory on a server contains the metadata files for all tablets on that server. Kudu will not permit reordering of this "metadata directory". For example if a cluster is configured with /data/1 as the first entry in --fs_data_dirs, all further configurations must be formatted as /data/1,<new directories>.

  Attention: All of the command line steps below should be executed as the Kudu UNIX user, typically kudu.

1. Since the tool can only run while the server is offline, establish a maintenance window to update the server. The tool itself runs quickly, so this offline window should be brief. Only the server to update needs to be offline.

   However, if the server is offline for too long (see the follower_unavailable_considered_failed_sec flag), the tablet replicas on it may be evicted from their Raft groups. To avoid this, it may be desirable to bring the entire cluster offline while performing the update.

2. Run the tool with the desired directory configuration flags. For example, if a cluster was set up with --fs_wal_dir=/wals and--fs_data_dirs=/data/1,/data/2 and a new /data/3 is desired, run the command:

   $ sudo -u kudu kudu fs update_dirs --fs_wal_dir=/wals --fs_data_dirs=/data/1,/data/2,/data/3

3. Modify the values of the fs_wal_dir and fs_data_dirs flags for the updated sever. If using Cloud Manager, make sure to only update the configurations of the updated server rather than of the entire Kudu service.
4. Once the update_dirs tool completes, start the server process. When Kudu is installed using system packages, service is typically used:

   $ sudo service kudu-tserver start

Kudu does not allow removing directories or changing the write-ahead-log (WAL) or metadata directories. To start a server with such directory configuration changes, the WAL and data directories on the server must be deleted and rebuilt, destroying the copy of the data for each tablet replica hosted on the local server. Kudu will automatically re-replicate tablet replicas removed in this way, provided the replication factor is at least three and all other servers are online and healthy.

  Note: These steps use a tablet server as an example, but the steps are the same for Kudu master servers.