13.9 Cluster Operations

Perform Cluster operations with a verified runbook after checking node configuration, roles, replication, and data state. Use the checks on this page to assess change impact and recovery paths, then include commands from the installed management tool version in the runbook.

Components

Design node counts and placement around availability, throughput, and failure domain requirements. Do not apply a fixed node count or hardware specification to every environment.

Check status

Before and after a change, check the full node configuration from the Coordinator and each node’s processes and resources. Use the installed tool’s help for status strings and options.

machcoordinatoradmin --help
machclusterctl --help

• Are all expected nodes registered?
• Do node roles, hosts, ports, and groups match deployment records?
• Are service, replication, and scrap states normal?
• Are CPU, memory, disk, or network resources unevenly used across nodes?
• Do actual connections and queries through the Broker succeed?

Connect and export configuration

When using machclusterctl connect, specify the target Broker and native port, then verify CURRENT_DATABASE() and a sample query. Configuration exports may contain hosts, ports, paths, and operational information. Restrict access and review the diff before importing.

Start and stop nodes

Before controlling a node, check:

1. Target node name, alias, host, and role.
2. Client connections and active queries and Appenders.
3. Warehouse group redundancy and data state.
4. Remaining capacity while the node is stopped.
5. Startup and shutdown order and rollback steps.
6. Acceptance criteria after maintenance.

Use a forced shutdown only after normal shutdown repeatedly fails and you have assessed its data and recovery impact. Do not immediately kill a process on a timeout.

Control the entire Cluster and use destroy

For full startup and shutdown, check the dependency order of Coordinator, Deployer, Broker, Warehouse, and Lookup in the runbook for the current release. destroy is destructive and may remove node configuration and data.

• Confirm that the target is not another cluster with a similar name.
• Verify a recent backup and a successful restore.
• Obtain service owner approval and block clients.
• Check the deletion scope, including external DBS_PATH locations.
• Identify changes that cannot be rolled back.
• After execution, check each host for remaining processes and paths.

Do not use destroy for routine state recovery.

Add and remove nodes

Before adding a node, check the package, version, ports, paths, file systems, and network. Before removal, verify data redundancy and completed migration, along with aliases, groups, and monitoring that reference the node. Removing a node may delete its home and data paths; check the command’s help and validate the exact scope in a test environment.

Change state

Disabling a Broker, making a Warehouse group read-only, and scrapping a node serve different purposes. Do not change state arbitrarily to hide a failing node.

Recover a Warehouse

1. Preserve the failure time and first error.
2. Check processes, disks, network, and replication state.
3. Assess redundancy on remaining nodes and the service impact.
4. Choose a supported recovery path: restart, reattach, or rebuild.
5. Monitor progress and errors.
6. After recovery, compare row counts, time ranges, and query results across nodes.

Do not force a node to normal state without checking for data corruption.

Constraints and checklist

For edition-specific support for SQL, ROLLUP, backups, and ALTER SYSTEM, see Support Scope.

• Are all node and client SDK releases compatible?
• Are permitted reads and writes during maintenance defined?
• Have backup, restore, and node recovery drills been completed?
• Have two operators cross-checked hosts, ports, and paths?
• Do monitoring and alerts reflect the new node configuration?
• Have Broker connections, queries, Append, and metadata been verified after the change?