Delete a dead node

To delete a dead node from Cassandra and DDB tables, do the following:

  1. Login to any of the other live nodes (referred as local node) and perform the following steps with administrative rights.
  2. Make sure that the local Cassandra service is in 'Up' and 'Normal' status.
    The remote dead Cassandra node cannot be in the 'Up' state. It must be 'Down' or its server needs to be shut down.
    The remote dead Cassandra node must not be started again after removing it.

    You can check the local and remote Cassandra node status by running the following command:

    $ "<Install folder>\Shared Services\Cassandra\bin\nodetool.bat" status

    The JAVA_HOME environment variable must be set to "<Install folder>\Shared Services\Cassandra\JRE" before running "nodetool status"
    If the dead node is visible in the output of “nodetool status” then take note of the value in the “Host Id” column. This is the cassandraHostId of the dead node.
  3. If the dead node is not visible in the output of “nodetool status” (or to double-check that the dead node was identified correctly in the previous step) then run the following command:
    $ NDISecTool.exe -getsfsnodes -u security_admuin_username -p security_admin_password -sfsurl https://<Hostname of the local node>:8181/SecurityFrameworkService
    Locate the record for the dead node in this output and take note of the ddbHostId or the cassandraHostId.
  4. Run either of these commands:
    $ "<Install folder>\SharedServices\SecurityFrameworkService\CassandraSetup\CassandraSetup.exe" removenode -chid <cassandraHostId of the dead node>
    Or:

    $ "<Install folder>\SharedServices\SecurityFrameworkService\CassandraSetup\CassandraSetup.exe" removenode -dhid <ddbHostId of the dead node>

    This will perform the following operations automatically:

    • Runs nodetool removenode <Cassandra Host Id of the remote node>
    • Deletes keyspace secrets of remote node from controlsuite_ddb.ddb_key_by_public_key
    • Deletes remote node record from controlsuite_ddb.ddbnode_by_host_id and controlsuite_ddb.ddbnode_by_datacenter_id
    • If needed it deletes the empty datacenter from controlsuite_configuration.datacenter_by_id
    • Updates keyspaces replication factors and run repair