Cassandra as a Docker

We dockerize our in house micro services, as well as 3rd party tools we use.
That includes Apache Solr, Zookeeper, Redis and many more.

The recent datastore we implemented is Apache Cassandra, version 3.11,
and I would like to share some of the hacks we had until getting our Cassandra cluster dockerized and running.

The next configurations support a cluster where each node is running on a separate host. For running multiple Cassandra nodes on the same host, such as in development mode, you will need to modify the configurations a bit.

The configurations are located in cassandra.yaml file and will be modified as part of the docker start script (entry point).

• Set the required addresses:
• listen_address - The address that Cassandra process will bind to. That should be the docker internal IP.
• broadcast_address - The address that Cassandra will publish to other nodes for nodes inter connectivity. That should be the host external IP.
• broadcast_rpc_address - The address that nodes will publish to connected clients. That, as well as the broadcast_address, should be the external ip.

       

sed -i 's/listen_address: [0-9a-z].*/listen_address: '${INTERNAL_IP}'/'  conf/cassandra.yaml
sed -i 's/broadcast_address: [0-9a-z].*/broadcast_address: '${EXTERNAL_IP}'/'  conf/cassandra.yaml
sed -i 's/broadcast_rpc_address: [0-9a-z].*/broadcast_rpc_address: '${EXTERNAL_IP}'/'  conf/cassandra.yaml
       

• Docker running params 
• Data Volumes - map the storage location to a permanent storage:
• -v  /data/cassandra-0:/data
• Each node has to map its inner Cassandra ports for external connections. Cassandra nodes / Clients will look for that ports when accessing the broadcast_address of other nodes.
• I didn't find a way to change the default 7000 and 9042 ports
• The ports are:
• 7000 - Cassandra inter-node cluster communication.
• 9042 - Cassandra client port.
• 9160 - Cassandra client port (Thrift).
• The best would be to play with the ports mapping and see for yourself, which is mandatory and which is not.
• Bottom line - Add -P  -p 7000:7000 -p 9042:9042 to the docker run command.

• Seed list
• This is tricky. Seeds hosts are a list of given host that will be used by the node to discover and join the cluster.
• From Datastax docs:
• Do not make all nodes seed nodes
• Note: It is a best practice to have more than one seed node per datacenter.
• The seed node designation has no purpose other than bootstrapping the gossip process for new nodes joining the cluster.
• Please read more about Seeds mechanism: https://docs.datastax.com/en/cassandra/3.0/cassandra/architecture/archGossipAbout.html
• Docker challenges: 
• Our Cassandra cluster is in some way, a micro services cluster. Any docker can die and another will start on a different host.
• How can we avoid cases where a new node is accessing a seed node that is about to disappear,  and then, live alone in a separate cluster, while all other nodes were connecting different seeds?
• We can't set a final list of seeds ips. 
• Solution - 
• X Cassandra docker will be deployed with a "seed" tag.
• The "seed" tag is a hack that we implemented on top of our micro services management system. For example, Lables can be used with Kuberenetes, to set 3 Cassandra pods to start on a "seed" labeled node. All other Cassandra pods will start without a "seed" label. 
• The Seed list is dynamic. As a docker starts up, and before starting the Cassandra node process on it, it will look for X other nodes that were tagged with a "seed" label. I assume you have a service discovery tool, that will support that step.
• The retrieved seed labeled ips, will be set as the  "seeds" param for the starting node.
• We will keep looking for at least X seed nodes in our service discovery endpoint, and will not start Cassandra process until getting enough seeds.

sed -i 's/seeds: \"[0-9.].*\"/seeds: \"'${seeds_list}'\"/'  conf/cassandra.yaml  
  

• Now, we ensure that all nodes will use the same ips as a seeds list. There is a problem with loosing all seeds labeled nodes at the same time, and such a case would require a manual solution. 
• Other solutions might include selecting seed nodes according to custom rules that will ensure a consistent seed nodes selection by all nodes. For example - picking the 3 hosts that have the highest ip, when sorting all Cassandra service hosts ips (pods in Kuberenetes). 
• Setting password and username 
• We are willing to avoid any manual intervention. The first node would have to run a CQL script in order to change the default credentials. 
• Take that in account and execute that script after launching Cassandra process, and once the CQL terminal is ready.
• Adding new nodes - The docker start script should also take care of new nodes that are joining an existing cluster.
• They should remove any dead nodes (otherwise the ew node will fail joining) 
• I am sure that you will find some other scenarios when executing disruptive tests.
• Concerns and Todos - 
• Depends on you docker version, make sure you are satisfied with performances. It is known that IO can get slower with docker. 
• Configure a cross datacenters cluster, where each datacenter has a different service discovery endpoint.
• Run a cluster on the same node
• Check Docker network to see if it is helpful for that use case. 
• Publish our docker files.

To conduct, it is possible to run a Dockerized Cassandra cluster. Pay attention to the required params in the cassandra.yaml file, and add any custom behaviour, especially for the seeds selection, to the docker start script. 

In the next article I will present the way our Cassandra java client is accessing the cluster. 

Good luck!