Apache Zeppelin as a Web Querying Interface for Cassandra

Hi!

An important thing that you are missing when starting to use Cassandra is a decent web tool
for executing CQL queries, and browsing the results.

Googling for a "Cassandra web ui" results in RazorSql, and some other not so promising engines, where most of them are not a web UI at all.


As you can tell, by the post's title, Apache Zeppelin turned out to be a great solution for our need.
It has a built in Cassandra Interpreter, which only requires basic connection properties.

Here is a video explaining all about Zeppelin and Cassandra 

We did had one issue with installing Zeppelin on our VPC, and trying to access it via Nginx and AWS load balancer:

• Apache Zeppelin is using a web socket for a client-server connectivity. 
• It requires special configuration on Nginx: how to configure Apache Zeppelin on Nginx
• It enforced us to use Application Load Balancer, instead of Classic Load Balancer, which doesn't support web sockets out of the box. 

As a result, we no longer have to ssh Cassandra nodes, in order to execute CQL queries.

I hope that everyone who is looking for a CQL web ui, would find his answer here (or a better answer somewhere else :) ).

Cassandra as a Docker

We dockerize our in house micro services, as well as 3rd party tools we use.
That includes Apache Solr, Zookeeper, Redis and many more.

The recent datastore we implemented is Apache Cassandra, version 3.11,
and I would like to share some of the hacks we had until getting our Cassandra cluster dockerized and running.

The next configurations support a cluster where each node is running on a separate host. For running multiple Cassandra nodes on the same host, such as in development mode, you will need to modify the configurations a bit.

The configurations are located in cassandra.yaml file and will be modified as part of the docker start script (entry point).

• Set the required addresses:
• listen_address - The address that Cassandra process will bind to. That should be the docker internal IP.
• broadcast_address - The address that Cassandra will publish to other nodes for nodes inter connectivity. That should be the host external IP.
• broadcast_rpc_address - The address that nodes will publish to connected clients. That, as well as the broadcast_address, should be the external ip.

       

sed -i 's/listen_address: [0-9a-z].*/listen_address: '${INTERNAL_IP}'/'  conf/cassandra.yaml
sed -i 's/broadcast_address: [0-9a-z].*/broadcast_address: '${EXTERNAL_IP}'/'  conf/cassandra.yaml
sed -i 's/broadcast_rpc_address: [0-9a-z].*/broadcast_rpc_address: '${EXTERNAL_IP}'/'  conf/cassandra.yaml
       

• Docker running params 
• Data Volumes - map the storage location to a permanent storage:
• -v  /data/cassandra-0:/data
• Each node has to map its inner Cassandra ports for external connections. Cassandra nodes / Clients will look for that ports when accessing the broadcast_address of other nodes.
• I didn't find a way to change the default 7000 and 9042 ports
• The ports are:
• 7000 - Cassandra inter-node cluster communication.
• 9042 - Cassandra client port.
• 9160 - Cassandra client port (Thrift).
• The best would be to play with the ports mapping and see for yourself, which is mandatory and which is not.
• Bottom line - Add -P  -p 7000:7000 -p 9042:9042 to the docker run command.

• Seed list
• This is tricky. Seeds hosts are a list of given host that will be used by the node to discover and join the cluster.
• From Datastax docs:
• Do not make all nodes seed nodes
• Note: It is a best practice to have more than one seed node per datacenter.
• The seed node designation has no purpose other than bootstrapping the gossip process for new nodes joining the cluster.
• Please read more about Seeds mechanism: https://docs.datastax.com/en/cassandra/3.0/cassandra/architecture/archGossipAbout.html
• Docker challenges: 
• Our Cassandra cluster is in some way, a micro services cluster. Any docker can die and another will start on a different host.
• How can we avoid cases where a new node is accessing a seed node that is about to disappear,  and then, live alone in a separate cluster, while all other nodes were connecting different seeds?
• We can't set a final list of seeds ips. 
• Solution - 
• X Cassandra docker will be deployed with a "seed" tag.
• The "seed" tag is a hack that we implemented on top of our micro services management system. For example, Lables can be used with Kuberenetes, to set 3 Cassandra pods to start on a "seed" labeled node. All other Cassandra pods will start without a "seed" label. 
• The Seed list is dynamic. As a docker starts up, and before starting the Cassandra node process on it, it will look for X other nodes that were tagged with a "seed" label. I assume you have a service discovery tool, that will support that step.
• The retrieved seed labeled ips, will be set as the  "seeds" param for the starting node.
• We will keep looking for at least X seed nodes in our service discovery endpoint, and will not start Cassandra process until getting enough seeds.

sed -i 's/seeds: \"[0-9.].*\"/seeds: \"'${seeds_list}'\"/'  conf/cassandra.yaml  
  

• Now, we ensure that all nodes will use the same ips as a seeds list. There is a problem with loosing all seeds labeled nodes at the same time, and such a case would require a manual solution. 
• Other solutions might include selecting seed nodes according to custom rules that will ensure a consistent seed nodes selection by all nodes. For example - picking the 3 hosts that have the highest ip, when sorting all Cassandra service hosts ips (pods in Kuberenetes). 
• Setting password and username 
• We are willing to avoid any manual intervention. The first node would have to run a CQL script in order to change the default credentials. 
• Take that in account and execute that script after launching Cassandra process, and once the CQL terminal is ready.
• Adding new nodes - The docker start script should also take care of new nodes that are joining an existing cluster.
• They should remove any dead nodes (otherwise the ew node will fail joining) 
• I am sure that you will find some other scenarios when executing disruptive tests.
• Concerns and Todos - 
• Depends on you docker version, make sure you are satisfied with performances. It is known that IO can get slower with docker. 
• Configure a cross datacenters cluster, where each datacenter has a different service discovery endpoint.
• Run a cluster on the same node
• Check Docker network to see if it is helpful for that use case. 
• Publish our docker files.

To conduct, it is possible to run a Dockerized Cassandra cluster. Pay attention to the required params in the cassandra.yaml file, and add any custom behaviour, especially for the seeds selection, to the docker start script. 

In the next article I will present the way our Cassandra java client is accessing the cluster. 

Good luck!

Apache Solr Cloud Sanity - 3 simple sanity tests

Having a clear picture of your services status is a fundamental requirement.

Otherwise you are blind to the system. 

I am going to present a general concept of sanity checks and some more specific Solr database sanity checks. 

There are many tools and methods to monitor a running service, whether it is a micro service, a 3rd party database or any other piece of code.

We use Datadog, and Grafana to have the system fully covered and to dive into relevant metrics when needed.

In addition to these too well known tools,  we implement, for each in-house service or a 3rd party tool, a custom sanity test, that in a failure we know we have a problem.

The sanity tests result are being collected by a centralised monitoring service and they are all being displayed in a monitoring dashboard.  

The dashboard has many features, as have been probably thinking of by yourself, such as deep dive into sanity test logs, get history of sanity tests and so on.

Let's take apache Solr as an example. 

In addition to our Solr docker instance, we deploy a sanity script, for each solr instance. 

We defined 3 sanity tests - They are all based on Solr cluster status output:

1. Replica status - If a replica is down or recovering, we mark the hosting solr instance of that replica as problematic. 
2. Balanced leaders within Collection - If a solr server is hosting more than 1 leader of shards from the same collection, it is also a problem (it might overload that host).
3. Not too many leaders per host - If a Solr server is hosting more than X leaders, we are marking the host as problematic. 

The test is running every M minutes (15 for Solr test).

To conclude, 

a custom sanity test is a simple and an efficient way to be aware of a service status. Our monitoring procedure starts from the centralised sanity dashboard, and only then goes into Datadog or Grafana.