Metrics Visualization With Collectd And Grafana

4 min readMay 6, 2019

Server is your application’s brain. It manages the most critical aspects of your app’s (or website’s) activities. With this in mind, there’s no doubt you should closely watch your server and react to anomalies as they appear. So, it is really important to have a monitoring system to visualize time series data. Time series data comes in many forms but one of the more common use cases is visualizing metrics. More specifically, you might be visualizing critical application/server information.

In fact, a perfect example of this use case in action is Grafana. According to the documentation, Grafana provides a powerful and elegant way to create, export and share dashboards and data with your team and the world. In this post, we are going to use Collectd, a tool for gathering Infrastructure metrics. Collectd gather metrics about your server and can send these data to a large number of outputs. We use graphite as our Collectd’s output plugin. This way, we are able to ship our metrics to Cyanite, a daemon which provides services to store and retrieve time series data. Using it, we are going to store time series data in Apache Cassandra database. The next step is get these metrics using graphite-api and visualize them using Grafana.

Environment

In this post, we use :

Ubuntu 16.04
Collectd 5.5.1
Grafana 4.0.2
Cyanite 0.5.1
Graphite-Api 1.1.2
Apache Cassandra 3.9

Install Cassandra

Download Cassandra, decompress it into your home directory, and launch it.

$ wget http://www-eu.apache.org/dist/cassandra/3.9/apache-cassandra-3.9-bin.tar.gz$ tar –xzvf apache-cassandra-3.9-bin.tar.gz$ cd apache-cassandra-3.9 && bin/cassandra

Install And Configure Cyanite

Clone the Cyanite Git repository in your home folder and import the database initial script.

$ git clone https://github.com/pyr/cyanite.git$ cd cyanite$ /home/user/apache-cassandra-3.9/bin/cqlsh –f doc/schema.cql

Now Install Leiningen. This is the build system tool used by the Cyanite project. Its very friendly seeming and installs locally into your home directory. This allows you to build JARs and other distributable versions of the code.

$ wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein$ sudo mv lein /usr/bin/$ sudo chmod a+x /usr/bin/lein$ lein

We need to distribute code as Debian packages for Ubuntu. Fortunately, we have a target to build just that.

$ cd /home/user/cyanite$ lein fatdeb

This produces artifacts in the target/ directory.

$ cd /home/user/cyanite/target$ sudo dpkg –i cyanite_0.5.1_all.deb

The cyanite daemon runs now at port 8080, and you need to configure it before using it correctly. If you are not familiar with cyanite configuration, it can be a bit challenging to understand everything. Fortunately, I already configured my own file, so you can download the it: cyanite.yaml, and put it under /etc folder. You can also change the configuration if needed.

Install And Configure Graphite-api

Download and install graphite-api:

$ wget https://github.com/brutasse/graphite-api/releases/download/1.1.2/graphite-api_1.1.2-1447943655-debian8_amd64.deb$ sudo dpkg –i graphite-api_1.1.2-1447943655-debian8_amd64.deb

Once installed, Graphite-api runs as a service on port 8888.

Now download this file: graphite-api.yaml, and put it under /etc folder, you can change the configuration if needed, and restart all your services.

$ sudo service graphite-api restart$ sudo service cyanite restart

Install and Configure Collectd

Now we need to Download, install and configure Collectd on our host.

Download and install Collectd.

$ sudo apt-get update$ sudo apt-get install collectd collectd-utils

Download this file: collectd.conf, and put it under /etc/collectd folder. Now restart collectd.

$ sudo service collectd restart

Our data are fed to Cassandra database, we can start to visualize them.

Install Grafana and visualize data

To install Grafana, just download it and decompress the deb file, then start the service. It runs on port 3000.

$ wget https://grafanarel.s3.amazonaws.com/builds/grafana_4.0.2-1481203731_amd64.deb$ sudo dpkg –i grafana_4.0.2-1481203731_amd64.deb$ sudo service grafana restart

Now go to http://localhost:3000
Login with the following credentials if you did not change them. username: admin, password: admin (default credentials)
Create a data source

To be able to receive data from Collectd, we need to connect our data source to graphite-api, which is running under port 8888. So, in the data source configuration put this as url: http://localhost:8888.

Now, create a new dashboard

Then, Choose a graph

Add a single metric to see how it works. In the example below, we can can monitor the CPU memory usage in our server.

You can also add many metrics as necessary for you monitoring.

Now you have an overview of how you can monitor your sever’s critical resources using open source tools. It enables you to take critical decisions and, using big data potential, gives you great opportunities.