data Archives | Clever Cloud

Enabling PromQL queries with Erlenmeyer

Aurélien Hebert — Tue, 12 Oct 2021 07:28:09 +0000

You are using Clever Cloud to deploy applications runtimes or add-ons. But do you know that all Clever Cloud services are producing metrics? And that you already have access to all of them? Indeed, we register an agent on our side, whose job is to collect all local data.Once it's done, they are stored in a Time Series database called Warp10.

To simplify the adoption of Warp10, the OVHcloud Metrics team built the Erlenmeyer tool. In the first place, this tool enables the use of several open source protocols as PromQL, OpenTSDB, InfluxQL or Graphite to query Warp10 stored metrics.

Secondly, Erlenmeyer works as a proxy in front of a Warp10 cluster. Then we deployed a couple of Erlenmeyer instances in front of our Warp10 infrastructure, naturally providing new ways to query your application metrics.

In this post, we will review what Erlenmeyer brings and how to handle Clever Cloud services metrics with PromQL.

Erlenmeyer protocols

Erlenmeyer almost entirely enables PromQL queries, OpenTSDB, InfluxQL and some of the Graphite functions. Moreover, you can retrieve the documentation about each protocol implementation and usage on Erlenmeyer's GitHub:

On Clever Cloud, we deployed an Erlenmeyer in front of our Warp10 backend. This means all these open source protocols can be used to query our application Metrics. The host is https://PROTO-c1-warp10-clevercloud-customers.services.clever-cloud.com/PROTO with PROTO being:

for PromQL: prometheus,
for OpenTSDB: opentsdb,
for InfluxQL: influxql,
for Graphite: graphite.

To authenticate your request, you need to use the Warp10 READ TOKEN as the password of the basic authentication. You can find it in the metrics panel of any Clever Cloud application:

Retrieve the memory data with PromQL

As a first step, let's store the Warp10 READ TOKEN and an APP_ID as an environment variable:

export TOKEN=MY_AWESOME_TOKEN # Replace the value by your Clever Cloud Warp10 token
export APP_ID=app_test # Replace the value by one of your own Clever Cloud application

Get raw data

To retrieve the mem.used_percent in PromQL, we need to send an HTTP GET request to Erlenmeyer, with the TOKEN as basic AUTH. In that case, the PromQL GET request requires four parameters: query, start, end and step. Similarly, to learn more about the PromQL language, you can read the Prometheus documentation. In our case we will create a simple query to get the raw data:

api/v1/query_range?
query=mem.used_percent{app_id="$APP_ID"}&
start=1623289800&
end=1623311400&
step=120s

The query parameter is used for the PromQL queries: start and end are milliseconds timestamps corresponding to the time limit to get data. And the step one is the sampling parameter.

As data are collected around every minute, let's set at least twice the duration: 120s.

Now, to get the result, execute the following cURL url encoded:

curl --request GET --url "https://u:${TOKEN}@prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/api/v1/query_range?query=mem.used_percent%7Bapp_id%3D%22${APP_ID}%22%7D&start=1623289800&end=1623311400&step=120s" --header 'Content-Type: application/json'

Group by example

A group by query can be useful in the Clever Cloud case to merge some series (multiple deployments or instances). Thus we will group data based on the app_id labels in the following example.

We will re-use the stored variable READ_TOKEN and APP_ID. To compute a group by in our case we apply the max PromQL function grouped by the series app_id label:

api/v1/query_range?
query=max(mem.used_percent{app_id="$APP_ID"}) by (app_id)&
start=1623289800&
end=1623311400&
step=120s

Finally to get the result, execute the following cURL url encoded:

curl --request GET --url "https://u:${TOKEN}@prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/api/v1/query_range?query=max(mem.used_percent%7Bapp_id%3D%22${APP_ID}%22%7D)%20by%20(app_id)&start=1623289800&end=1623311400&step=120s" --header 'Content-Type: application/json'

Grafana PromQL data source

You may use Grafana to create you own dashboards, which is an open source interactive visualisation web application (to deploy a Grafana as a Clever Cloud application, we provide a github example and you can also follow this French blog post). Of course you can configure a PromQL data source to get Clever Cloud applications metrics:

create a Prometheus source,
then set Prometheus source host to https://prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus,
in addition activate the basic authentication with a user set to metrics and a password containing your read Warp10 token,
to conclude, click on Save and Test, your Prometheus data source should now be working!

Prometheus UI with remote_read endpoint

Like with Grafana, you can use Prometheus and plot Clever Cloud metrics in the Web UI. And as a matter of fact, simply fill your Prometheus configuration with a remote_read source:

api/v1/query_range?
remote_read:
  - url: "https://prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/remote_read/"
  basic_auth:
    username: "metrics"
    password: "TOKEN"

Restart Prometheus, then simply execute a Prometheus query in the UI as below:

{ __name__= "mem.used_percent", app_id="$APP_ID" }

Our backend is able to correctly resolve . which are invalid prometheus characters. Nonetheless the backend response will always translate those . values in series data (name or labels) into valid _.

To Sum Up

As you can see with this post, you can get metrics of Clever Cloud applications and add-ons. Presently you now have one more way to query them, as Erlenmeyer allows the use of PromQL. However the drawback is that you will have to update regularly your token, as they expire in a few days. We are working on a solution to improve the user experience. First, we will soon provide our customers with access to a Grafana with configured data source and already pre-filled dashboards. The token will be handled on our side and you will then be able to build your own graphs with PromQL or WarpScript queries.

How to deploy Apache Superset on Clever Cloud

Aurélien Hebert — Tue, 04 May 2021 09:15:00 +0000

At Clever Cloud we manage most of our own data and when we want to gather a particular information, we open our SQL interpreter and query all the things manually. This somehow worked because most of us are technical but it's not necessary the case anymore. So we want a nice dashboarding solution to make data available in a nicer way. This is how we came to try Superset. What is Superset? In their own words:

Superset is fast, lightweight, intuitive, and loaded with options that make it easy for users of all skill sets to explore and visualize their data.

You can configure different sort of visualizations from simple line charts to highly detailed geo-spatial charts and organize them in dashboard. Take a look at their documentation to grasp the full extent of what you can do.

How to deploy Superset

Superset is written in Python and requires a PostgreSQL database. In Clever Cloud terms it means you will need to create a brand new Python runtime with a PostgreSQL addon.

# Clone last superset release
git clone --depth 1 -b 0.38.1  https://github.com/apache/superset.git

# Move in superset repository
cd superset

# Create the python application
clever create --type python Superset

# Create the PostgreSQL instance                                
clever addon create postgresql-addon --plan m SupersetPG
    
# link the addon 
clever service link-addon SupersetPG

Once the application is created, edit it's information. As you need to build Superset backend and frontend, you need to enable a dedicated build instance, select the L one. Don't foget to save the update.

Superset environment

Set the following environment, YOUR_APP_ID corresponds to your own application id, A_SECRET_KEY is a random secret key you choose, and AN_ADMIN_PASSWORD will be the admin password for the superset application:

# Use gunicorn as python mode 
clever env set CC_PYTHON_BACKEND gunicorn

# Start the app using create_app() 
clever env set CC_PYTHON_MODULE superset.app:create_app()

# Python version to use
clever env set CC_PYTHON_VERSION 3.7

# Load all superset requirements
clever env set CC_PIP_REQUIREMENTS_FILE requirements/base.txt

# Application port
clever env set PORT 8080

# PYTHONPATH
clever env set PYTHONPATH /home/bas/YOUR_APP_ID/config/

# App secret key
clever env set SECRET_KEY A_SECRET_KEY

# Post build commands
clever env set CC_POST_BUILD_HOOK ./init.sh

# Superset admin password
clever env set ADMIN_PASSWORD AN_ADMIN_PASSWORD

Configure superset

Superset need a local config python file. Wrote locally a clever_config.py:

import os
# Superset specific config
ROW_LIMIT = 5000

SUPERSET_WEBSERVER_PORT = os.getenv("PORT")
SUPERSET_WEBSERVER_ADDRESS = "0.0.0.0"

# Flask App Builder configuration
# Your App secret key
SECRET_KEY = os.getenv("SECRET_KEY")

# The SQLAlchemy connection string to your database backend
# This connection defines the path to the database that stores your
# superset metadata (slices, connections, tables, dashboards, ...).
# Note that the connection information to connect to the datasources
# you want to explore are managed directly in the web UI
SQLALCHEMY_DATABASE_URI = os.getenv("POSTGRESQL_ADDON_URI")

# Flask-WTF flag for CSRF
WTF_CSRF_ENABLED = True
# Add endpoints that need to be exempt from CSRF protection
WTF_CSRF_EXEMPT_LIST = []
# A CSRF token that expires in 1 year
WTF_CSRF_TIME_LIMIT = 60 * 60 * 24 * 365

# Set this API key to enable Mapbox visualizations
MAPBOX_API_KEY = ''

Superset build scipt

The next step is to write an init.shfile that will include all build tasks for superset:

#!/bin/bash

# Load configuration file in PYTHONPATH
mkdir $APP_HOME/config
cp clever_config.py $APP_HOME/config/superset_config.py

# Create an admin user (you will be prompted to set username, first and last name before setting a password)
superset fab create-admin \
    --username admin \
    --firstname admin \
    --lastname admin \
    --email admin@admin.com \
    --password ${ADMIN_PASSWORD}

# Upgrade the database
superset db upgrade

# Loading examples data set (optional)
superset load_examples

# Create default roles and permissions
superset init

# Build frontend
cd superset-frontend 
npm install -f --no-optional
npm run build

Clever supports post build hooks, this init.sh file will be played at the end of the build phase. It's used to set the config file we wrote before into PYTHONPATH used by the application. Then it will start some superset commands required for the first start of the application (create the admin user if it doesn't exists, upgrade the database, and file it with examples and finally init the superset appication). Those superset commands could be remove of the init file after the first success build. Finally, this script is used to also build the frontend of the superset application.

Superset requirements

When you added the CC_PIP_REQUIREMENTS_FILE, it told clever cloud to load custom requirement for superset which are locate in the requirements/base.txt file. As you are using the PostgreSQL add-on, you need to add its requirement in this base file:

echo "psycopg2>=2.7 --no-binary psycopg2" >> requirements/base.txt

The Clever Cloud python application still needs a local requirements.txt to start the install. Generate it with:

pip3 freeze > requirements.txt

As we use gunicorn and the PostgreSQL add-on, they are required in the requirements.txt file, add them:

echo "psycopg2>=2.7 --no-binary psycopg2" >> requirements.txt

# gunicorn valid versions are located in setup.py file
echo "gunicorn>=20.0.2, <20.1" >> requirements.txt

Deploy

Deploy the superset application into Clever cloud:

chmod u+x init.sh

# Add your files
git add .

# Create the first commit
git commit -m "clever init"

# Deploy the application
clever deploy

# Run the application
clever open

Once this is done you should be able to access the superset application, enjoy! After the first successful superset deployement you can remove on the init.sh file all commands starting with superset (they are required for the first start).

How to deploy Metabase to make sense of your data

Laurent Doguin — Wed, 20 Feb 2019 17:28:00 +0000

At Clever Cloud we manage most of our own data and when we want to gather a particular information, we open our SQL interpreter and query all the things manually. This somehow worked because most of us are technical but it's not necessary the case anymore. So we want a nice dashboarding solution to make data available in a nicer way. This is how we came upon Metabase. What is Metabase? In their own words:

The fastest, easiest way to share data and analytics inside your company.

You can configure different questions/queries/visualisations and organize them in dashboard. Take a look at their documentation to grasp the full extend of what you can do. You can now integrate Metabase directly from Clever Cloud.

How to deploy Metabase

Metabase is written in java and available as a jar you can download and requires a PostgreSQL database. In Clever Cloud terms it means you will need a Java runtime and a PostgreSQL addon. It's dead easy to setup. Here's what you need to do to deploy it:

mkdir metabase # Create a metabase Folder
touch metabase.jar # Create a dummy placeholder file for the jar we will download.
git init # Create the git repository
git add . # Add your files
git commit -m"init" # Create the first commit
clever create --type jar Metabase # Create the java application
clever addon create postgresql-addon --plan m MetabasePG # Create the PostgreSQL instance
clever sevice link-addon MetabasePG # link the addon
clever env set CC_JAR_PATH ./metabase.jar # Configure the jar path
clever env set CC_PRE_BUILD_HOOK 'rm ./metabase.jar && wget -O metabase.jar http://downloads.metabase.com/v0.33.0/metabase.jar' # make sure you download the latest metabase release
clever env set MB_ENCRYPTION_SECRET_KEY `openssl rand -base64 32` # Generate a random key to encrypt the database configuration
clever env set MB_JETTY_PORT  # Tell Metabase to use the port 8080
clever env set MB_DB_TYPE postgres # Tell Metabase to use Postgres instead of the default H2
# Metabase uses it's own environment variables to configure postgres so you have to copy the one we give you:
clever env set MB_DB_DBNAME dbName
clever env set MB_DB_HOST host
clever env set MB_DB_PASS password
clever env set MB_DB_PORT port
clever env set MB_DB_USER user
clever deploy # Deploy the application

Now you should see the logs showing up. Once it's over you can type clever open and it will take you straight to your Metabase instance. You should see a wizard to help you create your admin user and connect a first database. If you ave set the MB_ENCRYPTION_SECRET_KEY variable, don't wory the databases credentials you enter in the Metabase will be encrypted. Now you have a lot of things to do. And there is a lot more you can configure as you will see in their admin guide: Enable Emailing, Slack integration, add new authentication connectors… With that you should be ready to setup and query most of the datasources available to you and produce awesome dahsboards. Happy hacking!