They “push” their code, and we deploy it, while ensuring that the environment in which they operate is kept up to date and secure, with one clear objective: maintaining it in operational condition. Whatever the situation, their service responds.

Designing simple, robust and innovative services

We do this whilst minimizing the footprint on our servers, offering on-demand or automatic horizontal/vertical scaling. This means we can multiply instances or scale them up / down according to the load of your sites. This commitment to simplicity extends beyond applications, as we also offer a wide range of “as-a-service” database (DB) services, enabling you to deploy MongoDB, MySQL, PostgreSQL or Redis environments in a matter of seconds, with the same tools and guarantees.

However, over the past few years, we faced a number of recurring requests: not only did some of our customers wish to access other services, but they also regularly asked to take benefit of our infrastructure, spread over three datacenters, to obtain high-availability services. But when it comes to DB, this isn't always easy to implement while ensuring a high level of automation and data consistency. So we decided to develop our own range of products to achieve this.

The “Materia” project was born, based on the FoundationDB open-source transactional engine used, for example, by Apple for its iCloud services and by Snowflake. We're contributing to it and adapting it to our needs, while taking advantage of its distributed and resilient aspect, notably due to its significant simulation capacity. It allows us to imagine a whole universe of serverless services, very simple to use, invoiced on a pay-per-use basis, with compatibility layers we're developing to enable our customers to “talk” to it using existing protocols such as DynamoDB, GraphQL or PostgreSQL, for example.

Materia KV available to all

We started with the Redis protocol to introduce Materia KV, currently in Alpha testing phase. It's a key-value database synchronously replicated across our three Paris datacenters. When our servers respond, your data has been written three times. Above all, it's very easy to use: go to our Console, create a Materia KV add-on, and you'll instantly obtain a token that can be used with any Redis client or by sending commands to our servers directly through the TCP protocol.

Thus, you can now benefit from a high-availability, key-value data storage solution, without having to configure a single instance, and with no minimum monthly fee: you simply pay for what you use. Above all, we provide this service with compatibility to existing protocols. Today Redis, tomorrow DynamoDB and GraphQL. The list of supported commands is available here.

Launched at Devoxx France mid-April, Materia KV made a big splash on site and during our demonstrations since then. We've therefore decided to make it available to all, to enable our customers and the interested public to give us feedback on how it works for them, and what their needs are. While billing will be on a per-use basis starting with the Beta phase, right now it's free! So give it a try and let us know what you think.

Already built-in our tools and interfaces

Materia KV is of course available as an addon-provider in our API, Terraform (or OpenTofu, for example), but also in our CLI, Clever Tools. To create an access token, simply run the following command:

clever addon create kv NAME

This add-on can be linked directly to a Clever Cloud application, injecting its environment variables. But you can also use it from any third-party service, command-line client (CLI) or with graphical user interface (GUI).

While Materia KV and its various interfaces will be developed over time, we've started to think about integrating it into Clever Tools. A test branch, available as a binary and documented, offers a clever kv command.

With it, you can directly send requests to Materia KV by having the environment variable configured locally, or by specifying an ID in your commands:

# With Bash and Zsh
source <(clever addon env ADDON_ID --export)
# With Fish
clever addon env ADDON_ID --export | source

clever kv PING # PONG
clever kv set yourKey theValue
clever kv get yourKey

As this demo version doesn't handle all the commands managed by Materia KV, you can send a “raw” command and it will return the answer :

clever kv redis_raw 'hset key field1 value1 field2 value2' # 2
clever kv redis_raw 'hget key field2'                      # value2
clever kv redis_raw 'hgetall key'                          # the full hash

To demonstrate the possibilities of such integration, you also have the option of directly querying a JSON object stored in a key :

clever kv set simpleJson '{"key": "value"}' # the set data
clever kv getJson simpleJson key            # value
clever kv set jsonKey '[{"key": "value"}, {"bigKey": {"subKey1": "subValue1","subKey2": "subValue2"}}]'
clever kv getjson jsonKey bigKey.subKey2    # subValue1
clever kv getjson jsonKey ''                # the full JSON

To show just how simple such implementations can be, we've written a Materia KV client in Go using Go-Redis. If you'd rather understand how you can send commands and receive responses from Materia KV directly in TCP, you'll find examples in both V or Ruby.

Feel free to create your own and let us know ;)

A demonstration of Materia KV on Twitch, a community space

Want to find out more? Horacio González, our head of DevRel, will be giving a demonstration on Twitch on Tuesday June 18 at 13h15 (Paris Time). Pierre Zemb, creator of Materia at Clever Cloud, will also be discussing with our CTO Steven Le Roux in a video published at the end of June on Clever Cloud's YouTube channel.

And to share your views, questions and requirements, go to the Materia-dedicated section of our GitHub community space.

Clever Cloud was founded nearly fifteen years ago, out of a desire to innovate in the field of web hosting and deployment, by offering a reliable range of services in a simple, straightforward way. As a result, we became Europe's leading provider of PaaS services, attracting developers and key accounts alike.

More importantly, we are unique in offering these solutions on our own infrastructure, as well as on-premise or through various partners around the world. We now do this with ISO 9001 and 27001:2022 certification, with HDS and SecNumCloud as our next targets.

Over the years, our company and its product range evolved and expanded. But we've never compromised with our original vocation: innovation. That's why, alongside the continuous improvement of our services, interfaces and integrations, we are massively investing in the development of our future, in solutions that will meet our customers' needs ever more effectively, and over which we have full control.

This is where the independence and sovereignty of our offering comes in. That's why we've decided to build our own infrastructures, to choose a GNU/Linux system to which we actively contribute, and to use it on both our servers and our network equipment. It's also why we've spent the last few years developing a range of serverless, easy-to-use, pay-as-you-go products.

• What’s Clever in our cloud ?

Clever Functions: WASM stands for compatibility and efficiency

To achieve this, we didn't hesitate to create our own low-level virtualization and function execution layer. We wanted to maintain our strong isolation criteria by setting up virtual machines that we activate as close as possible to the CPU. We also wanted to be able to achieve good performance while opening up to a wide range of languages, rather than simply relying on v8 at the risk of focusing almost exclusively on JavaScript.

So we opted instead for a solution that leverages the capabilities of the WebAssembly standard (WASM/WASI), which can be used with a wide range of languages, from Go to Zig as well as C, Rust or AssemblyScript... and of course JavaScript.

After a successful first demo of our Tech Preview in Barcelona at the WASM.io conference last month, we'll be demonstrating again at Devoxx Paris, which takes place at the Palais des Congrès from April 17 to 19. Feel free to come and talk to us about it, and let us know if you're interested in taking part in the next stages of development of this product, which will be progressively opened up over the course of the year.

Materia KV: our distributed, multi-protocol, serverless revolution

Another major innovation from our team is the new Materia offering. To achieve this, we have designed our own serverless databases, built on the FoundationDB open source transactional engine, used, for example, by Apple for its iCloud services. We are also an active member of its community, participating in the development of its ecosystem.

Distributed and robust, notably thanks to its high simulation capacity, it has enabled us to design and build a whole range of services to meet the different needs expressed by our customers in recent years. Our approach is always open and resilient, with deployment across multiple availability zones, compatibility with existing protocols and clients, and pay-as-you-go billing.

The first to be publicly announced is Materia KV, which, as its name suggests, is a key-value database. To create one within your Clever Cloud account, you have no instance size to choose, no storage capacity to worry about. We simply provide you with a host address, a port and a token (based on the Biscuit technology to which we actively contribute), and you're ready to go!

You can then send us your keys and their contents. They will be synchronously replicated in our three Paris datacenters. So you don't have to set up any complex configurations to ensure high availability, or even activate any options. Materia KV is resilient by design! And it adapts to individual needs, as it's billed on a pay-as-you-go basis. It's the perfect solution for hobbyists, seasoned developers and key accounts alike.

Redis API compatibility… to start with

We didn't want this innovation to come at the cost of complex configuration, requiring the use of special clients and ORMs. That's why we've developed Materia KV's own compatibility layers.

To "talk" to it, you don't need a special API or tools specific to Clever Cloud. You can use it with existing solutions for DynamoDB, GraphQL or Redis. The latter is the first of the compatibility layers available in our test phase. It enables customers using a Redis instance for their application to migrate to Materia KV with just a few clicks in their interface or via Clever Tools.

This solution will be publicly demonstrated for the first time at Devoxx Paris, which takes place at the Palais des Congrès from April 17 to 19. It will then be rapidly made available to all our customers in alpha. Feel free to come and talk to us about it, and the other serverless services we'll be offering later this year.

We'll also be demonstrating other upcoming new features, such as our Keycloak-as-a-Service solution in partnership with Please Open It, and celebrating the release of version 1.0 of our in-house, open source load balancer, developed in Rust: SŌZU!

To keep up to date with our regular developments, be sure to follow our Changelog (via its RSS feed), or this blog. But also follow us on LinkedIn or X/Twitter.

We are delighted to announce the arrival of our new support system: the Ticket Center. In order to continue to provide one of the best support services on the market, we remain attentive to our customers' feedback on our quality of service. In this context, we have received numerous requests to implement a system that better distinguishes each conversation.

We therefore considered how best to continue to provide a fast and quality service and the introduction of a ticketing system seemed to be the most appropriate solution to meet the needs and expectations of our users.

What's new

Ticket-oriented use

Now, the platform usage is more "ticket" than "chat" oriented. This allows for a unique authentication of each request to ensure better follow-up.             

Adding participants to the conversation

You can now choose to automatically add members of the relevant organisation when opening a ticket. This improves your Crisp experience by ensuring that the response from our support team is accessible to all your organisation team.

When you open a ticket :

• If you are an admin, the other admins in the organisation will be added
• If your role is different of admin, members of the organisation with the same role as you will be added

How does the Ticket Center work?

Accessing the Ticket Center from an organisation

We have integrated a new button "Contact support team" in the console, at the top of an organisation. It will allow you to open the Ticket Center directly in the context of the organisation you are in. This allows support to know at a glance which organisation you are opening a ticket for.

Accessing the Ticket Center from the console's main menu

The button used until now, which was called "talk to a human" still exists, under the name "contact support". You can find it as before, at the bottom left of the main console menu. A "Select your organisation" page will then open to allow you to specify the organisation for which you wish to create a ticket.

Create a ticket

When you open the Ticket Center, you will find the history of your previous tickets. Please note that all your previous chat conversations with support have been converted into a single ticket, so you will be able to access all your former discussions if required.

Here you can also: get information on the status of your request, view the associated conversation and open a new ticket.

Giving your feedback

Once the ticket is closed, you will receive an email as usual to give your feedback, this is the time to give us your opinion on the service that was provided to you by the support.

What it allows

With the old way of working, conversations between support and a user often contained several different topics. Today, each request counts, in a unique way, which brings : 

• Better follow-up of requests.
• A more collaborative tool, by allowing you to add participants to the conversation
• A more efficient response thanks to the unique identification of each ticket.

We are confident this new Ticket Center will allow us to continue to provide a fast and quality service. Feel free to contact us and give us your feedback and ideas regarding this new system.

Keycloak is a modern and efficient way to deal with identity and access management. It allows for single sign-on, user federation across multiple user directories. It is also compliant with standard authentication protocols like OpenID Connect, OAuth2 or SAML. All details are available on its website. This blogpost will go through steps to deploy Keycloack and Grafana on Clever Cloud.

How to deploy Keycloak

Setup

Keycloak is written in Java, and gives a JAR file in their release, which means we will use a a JAVA runtime with a JAR deployment. It also requires a PostgreSQL Database, which means we will deploy a PostgreSQL addon.

# Get lastest Keycloak release
wget https://github.com/keycloak/keycloak/releases/download/17.0.1/keycloak-17.0.1.zip

# Extract the archive
unzip keycloak-17.0.1.zip

# Go to the folder
cd keycloak-17.0.1

# Create the jar application
clever create --type jar keycloak-server
export KEYCLOAK_DOMAIN=$(clever domain | xargs)

# Create the PG addon
clever addon create postgresql-addon --plan s_sml --addon-version 10 keycloak-postgres

# Link the addon
clever service link-addon keycloak-postgres

Configure

For now we will configure Keycloak through environment variables. Note that you can also do configure Keycloak through its CLI or through a file, as stated in its documentation.

Note: For the configuration through environment variables; the pattern to use is KC_ and the uppercase name of any configuration key, separated by underscores.

# Export env for templating purpose
export $(clever env | sed '/^#/d;s/\"//g')

clever env set CC_JAR_PATH lib/quarkus-run.jar
clever env set CC_JAVA_VERSION 11
clever env set CC_RUN_COMMAND "bin/kc.sh start --auto-build"

clever env set KC_DB postgres
clever env set KC_DB_USERNAME $POSTGRESQL_ADDON_USER
clever env set KC_DB_PASSWORD $POSTGRESQL_ADDON_PASSWORD
clever env set KC_DB_URL "jdbc:postgresql://$POSTGRESQL_ADDON_HOST:$POSTGRESQL_ADDON_PORT/$POSTGRESQL_ADDON_DB"
clever env set KC_HOSTNAME $KEYCLOAK_DOMAIN
clever env set KC_HTTP_ENABLED true
clever env set KC_PROXY passthrough
clever env set KC_PROXY_ADDRESS_FORWARDING true

Note: Consider setting KC_DB_POOL_MAX_SIZE at a lower value (default is 100) regarding the database size you ordered earlier. You can find the connection limit here. If you used the S_SML size that is in the command earlier, you can ignore this message. You can also fine tune the database connection pool using KC_DB_POOL_INITIAL_SIZE and KC_DB_POOL_MIN_SIZE

Now, you need to define KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD :

clever env set KEYCLOAK_ADMIN <choose an admin username>
clever env set KEYCLOAK_ADMIN_PASSWORD <choose an admin password>

Deploy

Deploy Keycloak on Clever Cloud:

# Init a git repository
git init

# Add your files
git add .

# Create the first commit
git commit -m "clever init"

# Deploy the application
clever deploy

# Open Keycloak in your browser
clever open

Validate the configuration

You can follow the Keycloak documentation starting here to ensure Keycloak works as expected. It will take you through a realm (tenant) creation, user creation and client creation/test.

Create an Initial Keycloak configuration

In the keycloak UI, ensure you create a Realm called "myrealm" (hover "Master" on the top left corner). Ensure the realm "myrealm" is selected.

Then, go to Users > Add a User. Fill the Username, Email, First Name and Last Name fields, then hit save.

In the credentials tab of the user, add the password of your choice and deselect "Temporary", the set password.

You're all set.

A real world scenario

Let's see how we can use Keycloak in the real world: as an SSO Proxy for Grafana, using Oauth 2.

Setup Grafana on Clever Cloud

If you need a in-depth guide on how to use or deploy Grafana please read this article, the following is a helper to illustrate the usages of Keycloak.

Let's use the grafana-example on Clever Cloud's Github repository.

# Clone the repository
git clone https://github.com/CleverCloud/grafana-example.git

# Enter the directory
cd grafana-example

# Create a NodeJS Application
clever create --type node grafana
export GRAFANA_DOMAIN=$(clever domain | xargs)

# Configure Grafana
clever env set GRAFANA_VERSION 8.5.2
clever env set GRAFANA_PLUGINS "grafana-worldmap-panel"
clever env set GF_PLUGIN_DIR ./data/plugins

clever env set GF_SERVER_HTTP_PORT 8080
clever env set GF_SERVER_ROOT_URL "https://$GRAFANA_DOMAIN"

Note: In this section you need to set the GRAFANA_SHA_256 variable yourself. You can find the SHA256SUM required here, next to "Standalone Linux Binaries"

clever env set GRAFANA_SHA_256 <SHA256SUM>

Configure Keycloak for OAuth 2

Note: In this section, you need to replace $GRAFANA_DOMAIN with the value you exported in the last section. You can find this value by running the following command: clever domain

In the Keyloak UI, Go to Configure > Clients > Create

Create a new client with these configurations:

Client ID: $GRAFANA_DOMAIN
Client Protocol: openid-connect
Root URL: https://$GRAFANA_DOMAIN

Further make these configurations:

Access Type: confidentials // The OAuth client must use a client id and secret.
Root URL: ${authBaseUrl}
Valid Redirect URIs: https://$GRAFANA_DOMAIN/login/generic_oauth
Base URL: /login/generic_oauth

Clear Admin URL and Web Origins.

Click save and open the Credentials tab. Copy the Secret into a separate note, we will need it in the second and third part of this tutorial.

Open the tab Roles and click Add Role. Create a new role with name admin. This role defines the access level for Grafana.

Head over to Scope tab and set Full Scope Allowed to OFF. We do not want to share any other details about the realm in the client token.

Then, we are going to configure a client mapper for the roles property. We must ensure that Grafana can extract the access role from the JWT token. Open the Mappers tab and click on Create. Create an entry with these options:

Name: Roles
Mapper Type: User Client Role
Client ID: $GRAFANA_DOMAIN
Token Claim Name: roles
Claim JSON type: string

Finally, assign the client role to your Keycloak user: Go to Users > View All Users and click on the ID of your user. Click on the Role Mapping tab, then in the Client Roles selector, select $GRAFANA_DOMAIN. In the Available Roles section, select admin and click Add selected.

Configure Grafana for OAuth 2

# Configure appropriate env vars
clever env set GF_SERVER_DOMAIN $GRAFANA_DOMAIN
clever env set GF_SERVER_ROOT_URL "https://$GRAFANA_DOMAIN"
clever env set GF_AUTH_GENERIC_OAUTH_ENABLED true
clever env set GF_AUTH_GENERIC_OAUTH_NAME Keycloak
clever env set GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP true
clever env set GF_AUTH_GENERIC_OAUTH_CLIENT_ID $GRAFANA_DOMAIN

clever env set GF_AUTH_GENERIC_OAUTH_SCOPES profile
clever env set GF_AUTH_GENERIC_OAUTH_AUTH_URL "https://$KEYCLOAK_DOMAIN/realms/myrealm/protocol/openid-connect/auth"
clever env set GF_AUTH_GENERIC_OAUTH_TOKEN_URL "https://$KEYCLOAK_DOMAIN/realms/myrealm/protocol/openid-connect/token"
clever env set GF_AUTH_GENERIC_OAUTH_API_URL "https://$KEYCLOAK_DOMAIN/realms/myrealm/protocol/openid-connect/userinfo"
clever env set GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH "contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'"

Note: In this section, you need to define GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET yourself, using the value in Keycloak UI, under Client > $GRAFANA_URL > Credentials

clever env set GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET <KEYCLOAK_CLIENT_SECRET>

You can now deploy Grafana:

# Deploy grafana
clever deploy

# Open grafana and try the Login with Keycloak button !
clever open

Now hit login with Keycloak, and use the username and password you defined for the user you created Earlier in Keycloak.

Using the same procedure describe earlier to create the first user, you can now create more users and roles. Create the roles "editor" and/or "viewer" and assign them to your new users within Keycloak. Try to connect with those on Grafana and you will see the roles are propagated to Grafana.

Sources:

• https://janikvonrotz.ch/2020/08/27/grafana-oauth-with-keycloak-and-how-to-validate-a-jwt-token/
• https://www.keycloak.org/getting-started/getting-started-zip

Over the last few months, we have started a major but gradual redesign of our web interface, the Console. Under the leadership of Hubert Sablonnière, whose team is welcoming new developers, the modules are refined and managed in an increasing way through Web Components (see this talk).

An editor accepting JSON format

The last change released is related to the environment variables management. Until now it was proposed with a Simple or Expert mode. We have added a third, using JSON format. You can read/write in it, and so export or paste a complete configuration:

The Simple editor (left) and the new JSON one (right)

The first one displays environment variables as name/value fields, you can add or remove them depending on your needs. The other two uses editable fields, but with a text or JSON format, accordingly. When the content is modified, checks are run to inform you if there are any formatting or variable management errors.

These checks are now more strict, with one exception: in some applications, such as those in the Java ecosystem, variables name may contain a dot. In this case, they will be injected as properties and not as environment variables. The rules are detailed here. The reference of our environment variables is documented there.

The Config Provider is now using a Web Component

Please note that the new editor has been designed as a Web Component. It is thus reused to display the environment variables of add-ons and has just replaced the one used for the Config Provider which was a simple multi-lines text editing field. 

As a reminder, this add-on allows you to create a list of environment variables that can be shared by different applications. To use it, you just need to integrate the Config Provider as a service dependency in their settings.

The Config Provider with the Web Component editor (left) vs the previous basic editor (right)

Dans cet incroyable épisode, il est question de l'accord entre Elastic et Amazon, du nouveau site de MDN, de Mozilla et du WEB3, de JS, de compression Zstd avec PostgreSQL, de Java et de React 18, de la série à regarder JS idiomatique… Pour enfin finir en grande musique avec Pierre et le Loup, pour s'ukrainiser civilement le neurone symphonique.

👋 Venez discuter avec nous sur @clever_cloudFR pour nous dire ce que vous avez pensé de ce nouvel épisode.

➡️ Pour découvrir ou réécouter d'anciens épisodes c'est par ici !

Timecodes & liens :

00:00:00 : Introduction et présentation de l’invité
https://twitter.com/fcamblor
https://www.bdxio.fr/
http://bordeauxjug.org/
https://vitemadose.covidtracker.fr/

00:07:00 Elastic and Amazon Reach Agreement on Trademark Infringement Lawsuit
https://www.elastic.co/blog/elastic-and-amazon-reach-agreement-on-trademark-infringement-lawsuit

00:09:44 : A new year, a new MDN
https://hacks.mozilla.org/2022/03/a-new-year-a-new-mdn/
https://en.wikipedia.org/wiki/OpenSearch

00:19:26 : Mozilla s’attaque au web3?
https://hubs.mozilla.com/

00:23:03 : Les résultats du State of JS 2021 (et début 2022 :)) sont sortis
https://2021.stateofjs.com/en-US/

00:34:17 : PostgreSQL Begins Working On Zstd Compression Support
https://www.phoronix.com/scan.php?page=news_item&px=PostgreSQL-ZStd-Start

00:38:35 : Java 18 Final RC (GA prévue le 22/03)
https://openjdk.java.net/projects/jdk/18/
https://www.loicmathieu.fr/wordpress/informatique/java-18-quoi-de-neuf/

00:47:53
React 18 va-t-il apporter (enfin!) un support des custom elements ?
https://twitter.com/dan_abramov/status/1483892375223414785
https://custom-elements-everywhere.com/

00:54:44 : React Query v4 alpha, connection mode
https://react-query.tanstack.com/

01:03:50 : La série JS Idiomatique de Delicious Insights (Christophe Porteneuve) qui reprend les bases d’un certain nombre de features EcmaScript
https://www.youtube.com/playlist?list=PLPoAfTkDs_Jad8tGdMkixD1r7lMjpRstB

01:07:20 : Le Choix musical de Frédéric Camblor : Pierre & le Loup (Sergueï Prokofiev)
https://youtu.be/eN4Iyb8HKjM?t=596

Surprise as it often happens: the report that was made on your company two months ago is going to be programmed tomorrow evening. Or, in the middle of a hot topic, your product is going to be the subject of an item during the 8 o'clock news.

The TV appearance is always a bad match for the web: because when you size your IT infrastructure, it's more to manage your growth over 3 years. On a nice upward curve, usually linear. And when the traffic peaks by 800%, your architecture will be on its knees.

Except with Clever Cloud. Configuring your applications to absorb a traffic spike can be done with a few clicks in the interface:

Only, managing site traffic is easy when you are already hosted by Clever Cloud. But if this is not the case, there is no fatality.

Yesterday, M6 broadcast the show "Who wants to be my partner? And the young company BioDemain, which helps agricultural producers during the three years of their organic transition, took part. Hosted on a traditional architecture, the risk of the service falling down is great. So we helped BioDemain migrate their site to Clever Cloud in less than 24 hours!

How Clever Cloud handles these traffic spikes

First of all, managing site traffic will be done on the Clever Cloud side by load-balancers before reaching your application. These load-balancers (formerly HAProxy instances, and now instances of Sōzu, an in-house software) are going to distribute the traffic among themselves. Just like if you had Amazon ELBs (Elastic Load Balancer), except that here it's included by default, and already configured.

Your application will then receive the traffic. Depending on your scaling configuration, your application will be able to trigger alerts to our monitoring system if CPU and RAM increase. And therefore multiply the number of instances. Load-balancers and multiple instances in a few seconds: and you are ready to handle the traffic.

The migration itself

To move to Clever Cloud so quickly, we were put in touch with the BioDemain team. And it was the formidable support organisation that was at their side to migrate their site (here a Wordpress) from a shared solution to Clever Cloud. Despite a rather… tight schedule.

The recipe for success was a very quick migration. Indeed, there are two ways to deploy your Wordpress on Clever Cloud:

• Quickly, with a simple FTP transfer.
• More elegantly, using git for the code, Cellar for the media and Composer for the plug-ins.

Obviously we went for the first solution.
After contacting the developer of BioDemain, the first step was to transfer the files between the old host and Clever.
Several tens of thousands of files, so in FTP it takes time.

While waiting for the end of the file transfer, our team and BioDemain were able to deploy two other instances: MySQL and Redis. Then came the configuration of the PHP application itself (still during the transfer).
Here we had to:

• Activate ProxySQL to mutualize the application connections
• Configure proxySQL via simple environment variables:
  • CC_ENABLE_MYSQL_PROXYSQL=true for activation
  • CC_MYSQL_PROXYSQL_MAX_CONNECTIONS =15 to set the maximum number of connections to be managed
  • CC_MYSQL_PROXYSQL_USE_TLS=true if needed
• Make a dump of the original MySQL database and import this dump into the new one
• Enable the Redis session in PHP via environment variable].

Extra tech tips

• You can also enable the Varnish built into Clever Cloud if you feel like it:
• Put in a CDN: if you don't have time to optimize your SQL queries, caching strategies, minifications etc., CDNs are there to offload some of the traffic to them.
• Before anything else, lower the TTL of your CNAME and A records of your domain name. At the time of migration, if you are short of time and the changes to the domain name are propagated over 24 hours, you will find it a shame.
• And if you ever have a down, redirect your home page to an application that displays a contact form: users can then leave an email to be contacted again.

Finally, evaluate the performance. The fateful moment of the TV pass does not suffer from a single oversight. To do this: JMeter locally (beware, it requires resources), or great online services like gatling.io

You have all the cards in hand for the next step :) Good luck!

In one of our previous posts, we announced the release of a dedicated Clever Cloud Grafana for all our metrics. What if now, you want to receive an alert when an application is in an abnormal state? Grafana provides an alerting service that is compatible with our Warp10 backend. Let’s see how we can create Slack alerts with our or any Grafana with a PromQL data source.

Grafana Notifier

First, before creating any alerts, we need to configure a Grafana notifier. You can have a full list of supported Grafana notifier here. However, you won't be able to configure notifiers that require an admin access, like email or SMTP. In this example, we will configure a slack channel to receive all Grafana alerts.

Configure Slack

To be able to receive alerts on Slack, Grafana will ask for a web hook. The process to create one with slack is here. At the end you should obtain an active incoming web hook.

Simply copy the web hook URL and then go to the alerting main panel (bell on the left) and add a new slack notification channel. Configure it as done below, within the field web hook URL, the one from the slack application. You can set your own custom name.

Before you validate, you can test your slack integration with the Test button. Without delay, in the configure slack channel, you will receive a test message.

Grafana Alerting

Now that we have enabled an alert notifier, let’s create an alert in Grafana.

Clever Cloud Available Metrics

We will start from one of the template dashboards we release with Grafana. We deployed a preview application on a Clever Cloud organisation.

For the rest of this post, we will work on the load average metrics. As we can see below, it’s a simple PromQL query. However, it includes custom dashboard variables APP_ID and DEPLOY_ID that are not supported in Grafana alerting!

Before enabling the alerting, we will keep somewhere this graph query system.load1_per_cpu{app_id="APP_ID"}. I will only replace the variable APP_ID by its value, which is available on top of the dashboard.

Then in order not to have our custom alerts removed at each update of the Clever Cloud dashboard templates, create a new dashboard.

Also save it in a dedicated dashboard which is not Clever Cloud, in our example, it’s alerting.

Finally, in this new dashboard create the same load graph with the query system.load1_per_cpu{app_id="APP_ID"}. As well as seen previously replace APP_ID by your own application ID. For the data source, select the PromQL one.

Set Up The Alerting

At this instant we have a new graph where we can set up a first alert on the load of our application. For that, go on the Alert panel, and configure the alert based on the parameter you want. Quickly, to get an alert working, I set up the condition to check if my metrics are below the value 0.2.

Then still on the Alert panel add the notifier you created in the first section. Here, you can customise the alert you will receive in slack: adding a message and custom tags.

Once you have set up your alert, you can test the rule with the test rule button.

Next all you have to do is to wait for an alert to appear on Slack! Besides each alert state change will be ploted on the alert graph. After about five minutes, as I was clearly matching my alert state condition, I got two states change: one pending and one alerting.

As soon as the alert state happens, a slack message is sent through the notifier.

Finally, I would like to retrieve a more ‘normal’ state. To do it, I update my alert condition and set it to above 0.8, which is a more classical value for an alert threshold for the load metrics.

Instantly or almost, the state of the alert should change and comes OK!

Immediately after Grafana also send a message to slack to end the issue!

To Sum Up

You should now be able to configure your own alerts on Slack for Grafana! You can play with the pending time, the alert condition, when to trigger the alert, the alert message the tags and so on. In fact, all you need is a working PromQL query. Nonetheless remember that the Grafana alerting doesn’t support dashboard variables. Other than that, you should be able to custom the alert as you wish!

I hope you enjoy and feel free to share with us your thought on the Alerting and the Metrics product. We will be happy to have your feedback. You can contact me directly on twitter, or send us a message through the metrics overview comments.

At Clever Cloud we aim to provide the best services for all kinds of data. Time Series data are no exception to this. We're thrilled to announce a time series database, TARDIS, in the next few weeks.

About TARDIS Features

A Time Series Database (TSDB) store series of chronological data, in an optimised way. The use cases are numerous:

1. Time Series and Signal processing : TARDIS will be able to store your metrics and states. It will also offer an interactive and analytical queryable store to monitor changes over time.
2. BI & Analytics : You can use TARDIS as an analytical solution for Time Series. It will power Time Series analytical queries as well as dashboards. We are planning 3rd party integration too.
3. Monitoring: TARDIS will help you to integrate KPIs from a specialised service.

Technology & Supported Protocols

This TSDB will be backed by our partner SenX, the company behind Warp10, on a distributed infrastructure backed by OVHcloud.

One of the most interesting news about TARDIS is its ability to be a polyglot TSDB and to support multiple query languages and protocols. For instance, you can push your data with Prometheus, and you can retrieve it with WarpScript or PromQL. It’s the same data, without duplication.

The architecture uses proxies (hosted and managed by Clever Cloud) that bridges the gap between Warp10 and various requests protocols, similar to our metrics backend.

Reliability FTW

TARDIS stores data in a strongly consistent fashion, thanks to a transactional pipeline. So, we won’t lose any data once a push is acknowledged.

Regarding high availability, we operate a special care for the storage too. Namely, rack awareness optimisations, network topology fault isolation and data placements.

We’ve designed everything in TARDIS for very high scalability. We clocked this technology with production workloads with millions data points per second, and over a billion unique series for a single region.

Pay as You Go

TARDIS only bills for what users use. It start at 5 millions points (perfect for an app that produce 100 data points every minute, during a month). We will release the final pricing within the next few days. We will propose a beta program at a relatively low price first. Then the stabilised product pricing will change a bit, still at a competitive level (as always!).

Integrated Authentication

TARDIS will be integrated with Clever Cloud IAM, as it is with our Apache Pulsar and Elasticsearch services. In the meantime, we provide crypto tokens that let users define read or write access, to their entire data set or subscope to specific labels.

Deletion & GDPR

TARDIS complies with GDPR by allowing customers to delete their data. Deletes can occur on a single data point, a whole series, or a set of series, over a specified time range. Which is particularly useful nowadays. To learn more about this, check the documentation. 

Release date and beta program

We're planning  the public release by the end of the 2021. But we can provide the service in advance as of today. You can reach us via the following form :

APM means Application Performance Monitoring. When we say APM in the IT context, we refer to solutions that monitor the platform hosting the application and the app itself.

APM solutions extract information and data about processes, memory usage, disk reading or writing operations/seconds, and other information about the hosting platform. On the application side, they monitor in and out requests, errors, issues, and even code execution to highlight memory and time consuming processes.

These tools can be very important when you are on a cloud based service that auto scales, because some of these issues may be hidden by the auto allocation of more resources to an application. They also give the opportunity to check if the cloud provider's SLA is honored.

APM by Elastic

Available since version 6.2 (released in 2018), Elastic APM has proven its efficiency on monitoring and improving applications. Elastic APM monitors in real-time memory allocations, bandwidth usage, data rates, response time, database queries, CPU resources on the client side, calls to cache, unhandled errors and exceptions with their stack trace, aggregates logs in a pretty format and much more thanks to client and server side APM agents.

Distributed tracing feature allows you to see how a request flows into your micro-services architecture and to identify what exactly needs to be improved.

As all the collected data are stored and indexed in Elasticsearch, it's very easy to query them and point out bottlenecks.

Additionally, data is automatically available to consultation in a Kibana dashboard. This dashboard comes with a special UI from Elastic where you can conveniently explore your data via generated charts, see metrics about CPU and system memory usage, get the best out of their default filtering by request type, request results, errors with the already identified culprit. And of course you can investigate by search, all of this working out of the box.

And as if it was not enough to really improve our "coding, debugging and releasing" workflows, Elastic APM can send email alerts or Slack notifications when issues arise.

From a practical point of view, APM needs an instance of Elasticsearch, an APM server, Kibana and an agent on the application side. On Clever Cloud, all you need to do is to configure your application to use the agent. Let's see how to do that.

Elastic APM on Clever Cloud

If you have read other blog posts of mine, you may know that I'm a Ruby on Rails lover, so no surprise if I chose this Framework to demonstrate how easy it is to get started with the Elastic APM on Clever Cloud. However, APM Agents are available in many languages, so you can follow this tutorial and replace the Ruby on Rails application by one of yours hosted on Clever Cloud that is written in one of the following languages: Go, Java, JavaScript (backend or frontend) or Python.

Creating the Rails application

Well, I'm not really a person that likes to do the same thing twice so as I already wrote an article on how to deploy Ruby on Rails applications on Clever Cloud, I will start this tutorial by considering you already have an application running like at the end of my previous article. If you don't, do not hesitate to take a break to follow the tutorial, it takes five minutes to be all set.

Creating the ES stack on Clever Cloud

Well, it's pretty easy:

• Go to your Clever Cloud console, under the organization where your application lives, select Create then an add-on. Scroll down until you find and select Elastic Stack.
• On the next screen select the plan you need, there is no minimal size to benefit from these features.
• On the next screen enable APM and Kibana
• You will now be prompted to name your add-on, and you will be able to select the deployment region. Fine tune as you need and press next.
• Go to the Service dependencies section of your application, then select the APM server in the Link applications drop down menu.

The Elastic Stack add-on will be automatically created: Elasticsearch, an APM server and a Kibana instance will start. You will see them appear shortly under your organization; there's nothing more you need to do to start using them.

If you navigate to your application's Environment variables menu, you will see that you can now use environment variables provided by the add-on we just created. And fortunately this is almost everything we need to finish this configuration.

⚠️ Warning: Make sure that you linked the APM server, and not the Elastic add-on. Otherwise, you won't have the correct ENV VAR for the next steps.

Configure your Application for APM

Ruby on Rails

Go in your terminal, under your application repository.

• with your favorite text editor, edit the Gemfile to add gem 'elastic-apm'.
• Create a file named config/elastic_apm.yml that you will fill with this:

server_url: <%= ENV["ELASTIC_APM_SERVER_URLS"] %> 
secret_token: <%= ENV["ELASTIC_APM_SECRET_TOKEN"] %>

You may here notice the usage of environment variables to protect APM server's secrets. You can use these environment variables thanks to linking your application to your APM server on Clever Cloud.

Other languages

You can find here the list of all the APM agents by language. You can just follow the simple steps and refer to the environment variables mentioned in the Ruby on Rails section to replace host url and secret token values. You may also need to use additional environment variables, read this to add them to your Clever Cloud application.

Deploying the APM agent

Okay, that was it. Now, you can add, commit and push these changes to Clever Cloud either via the CLI clever deploy or by pushing on the Clever Cloud git remote (usually named clever) of your application using git push clever master. Your application will restart within seconds and instantly connect to the APM server thanks to the environment variables we provided.

Enjoying your APM / Kibana

Now that our application is restarted, we can go to our Kibana application in Clever Cloud and click the link icon at the top right of the application menus. This will open a new tab with a Clever Cloud login form. Connect with your Clever Cloud credentials. This will open your Kibana after a brief apparition of a Keycloak page.

As Keycloak is not the topic here, all I'll say is that authentication to Kibana is delegated to Clever Cloud using this solution. If you want to know more about it, read this.

In Kibana, find the APM section in the menu on the left (tip: you can unroll the menu with a button at the bottom of the menu). Click it and Wow! you already can visualize your first application performance metrics in the beautiful Elastic UI for Kibana.

Here's a step by step video:

Going further

What we've done so far is pretty great as we can now monitor our applications performances in real time and we needed to setup almost nothing. This is way better than having nothing, but if you're familiar with APM tools you may want to customize and/or add more data to your reportings.

You can read the configuration section of each agent to get the best out of the configuration file we created. You will be able to extend the APM server side agent's configuration like adding your own tags and key filters, decide what exactly you want to capture for each request to your server, fine tune the way the data is sent to the APM server including size and time of the requests and much more. You can learn more about the data model of the events captured by APM here. You are going to learn here how to enrich your events with metadata to really customize your data. If your application is client side, you can enable the RUM (Real User Monitoring) by starting here.

Happy monitoring!