It is now possible to force HTTPS redirection! No more .htaccess black magic, no redirection handling code… Let's have a tour of this new feature.

Forcing HTTPS on Clever Cloud

Let's start practical. There are two ways to achieve our goal.

Using the CLI

In your linked application, in your own terminal just type in clever config set force-https enabled.

clever config is a new CLI command that allows you to display in your console the current configuration of your application. You can edit any setting with clever config set {name} {value} and bulk edit with clever config update [options].

Here we just set the force https option to enabled and if we want to set if off we can just clever config set force-https disabled.

If this does not work for you, make sure you are running version 2.5.0 or newer (with clever version).

Using the Clever Cloud console

In the information page of each application, you will find a checkbox form with options you can tune in and out. One of them is Force HTTPS. Enable its checkbox and save.

From now on and as long as you do not disable it, every non-secured HTTP request to this application will be redirected to HTTPS with a 301 Moved Permanently status code.

How the magic happens

The Clever Cloud way

The redirection is handled at the reverse proxy level so you don't need to update your application to use it. Each time the browser will request a resource using HTTP, it will get a 301 response back with the same resource prefixed by https in the Location header field. The 301 redirection is recognized as the best practice for HTTPS upgrade.

Why enforce HTTPS

Before explaining why it is important, I shall provide a definition of HTTPS.

Simple definition

First of all, HTTPS is an extension of the HTTP protocol. Which itself is the application protocol the world wide web is relying on for communications. Full definition to be found on wikipedia.

The main difference between HTTP and HTTPS communication is encryption.

Using HTTPS, the communication is bidirectionaly encrypted between the client and the server. Both the headers and the response/request data are ciphered. This is achieved using TLS cryptographic protocol in addition to operations involved in a simple HTTP transmission.

The identification process requires the server administrator to create a public key certificate signed by a trusted certificate authority.

We have automated this part. So every time you create a new application on Clever Cloud, a cleverapps.io subdomain name is given to your application and comes along with encryption because a certificate covers cleverapps.io and all its subdomains. And each time you add a custom domain to your application like example.com, Clever Cloud asks Let's Encrypt for a TLS public key certificate that we automatically add to your application's configuration on our reverse proxies. Thanks to that, you can accept HTTPS requests on a brand new application without further configuration.

Enforcing HTTPS is just the beginning

I explained to you how HTTPS enforcing was working on Clever Cloud and that felt very simple, right?

To be honest this is because the tech team has decided only to handle the first step of the process.

Strong security on an application requires knowledge on many topics.

Sometimes, your application itself will link to HTTP content, which will result in mixed-content. When HTTP resources are served by the application on a page requested using HTTPS the browser will usually not load those resources.

You will also probably want to know more about HTTP Strict Transport Security (HSTS), and the Content-Security-Policy header.

You can start by reading this great article from Scott Helme to start your journey. If you cannot handle advanced security yourself, think about asking for the help of a security expert.

HTTPS is good for everyone, except hackers

In 2020 the HTTPS protocol is ubiquitous. It's more and more widely used thanks to the Let's Encrypt initiative which made it free for everyone; which makes the web more secure for both applications creators and clients.

The clients know they can trust the integrity of the content that's being displayed to them and feel safer about using the Internet in general. This safety feeling extends to their privacy. Today the regular user's main concern is about what the owner of the website do with their data. Not about that black hooded bad guy they were freaking out about a few years ago. The most used websites have raised their confidence about communications ciphering to such a level it's now the norm. Everyone puts credit-card numbers in forms eyes wide shut.

Yeah, but me ? Having a website to offer to the world, I want to meet my client’s expectations, of course. But HTTPS has more to offer me. From the SEO perspective, having it will naturally boost ranking. I also ensure my user's privacy and confidentiality. Most importantly, I can trust that the data I receive from my client has not been altered. That doesn’t mean that I shall trust my user intentions obviously. But now if I need to watch out for a hacker, it will not come from a man in the middle or this kind of attack.

Anyway, this is everywhere now and already almost mandatory. Take this train now! This feature is free, does not require extra code and works out of the box with the automatically generated Let's Encrypt certificates.

How to get your access logs

{
  "t": "2020-03-10T11:24:34.220324Z",
  "a": "app_87d748d4-3c7b-4baa-8f52-c748e1123ab3",
  "adc": "par-cleverapps-adc-n6",
  "o": "user_51f7098f-604e-4d9e-8fb6-4026fd1767bb",
  "i": "0805330e-cff6-4fd5-87c0-26974d8bcb4e",
  "ipS": "255.255.255.255",
  "pS": 38566,
  "s": {
    "lt": 48.6719,
    "lg": 1.3002,
    "ct": "Aunay-sous-Crecy",
    "co": "FR"
  },
  "ipD": "255.258.258.258",
  "pD": 11810,
  "d": {
    "lt": 48.7967,
    "lg": 2.3338,
    "ct": "Cachan",
    "co": "FR"
  },
  "vb": "GET",
  "path": "/status-codes/500",
  "bIn": 411,
  "bOut": 443,
  "h": "app-87d748d4-3c7b-4baa-8f52-c748e1123ab3.cleverapps.io",
  "rTime": "3ms",
  "sTime": "115μs",
  "scheme": "HTTPS",
  "sC": 500,
  "sT": "Internal Server Error",
  "w": "WRK-01",
  "r": "b6bcab8a-c6d0-4546-9000-57b43d009a49",
  "tlsV": "TLS1.3"
}

What you can do with your logs

clever accesslogs --after 2020-03-10 --before 2020-03-20 -F json | jq .sC | sort | uniq -c | sort -nr

  99999 200
     56 500
     33 404
      2 400
      1 403
      1 301
      1 202

#!/bin/bash

input_start=2020-01-01
input_end=2020-03-27

startdate=$(date -I -d "$input_start") || exit -1
enddate=$(date -I -d "$input_end")     || exit -1

d="$startdate"
while [ "$d" != "$enddate" ]; do 
  clever accesslogs --alias prod --after $d --before $(date -I -d "$bv d + 1 day") -F clf > access-$d.out
  d=$(date -I -d "$d + 1 day")
done

We have some news about our dear clever-tools, the Clever Cloud CLI for humans, robots and others.

Back in January, we released version 0.10.1 which contained some new commands and a few bug fixes. Today we're releasing version 1.0.0!

You can find all the details about the user oriented changes in the CHANGELOG. In this article, we'll cover the structural changes related to the project:

• Improved error handling = better usage in CI/CD pipelines
• ES2015 refactor + ESLint = better contributing experience
• Goodbye nodegit, hello isomorphic-git!
• Improved Jenkins build = more release methods and beta channel (npm, deb, rpm, arch, homebrew and chocolatey)

Now let me tell you the story of this release...

Improved error handling = better usage in CI/CD pipelines

It all started with issue 195. With the previous version, when a user tried to clever deploy a non existing branch, the error message was a bit too raw. Our colleague Philippe wanted emojis, but we had to decline 😭.

While investigating this error message and how to improve it, we realized that the command was not returning a proper error exit status. This is a very important rule to follow when you're writing a terminal command/script:

• If everything went well: return exit status 0
• If something went wrong: return exit status 1 (or another one more specific)

Our CLI is built for humans AND robots. We want to empower our terminal addicted users. Moreover, we want to empower our users to automate anything related to Clever Cloud in their CI/CD pipeline. In such cases, returning a proper error exit status is crucial to mark a CI/CD job as successful or failed.

We took a quick look at the rest of the codebase and noticed that other commands were concerned by this problem. We reworked the code of clever deploy and decided to apply the same error handling to all the other commands:

• For humans: Always have an [ERROR] prefix (colored in red) when we display an error message.
• For robots: Always log error messages to stderr.
• For robots: Always return 1 as exit status when something goes wrong.

This obviously required some significant work. We had to go through all the 28 commands. Nevertheless, we're sure this will help you to build more robust and automated jobs, interacting with the Clever Cloud platform.

ES2015 refactor + ESLint = better contributing experience

While we went through the codebase, we took the opportunity to do a bit of refactoring.

First, we decided to update all our code to ES2015+ syntax. We tried to use most of the modern good practices of the JavaScript ecosystem when it made sense (destructuring, rest parameters, fat arrows...). We also decided to setup an ESLint config file (based on StandardJS) to enforce our codestyle for the next iterations. This lint check is now part of our automated test suite on Travis CI which is triggered on each pull-request.

This big refactoring took some time but it allowed us to reduce the number of lines of code while improving the overall consistency and readability. We hope this will drive more users to customize the clever-tools and propose pull-requests for bug fixes and feature requests.

Looking at how easily the new clever console was added by an outside contributor, it seems like we're on the right path.

Goodbye nodegit

During this refactoring, we updated most of our dependencies but one of them required special attention: NodeGit.

NodeGit is a node.js module which provides bindings over libgit2, a portable and pure C implementation of the git core methods. This is commonly referred to as a "native module" in the node.js ecosystem. Each time you install NodeGit, npm will try to build libgit2 from source (in some situations, it will try to download pre-builds). Because we want our users to have a straightforward installation experience, this native module thing had many drawbacks:

• In our Travis CI config, we had to install some native dependencies so npm could build libgit2 from source.
• Some of our users tried to install the clever-tools via npm and had difficulties because of this.
• Our users don't all use the same version of node.js, this also multiplied the number of weird cases we had to support.
• Each time we wanted to update NodeGit, it was a puzzle to resolve.

All those problems pushed us to stop releasing the clever-tools via npm. ⚠️ SPOILER ALERT: it's back for version 1.0.0.

The second problem is that we use pkg. This tool is great. It lets you compile a node.js project to different portable binaries for MacOS, Windows and GNU/Linux. We've been using it for a while to provide various installation methods. When we dropped installation via npm for 0.10.1, the binaries built with pkg where the only official way to use the clever-tools.

This mix of NodeGit + pkg complicated the situation even more:

• The binaries built with pkg did not include native dependencies. We had to distribute the right nodegit.node file along with our main binary file and users had to have it in their PATH.
• On some GNU/Linux distributions, we had problems with native dependencies like libcurl-gnutls, libnghttp2-git or libssh2

We already looked for a replacement solution but a fairly new one came up to our attention in February...

Hello isomorphic-git

isomorphic-git is a "pure JavaScript implementation of git that works in node and browser environments". The project was created by William Hilton. This guy is awesome!

As I said, we heard about it in February but we only tried to use it for our use-case in July. It does not support all the git feature set (yet) but it has strong foundations and we only need a few things for the clever-tools:

• list and resolve branches
• list and resolve commits
• add/rm remotes
• push to repos (via HTTPS)

We were missing a few features so we reached out to William to see if we could help and/or contribute. In the end, we contributed a few features in the codebase and we sponsored the hosting of the CORS-proxy which is used to demo the project in browsers.

By replacing NodeGit with isomorphic-git, we can say goodbye to most of the problems we described earlier:

• no need to worry about different node.js versions
• no need to have native dependencies on our CI server to build/test the project
• no need to distribute nodegit.node with the pkg binary
• no need for a user to have the right native dependencies on his/her system to use it
• npm releases are back

Improved Jenkins build = more release methods and beta channel

Because we decided to ditch NodeGit, we were able to simplify the packaging (no more nodegit.node and no native dependencies). This required some rework on our Jenkins builds but it was a good occasion to migrate 5 different jobs to one multibranch pipeline project.

We'll discuss the details of this new build system and how we set it up in another article. For now though, here's what changed:

• All platforms:
  • Added: installation via npm is back
• GNU/Linux:
  • Added: rpm packages (stable & beta) with a yum repo on Bintray
  • Added: deb packages (stable & beta) with an apt repo on Bintray
  • Added: Exherbo exheres clever-tools-bin
  • Changed: dedicated AUR package for beta versions
• MacOS:
  • Changed: dedicated homebrew tap for beta versions
• Windows:
  • Changed: chocolatey packages are now automatically published (no more manual upload, no more manual review from chocolatey's team)
  • Changed: chocolatey packages are no longer published on chocolatey.org
  • Changed: chocolatey packages are published (beta & stable) on Bintray but you can still use the choco CLI to install them (see chocolatey installation docs for the specifics).

The structural changes of this release will help us to be more reactive on the project. We'll be able to publish new features and bug fixes more easily and more often. Hopefully, this will improve the quality of the clever-tools and push more customers who only use the Web console to try the CLI way of life.

We're looking forward to read your feedbacks on this release through are usual channels: support chat, email, Twitter, GitHub issues…

Welcome to the first edition of One Framework a Day keeps the Boredom Away. In this series I will show you how to deploy a particular framework on Clever Cloud.

This post 0 is a little different as its goal is to explain you how Clever Cloud works and to help you get setup. If you already know Clever Cloud and have the CLI installed, wait for the next post tomorrow :)

What is Clever Cloud?

Clever Cloud is a Platform as a Service. We like to say we provide IT Automation and Application Sustainability. That and additional sleeping hours and serenity for our users.

IT Automation

What we mean by IT Automation is that we automate things. Beyond this obvious statement, what is important is that we do it, so you don't have to. You should be able to focus on writing code because it is what brings value to your organization. Anything else than code is infrastructure. It's a cost. We'll manage it.

To make it work you just have to agree with a couple of loose rules (well, 12 actually) and everything will be fine. If you use a build tool, just add a json file telling us which build tool goal we have to run, push your code and we do the rest.

From now on your application is up and running on Clever Cloud. That's where Application Sustainability kicks in.

Application Sustainability

What we mean by Application Sustainability is making sure your application stays in production. To do so we provide hardware monitoring out of the box. Even better, since you are usually using a build tool we know what runtime you are using so we know how to monitor it. We automatically integrate with Java's JMX or Haskell's EKG for instance.

This monitoring allow* us to give you additional features like self-healing: If your application crashes or does not answer we restart it automatically. We also bring automatic scalability both vertically and horizontally. And everything happens without downtime because we do blue/green deployment. Meaning, when we need to update a VM (say for an OS update or a security fix), we don't update it per se. We restart the same cycle on a brand new, updated VM. Once it's up, our reverse proxy simply redirects traffic to the new one(s). In that way, we decorelate service uptime from server uptime.

We can do all that because we embraced Immutable Infrastructure. This concept appeared somewhere around 2010 and is a full paradigm shift. Basically, not a single part of your infrastructure will ever be updated in place. It will either be created or removed, never modified. Modifying something creates uncertainty and possibly an inability to be reproduced. Immutable Infrastructure ensures reproducibility.

Applications or add-ons?

To support this, we manage two different concepts: Application and Add-on. Applications are stateless. They are represented by a runtime and sometimes a build tool. For example a Wordpress website does not need a build tool, just a runtime: a web server. As developers, application is where we spend the most time on. This is where we create value. All applications instances (VMs) are immutable. So you need to make sure your deployments are stateless. You have to store state in a database, object store or filesystem.

Which is where add-ons come in handy. They are not yet managed as Immutable Infrastructure (it is however part of our plan). They all our DBaaS (which all have automatic and on-demand backups), our shared filesystem, object storage like S3 etc… It also includes stuff like Jenkins, Artifactory, GitHub Enteprise, Gitlab and more.

Hopefuly this gave you a much better idea of what is Clever Cloud. We could go on longer but I will assume you are bored already. So moving on to practical things :D

Deploy your first Application

To create an account, simply go to our website and click on 'Sign UP and Try'. If you do not have a GitHub account you can signup manually here. Once you have an account you can start playing around with your free credits. Take a look at some of the videos we made on our YouTube channel.

Setup the CLI

After you have setup your accout you can install our CLI. It will be heavily used on most posts of the One Framework a Day series. There are a couple of ways you can install it. First you can checkout the GitHub project and build it from source. It's Node.js app so you will need npm, node and a couple of other dependencies described in the Readme.

You can also use a package manager or installer. Head over to our documentation to get the details for MacOS and homebrew, Windows and chocolatey or Linux and its prebuilt binaries.

Test the CLI

If you have an account and installed the CLI, you should be able to test your first application. First thing to do is to link your account to the CLI. Simply type clever login. This will get you through a login prompt to retrieve your newly created identification tokens.

Create a new directory called hello-world-app and inside create an index.html file containing the string Hello World. Then initiate a new git repository and commit the html file. Here are the steps you need to follow if you have a Unix system:

mkdir hello-world-app
cd hello-world-app
touch index.html
echo 'Hello World' > index.html
git init
git add index.html
git commit -m "my new website"

You now have a static website ready to be deployed to Clever Cloud. First you need to create an application of type static, then deploy the content of your site to the newly created application. Once it's done, open the website in your default browser:

clever create --type static-apache hello
clever deploy
clever open

Conclusion

Now you should know more about Clever Cloud and you have deployed your first application. Please note that if you don't like command line, you can also do all of this from our web console or our API. Actually, both the CLI and the console use our API :) This also means you can script whatever you want! And now remember that all the next posts will be dedicated to deploying a particular framework on Clever Cloud. See you soon :)

Today we announce the general availability of our SSH gateway. It was in Beta for a while but is now production ready. All of this means you can now SSH to your VMs \o/.

If you are in a hurry, everything is explained in our documentation. Now I would advise you to still read what follows because it's important.

Friends don't let Friends SSH on production

Or do they? Delicate question. At Clever Cloud we get scared when an SSH connection is opened to a production VM. This usually means something is wrong. Then there are two options.

• A user gets on the machine to debug something, fixes it on their code, pushes and redeploys.
• A user gets on the machine to debug something, fixes it on the machine and calls it a day.

The second option is the one we dread. Since Clever Cloud embraces Immutable Infrastructure, all our machines can be replaced by new ones at any time. Which means that if our user logged in a machine with SSH and fixed something wrong on that very machine, all they did will be lost after a new deployment.

So as a reminder, we really want you to be careful and to always remember that doing things manually through SSH is a bad practice, that it should have been automated. It is however perfectly ok to use SSH to figure out what is wrong on the machine and then fix your code :)

How does it work?

You can SSH to an instance of your application in two different ways. You can use SSH or the clever-tools CLI.

With SSH

Super simple, simply type ssh -t ssh@sshgateway-clevercloud-customers.services.clever-cloud.com <app_id> and you are in.

With Clever Tools

Even simpler, if you have our CLI tool installed:

clever ssh
> Opening an ssh shell
> bas@67fbf787-3518-47bb-abd9-2c2575844edd ~ $ 

If multiple instances are running, you will be asked which one to use:

clever ssh
> 1) Sleepy Ponita - Instance 0 - UP (11281f38-31ff-43a7-8595-a2d82630c32b)
> 2) Tense Caterpie - Instance 1 - UP (b10d19d9-5238-408b-b038-3e32c7a301c2)
> Your choice: 1
> Opening an ssh shell
> bas@11281f38-31ff-43a7-8595-a2d82630c32b ~ $

And that's all! We hope you like that new feature and that you will put it to good use. Remember that with great power comes great responsibilities!

Clever Tools should better be called Clever Cool

# Deploying on the production server
$ clever deploy --alias prod
# Displaying the (live-updated) status of an app
$ clever activity --follow --alias preprod
# Show the logs
$ clever logs
# Create a free plan PG addon named my-addon in the Montreal zone
$ clever addon create postgresql-addon my-addon-pg --plan dev --region mtl
# Create a node.js app named my-app-dev in the Paris zone
$ clever create my-app-dev --type node --alias dev --region par

How to install on MacOS?

brew install CleverCloud/tap/clever-tools

Autocompletion

# In ~/.bash_profile
./usr/local/etc/bash_completion

# In ~/.zshrc
fpath=(/usr/local/share/zsh-completions $fpath)

How to install on Windows?

choco install clever-tools

For GNU/Linux

Autocompletion

# for bash
clever --bash-autocomplete-script $(which clever) | sudo tee /usr/share/bash-completion/completions/clever

# for zsh
clever --zsh-autocomplete-script $(which clever) | sudo tee /usr/share/zsh/site-functions

In case of libcurl version mismatch

# install libcurl-compat
pacman -S libcurl-compat # on archlinux

# add an alias to start clever tools with a compatible libcurl version
alias clever='LD_PRELOAD=libcurl.so.3 clever'

Manual installation (advanced)

That's all folks!

Since its first release, we've been quietly improving our CLI tool, clever-tools. Today, we've released version 0.7.0, with better support for node 7 and many improvements.

New features in 0.7.0

The most important change is better support for node 7. The installation phase is now way faster for node 7 users.

Another important feature is clever ssh, available in preview mode for clever-tools users. clever ssh allows you to ssh into a running instance for debugging purposes (eg. list files, view logs locally, …). Please get in touch with the support if you want to try it out. We'll release complete documentation and examples once this feature hits GA.

Why use Clever Tools?

The web console is nice to use and doesn't require any installation. Why bother with clever-tools then? Here are a few use cases where clever-tools really shines.

Perfect for terminal-heavy multitaskers

If, like me, you're working on several projects in several terminal windows, going back to the web console to check logs and deployments can get tedious. With clever-tools, you can work on different projects without ever leaving your comfy terminal window. You can launch 5 deployments in parallel and see how they go, side-by-side without having to switch from app to app in your browser.

Fine-grained access to logs

The web console is great for displaying logs as they come, but reading old logs requires a lot of scrolling. If you want to inspect logs around a specific date/time to investigate e.g. a user report, worry not:

clever logs --after "2017-02-08T08:28" --before "2017-02-08T08:32"

Also, if you want to search for a specific term you can just grep the output:

clever logs | grep NullPointerException

The best part: new lines will appear in real time.

No SSH configuration necessary

From version 0.6.0 upwards, clever deploy pushes over HTTPS and doesn't require SSH configuration. If your platform makes ssh configuration more difficult than it should be (I'm looking at you, Windows), then just use clever deploy.

What's a Webhook though?

A webhook in web development is a method of augmenting or altering the behavior of a web page, or web application, with custom callbacks. These callbacks may be maintained, modified, and managed by third-party users and developers who may not necessarily be affiliated with the originating website or application.

Neat! How do you do that?

Coming soon

Two things, today: the Clever Cloud public API is here and so is the Clever CLI. Let's have a look at these new things we're thrilled to release today!

Clever API

Since the beginning of Clever Cloud, we have been working exclusively with our own internal API. We didn't open it to the public immediately. We wanted to bring it further, with enough iterations to get a stable release.

Getting started with the API

The API uses OAuth v1 for authentication. The first step to start playing with the API, is to get your tokens: https://console.clever-cloud.com/users/me/tokens.

Once you have your tokens you can interact with the API. To see what you can do with the API, here is the interactive reference: Clever API reference. Don't forget to log-in to be able to interact with the API directly from the documentation.

Clever Tools

The Clever Tools is a CLI (Command-Line Interface) for Clever Cloud. This tool allows you to manage your addons and applications directly from the terminal. Think of it as a lightweight, text-based alternative to the console. It's also great if you want to script your interactions with clever cloud. It's not feature-complete compared to the console yet, so feel free to tell us what you need from it.

Installation and Setup

The installation and setup are pretty straightforward:

• Type npm install clever-tools -g in your terminal. It will download and install it. If you don't have npm, consider installing node.js.
• Once it's installed, type clever login and follow the instructions.
• Navigate in one of your apps' Git repository and enter clever link "My Organisation/My application" (also works with your App ID). This will link your local repo to the Clever Cloud application. You can use --alias <my-alias> if you need to deploy the same code base to several Clever Cloud applications (usually for production / pre-production settings).

Here's a quick overview of what you can do with it.

# Deploying on the production server
$ clever deploy --alias prod

# Displaying the (live-updated) status of an app
$ clever activity --follow --alias preprod

# Show the logs
$ clever logs

# Create a free plan PG addon named my-addon in the Montreal zone
$ clever addon create postgresql-addon my-addon-pg --plan dev --region mtl

# Create a node.js app named my-app-dev in the Paris zone
$ clever create my-app-dev --type node --alias dev --region par

Note: The available types of apps are:

• apache+php54
• apache+php55
• docker
• go
• java+maven
• java+play1
• java+war
• node
• python27
• ruby
• sbt
• static

Contributions Welcome

Open-Source is something really important to us. That's why we're releasing the Clever Tools on our Github. It's totally open to contribution, the repository is here.

Here's some other repositories, also open to contribution:

• clever-client: JavaScript client for Clever-Cloud API
• doc.clever-cloud.com: The official Clever Cloud documentation

Happy coding!