clever tools Archives | Clever Cloud From Code to Product Tue, 24 Oct 2023 09:01:53 +0000 en-GB hourly 1 https://staging-cc-assetsv6.cellar-c2.services.clever-cloud.com/uploads/2023/03/cropped-cropped-favicon-32x32.png clever tools Archives | Clever Cloud 32 32 The story of clever-tools 1.0.0 https://stagingv6.cleverapps.io/blog/features/2018/10/15/the-story-of-clever-tools-1-0-0/ Mon, 15 Oct 2018 16:00:00 +0000 https://www2.cleverapps.io/wp/blog/technology/2018/10/15/the-story-of-clever-tools-1-0-0/

We have some news about our dear clever-tools, the Clever Cloud CLI for humans, robots and others.

Back in January, we released version 0.10.1 which contained some new commands and a few bug fixes. Today we're releasing version 1.0.0!

You can find all the details about the user oriented changes in the CHANGELOG. In this article, we'll cover the structural changes related to the project:

  • Improved error handling = better usage in CI/CD pipelines
  • ES2015 refactor + ESLint = better contributing experience
  • Goodbye nodegit, hello isomorphic-git!
  • Improved Jenkins build = more release methods and beta channel (npm, deb, rpm, arch, homebrew and chocolatey)

Now let me tell you the story of this release...

Improved error handling = better usage in CI/CD pipelines

It all started with issue 195. With the previous version, when a user tried to clever deploy a non existing branch, the error message was a bit too raw. Our colleague Philippe wanted emojis, but we had to decline 😭.

While investigating this error message and how to improve it, we realized that the command was not returning a proper error exit status. This is a very important rule to follow when you're writing a terminal command/script:

Our CLI is built for humans AND robots. We want to empower our terminal addicted users. Moreover, we want to empower our users to automate anything related to Clever Cloud in their CI/CD pipeline. In such cases, returning a proper error exit status is crucial to mark a CI/CD job as successful or failed.

We took a quick look at the rest of the codebase and noticed that other commands were concerned by this problem. We reworked the code of clever deploy and decided to apply the same error handling to all the other commands:

  • For humans: Always have an [ERROR] prefix (colored in red) when we display an error message.
  • For robots: Always log error messages to stderr.
  • For robots: Always return 1 as exit status when something goes wrong.

This obviously required some significant work. We had to go through all the 28 commands. Nevertheless, we're sure this will help you to build more robust and automated jobs, interacting with the Clever Cloud platform.

ES2015 refactor + ESLint = better contributing experience

While we went through the codebase, we took the opportunity to do a bit of refactoring.

First, we decided to update all our code to ES2015+ syntax. We tried to use most of the modern good practices of the JavaScript ecosystem when it made sense (destructuring, rest parameters, fat arrows...). We also decided to setup an ESLint config file (based on StandardJS) to enforce our codestyle for the next iterations. This lint check is now part of our automated test suite on Travis CI which is triggered on each pull-request.

This big refactoring took some time but it allowed us to reduce the number of lines of code while improving the overall consistency and readability. We hope this will drive more users to customize the clever-tools and propose pull-requests for bug fixes and feature requests.

Looking at how easily the new clever console was added by an outside contributor, it seems like we're on the right path.

Goodbye nodegit

During this refactoring, we updated most of our dependencies but one of them required special attention: NodeGit.

NodeGit is a node.js module which provides bindings over libgit2, a portable and pure C implementation of the git core methods. This is commonly referred to as a "native module" in the node.js ecosystem. Each time you install NodeGit, npm will try to build libgit2 from source (in some situations, it will try to download pre-builds). Because we want our users to have a straightforward installation experience, this native module thing had many drawbacks:

  • In our Travis CI config, we had to install some native dependencies so npm could build libgit2 from source.
  • Some of our users tried to install the clever-tools via npm and had difficulties because of this.
  • Our users don't all use the same version of node.js, this also multiplied the number of weird cases we had to support.
  • Each time we wanted to update NodeGit, it was a puzzle to resolve.

All those problems pushed us to stop releasing the clever-tools via npm. ⚠️ SPOILER ALERT: it's back for version 1.0.0.

The second problem is that we use pkg. This tool is great. It lets you compile a node.js project to different portable binaries for MacOS, Windows and GNU/Linux. We've been using it for a while to provide various installation methods. When we dropped installation via npm for 0.10.1, the binaries built with pkg where the only official way to use the clever-tools.

This mix of NodeGit + pkg complicated the situation even more:

  • The binaries built with pkg did not include native dependencies. We had to distribute the right nodegit.node file along with our main binary file and users had to have it in their PATH.
  • On some GNU/Linux distributions, we had problems with native dependencies like libcurl-gnutls, libnghttp2-git or libssh2

We already looked for a replacement solution but a fairly new one came up to our attention in February...

Hello isomorphic-git

isomorphic-git is a "pure JavaScript implementation of git that works in node and browser environments". The project was created by William Hilton. This guy is awesome!

As I said, we heard about it in February but we only tried to use it for our use-case in July. It does not support all the git feature set (yet) but it has strong foundations and we only need a few things for the clever-tools:

  • list and resolve branches
  • list and resolve commits
  • add/rm remotes
  • push to repos (via HTTPS)

We were missing a few features so we reached out to William to see if we could help and/or contribute. In the end, we contributed a few features in the codebase and we sponsored the hosting of the CORS-proxy which is used to demo the project in browsers.

By replacing NodeGit with isomorphic-git, we can say goodbye to most of the problems we described earlier:

  • no need to worry about different node.js versions
  • no need to have native dependencies on our CI server to build/test the project
  • no need to distribute nodegit.node with the pkg binary
  • no need for a user to have the right native dependencies on his/her system to use it
  • npm releases are back

Improved Jenkins build = more release methods and beta channel

Because we decided to ditch NodeGit, we were able to simplify the packaging (no more nodegit.node and no native dependencies). This required some rework on our Jenkins builds but it was a good occasion to migrate 5 different jobs to one multibranch pipeline project.

We'll discuss the details of this new build system and how we set it up in another article. For now though, here's what changed:

  • All platforms:
    • Added: installation via npm is back
  • GNU/Linux:
  • MacOS:
  • Windows:
    • Changed: chocolatey packages are now automatically published (no more manual upload, no more manual review from chocolatey's team)
    • Changed: chocolatey packages are no longer published on chocolatey.org
    • Changed: chocolatey packages are published (beta & stable) on Bintray but you can still use the choco CLI to install them (see chocolatey installation docs for the specifics).

The structural changes of this release will help us to be more reactive on the project. We'll be able to publish new features and bug fixes more easily and more often. Hopefully, this will improve the quality of the clever-tools and push more customers who only use the Web console to try the CLI way of life.

We're looking forward to read your feedbacks on this release through are usual channels: support chat, email, Twitter, GitHub issues…

]]>

We have some news about our dear clever-tools, the Clever Cloud CLI for humans, robots and others.

Back in January, we released version 0.10.1 which contained some new commands and a few bug fixes. Today we're releasing version 1.0.0!

You can find all the details about the user oriented changes in the CHANGELOG. In this article, we'll cover the structural changes related to the project:

  • Improved error handling = better usage in CI/CD pipelines
  • ES2015 refactor + ESLint = better contributing experience
  • Goodbye nodegit, hello isomorphic-git!
  • Improved Jenkins build = more release methods and beta channel (npm, deb, rpm, arch, homebrew and chocolatey)

Now let me tell you the story of this release...

Improved error handling = better usage in CI/CD pipelines

It all started with issue 195. With the previous version, when a user tried to clever deploy a non existing branch, the error message was a bit too raw. Our colleague Philippe wanted emojis, but we had to decline 😭.

While investigating this error message and how to improve it, we realized that the command was not returning a proper error exit status. This is a very important rule to follow when you're writing a terminal command/script:

Our CLI is built for humans AND robots. We want to empower our terminal addicted users. Moreover, we want to empower our users to automate anything related to Clever Cloud in their CI/CD pipeline. In such cases, returning a proper error exit status is crucial to mark a CI/CD job as successful or failed.

We took a quick look at the rest of the codebase and noticed that other commands were concerned by this problem. We reworked the code of clever deploy and decided to apply the same error handling to all the other commands:

  • For humans: Always have an [ERROR] prefix (colored in red) when we display an error message.
  • For robots: Always log error messages to stderr.
  • For robots: Always return 1 as exit status when something goes wrong.

This obviously required some significant work. We had to go through all the 28 commands. Nevertheless, we're sure this will help you to build more robust and automated jobs, interacting with the Clever Cloud platform.

ES2015 refactor + ESLint = better contributing experience

While we went through the codebase, we took the opportunity to do a bit of refactoring.

First, we decided to update all our code to ES2015+ syntax. We tried to use most of the modern good practices of the JavaScript ecosystem when it made sense (destructuring, rest parameters, fat arrows...). We also decided to setup an ESLint config file (based on StandardJS) to enforce our codestyle for the next iterations. This lint check is now part of our automated test suite on Travis CI which is triggered on each pull-request.

This big refactoring took some time but it allowed us to reduce the number of lines of code while improving the overall consistency and readability. We hope this will drive more users to customize the clever-tools and propose pull-requests for bug fixes and feature requests.

Looking at how easily the new clever console was added by an outside contributor, it seems like we're on the right path.

Goodbye nodegit

During this refactoring, we updated most of our dependencies but one of them required special attention: NodeGit.

NodeGit is a node.js module which provides bindings over libgit2, a portable and pure C implementation of the git core methods. This is commonly referred to as a "native module" in the node.js ecosystem. Each time you install NodeGit, npm will try to build libgit2 from source (in some situations, it will try to download pre-builds). Because we want our users to have a straightforward installation experience, this native module thing had many drawbacks:

  • In our Travis CI config, we had to install some native dependencies so npm could build libgit2 from source.
  • Some of our users tried to install the clever-tools via npm and had difficulties because of this.
  • Our users don't all use the same version of node.js, this also multiplied the number of weird cases we had to support.
  • Each time we wanted to update NodeGit, it was a puzzle to resolve.

All those problems pushed us to stop releasing the clever-tools via npm. ⚠️ SPOILER ALERT: it's back for version 1.0.0.

The second problem is that we use pkg. This tool is great. It lets you compile a node.js project to different portable binaries for MacOS, Windows and GNU/Linux. We've been using it for a while to provide various installation methods. When we dropped installation via npm for 0.10.1, the binaries built with pkg where the only official way to use the clever-tools.

This mix of NodeGit + pkg complicated the situation even more:

  • The binaries built with pkg did not include native dependencies. We had to distribute the right nodegit.node file along with our main binary file and users had to have it in their PATH.
  • On some GNU/Linux distributions, we had problems with native dependencies like libcurl-gnutls, libnghttp2-git or libssh2

We already looked for a replacement solution but a fairly new one came up to our attention in February...

Hello isomorphic-git

isomorphic-git is a "pure JavaScript implementation of git that works in node and browser environments". The project was created by William Hilton. This guy is awesome!

As I said, we heard about it in February but we only tried to use it for our use-case in July. It does not support all the git feature set (yet) but it has strong foundations and we only need a few things for the clever-tools:

  • list and resolve branches
  • list and resolve commits
  • add/rm remotes
  • push to repos (via HTTPS)

We were missing a few features so we reached out to William to see if we could help and/or contribute. In the end, we contributed a few features in the codebase and we sponsored the hosting of the CORS-proxy which is used to demo the project in browsers.

By replacing NodeGit with isomorphic-git, we can say goodbye to most of the problems we described earlier:

  • no need to worry about different node.js versions
  • no need to have native dependencies on our CI server to build/test the project
  • no need to distribute nodegit.node with the pkg binary
  • no need for a user to have the right native dependencies on his/her system to use it
  • npm releases are back

Improved Jenkins build = more release methods and beta channel

Because we decided to ditch NodeGit, we were able to simplify the packaging (no more nodegit.node and no native dependencies). This required some rework on our Jenkins builds but it was a good occasion to migrate 5 different jobs to one multibranch pipeline project.

We'll discuss the details of this new build system and how we set it up in another article. For now though, here's what changed:

  • All platforms:
    • Added: installation via npm is back
  • GNU/Linux:
  • MacOS:
  • Windows:
    • Changed: chocolatey packages are now automatically published (no more manual upload, no more manual review from chocolatey's team)
    • Changed: chocolatey packages are no longer published on chocolatey.org
    • Changed: chocolatey packages are published (beta & stable) on Bintray but you can still use the choco CLI to install them (see chocolatey installation docs for the specifics).

The structural changes of this release will help us to be more reactive on the project. We'll be able to publish new features and bug fixes more easily and more often. Hopefully, this will improve the quality of the clever-tools and push more customers who only use the Web console to try the CLI way of life.

We're looking forward to read your feedbacks on this release through are usual channels: support chat, email, Twitter, GitHub issues…

]]> Clever Tools Packaged! https://stagingv6.cleverapps.io/blog/features/2017/09/06/clever-tools-packaged-windows-macos-gnu-linux/ Wed, 06 Sep 2017 11:47:00 +0000 https://www2.cleverapps.io/wp/blog/technology/2017/09/06/clever-tools-packaged-windows-macos-gnu-linux/

We have packaged the Clever Tools CLI for Windows, MacOs & GNU/Linux. This makes it way easier to setup. Discover how to do it for these platforms.

Clever Tools should better be called Clever Cool

First of all, what is Clever Tools? Ok, basically it's all what you can do in the console.clever-cloud.com web UI but through a command line tool. How is that awesome? Well, it is. For most of our users, going back and forth in the web console is slowing their development workflow. Even if the web console is indispensable for people more comfortable with it, we wanted our users to be able to exactly to do the same in both. By the way, did you know you can also use the API to do everything? Anyway, the cool part with Clever Tools is that you stay in the comfy shell. You can use Clever Tools along with vim, git and even grep. That's pretty handy to manage logs for instance. Earlier this year, we released new features in the CLI 0.7.0 such as fine-grained access to logs, ssh into a running instance for debugging purposes (eg. list files, view logs locally,...) So here is a short review of whay you can do: 
# Deploying on the production server
$ clever deploy --alias prod
# Displaying the (live-updated) status of an app
$ clever activity --follow --alias preprod
# Show the logs
$ clever logs
# Create a free plan PG addon named my-addon in the Montreal zone
$ clever addon create postgresql-addon my-addon-pg --plan dev --region mtl
# Create a node.js app named my-app-dev in the Paris zone
$ clever create my-app-dev --type node --alias dev --region par
But you can also set notifications, start and stop your apps... well, as we said: everything! If you want to dig it a bit more, you will find everything you can do within the CLI. Ok, but I already knew all that! Ok, now what's new is that we packaged Clever Tools for Windows, GNU/Linux and MacOS. But why is that a big deal? Before that, you had to download and install node and npm to install clever tools. Now it's over, it's all packaged in a binary and very easy to setup.

How to install on MacOS?

Clever Tools is packaged using Homebrew: 
brew install CleverCloud/tap/clever-tools
If you don't want to use brew, a pre-compiled version is available: clever-tools-0.9.0_macos.tar.gz. You need to put both files (clever and nodegit.node) in your PATH to use the application.

Autocompletion

Clever Tools comes with a comprehensive auto-completion system. The brew package installs it automatically (for bash and zsh). Make sure bash-completions or zsh-completions are properly set up. 
# In ~/.bash_profile
./usr/local/etc/bash_completion

# In ~/.zshrc
fpath=(/usr/local/share/zsh-completions $fpath)
Documentation.

How to install on Windows?

Clever Tools is packaged using Chocolatey: 
choco install clever-tools
If you don't want to use chocolatey, a pre-compiled version is available: clever-tools-0.9.0_windows.zip. You need to add both files (clever.exe and nodegit.node) to your PATH to use the application. Documentation.

For GNU/Linux

Now what, we are still hesitating on the distributions to package it for. We would like to have your feedback on that. Feel free to give your point of view in the comments below are by email @ support@clever-cloud.com. Also, if someone wants to manage a PPA, that would be great. A pre-compiled version is available: clever-tools-0.9.0_linux.tar.gz. You need to add both files (clever and nodegit.node) to your PATH to use the application.

Autocompletion

Clever Tools comes with a comprehensive auto-completion system, this is one way to install it: 
# for bash
clever --bash-autocomplete-script $(which clever) | sudo tee /usr/share/bash-completion/completions/clever

# for zsh
clever --zsh-autocomplete-script $(which clever) | sudo tee /usr/share/zsh/site-functions

In case of libcurl version mismatch

On some distributions (eg. archlinux), a libcurl version mismatch between the system default and the expected version can happen. If you see Error: /usr/lib/libcurl.so.4: version 'CURL_OPENSSL_3' not found (required by /usr/bin/nodegit.node) in the logs, you can fix the issue by using a compatible libcurl version: 
# install libcurl-compat
pacman -S libcurl-compat # on archlinux

# add an alias to start clever tools with a compatible libcurl version
alias clever='LD_PRELOAD=libcurl.so.3 clever'

Manual installation (advanced)

Even though we provide packaged versions, you can still install Clever Tools with npm: npm install -g clever-tools. You can also install it from source.

That's all folks!

We hope you will enjoy these improvements and use more Clever Tools. As always, feel free to contact us for any feedback. Have a look at the open issues.]]>

We have packaged the Clever Tools CLI for Windows, MacOs & GNU/Linux. This makes it way easier to setup. Discover how to do it for these platforms.

Clever Tools should better be called Clever Cool

First of all, what is Clever Tools? Ok, basically it's all what you can do in the console.clever-cloud.com web UI but through a command line tool. How is that awesome? Well, it is. For most of our users, going back and forth in the web console is slowing their development workflow. Even if the web console is indispensable for people more comfortable with it, we wanted our users to be able to exactly to do the same in both. By the way, did you know you can also use the API to do everything? Anyway, the cool part with Clever Tools is that you stay in the comfy shell. You can use Clever Tools along with vim, git and even grep. That's pretty handy to manage logs for instance. Earlier this year, we released new features in the CLI 0.7.0 such as fine-grained access to logs, ssh into a running instance for debugging purposes (eg. list files, view logs locally,...) So here is a short review of whay you can do: 
# Deploying on the production server
$ clever deploy --alias prod
# Displaying the (live-updated) status of an app
$ clever activity --follow --alias preprod
# Show the logs
$ clever logs
# Create a free plan PG addon named my-addon in the Montreal zone
$ clever addon create postgresql-addon my-addon-pg --plan dev --region mtl
# Create a node.js app named my-app-dev in the Paris zone
$ clever create my-app-dev --type node --alias dev --region par
But you can also set notifications, start and stop your apps... well, as we said: everything! If you want to dig it a bit more, you will find everything you can do within the CLI. Ok, but I already knew all that! Ok, now what's new is that we packaged Clever Tools for Windows, GNU/Linux and MacOS. But why is that a big deal? Before that, you had to download and install node and npm to install clever tools. Now it's over, it's all packaged in a binary and very easy to setup.

How to install on MacOS?

Clever Tools is packaged using Homebrew: 
brew install CleverCloud/tap/clever-tools
If you don't want to use brew, a pre-compiled version is available: clever-tools-0.9.0_macos.tar.gz. You need to put both files (clever and nodegit.node) in your PATH to use the application.

Autocompletion

Clever Tools comes with a comprehensive auto-completion system. The brew package installs it automatically (for bash and zsh). Make sure bash-completions or zsh-completions are properly set up. 
# In ~/.bash_profile
./usr/local/etc/bash_completion

# In ~/.zshrc
fpath=(/usr/local/share/zsh-completions $fpath)
Documentation.

How to install on Windows?

Clever Tools is packaged using Chocolatey: 
choco install clever-tools
If you don't want to use chocolatey, a pre-compiled version is available: clever-tools-0.9.0_windows.zip. You need to add both files (clever.exe and nodegit.node) to your PATH to use the application. Documentation.

For GNU/Linux

Now what, we are still hesitating on the distributions to package it for. We would like to have your feedback on that. Feel free to give your point of view in the comments below are by email @ support@clever-cloud.com. Also, if someone wants to manage a PPA, that would be great. A pre-compiled version is available: clever-tools-0.9.0_linux.tar.gz. You need to add both files (clever and nodegit.node) to your PATH to use the application.

Autocompletion

Clever Tools comes with a comprehensive auto-completion system, this is one way to install it: 
# for bash
clever --bash-autocomplete-script $(which clever) | sudo tee /usr/share/bash-completion/completions/clever

# for zsh
clever --zsh-autocomplete-script $(which clever) | sudo tee /usr/share/zsh/site-functions

In case of libcurl version mismatch

On some distributions (eg. archlinux), a libcurl version mismatch between the system default and the expected version can happen. If you see Error: /usr/lib/libcurl.so.4: version 'CURL_OPENSSL_3' not found (required by /usr/bin/nodegit.node) in the logs, you can fix the issue by using a compatible libcurl version: 
# install libcurl-compat
pacman -S libcurl-compat # on archlinux

# add an alias to start clever tools with a compatible libcurl version
alias clever='LD_PRELOAD=libcurl.so.3 clever'

Manual installation (advanced)

Even though we provide packaged versions, you can still install Clever Tools with npm: npm install -g clever-tools. You can also install it from source.

That's all folks!

We hope you will enjoy these improvements and use more Clever Tools. As always, feel free to contact us for any feedback. Have a look at the open issues.]]>