Superset is fast, lightweight, intuitive, and loaded with options that make it easy for users of all skill sets to explore and visualize their data.

How to deploy Superset

# Clone last superset release
git clone --depth 1 -b 0.38.1  https://github.com/apache/superset.git

# Move in superset repository
cd superset

# Create the python application
clever create --type python Superset

# Create the PostgreSQL instance                                
clever addon create postgresql-addon --plan m SupersetPG
    
# link the addon 
clever service link-addon SupersetPG

Superset environment

# Use gunicorn as python mode 
clever env set CC_PYTHON_BACKEND gunicorn

# Start the app using create_app() 
clever env set CC_PYTHON_MODULE superset.app:create_app()

# Python version to use
clever env set CC_PYTHON_VERSION 3.7

# Load all superset requirements
clever env set CC_PIP_REQUIREMENTS_FILE requirements/base.txt

# Application port
clever env set PORT 8080

# PYTHONPATH
clever env set PYTHONPATH /home/bas/YOUR_APP_ID/config/

# App secret key
clever env set SECRET_KEY A_SECRET_KEY

# Post build commands
clever env set CC_POST_BUILD_HOOK ./init.sh

# Superset admin password
clever env set ADMIN_PASSWORD AN_ADMIN_PASSWORD

Configure superset

import os
# Superset specific config
ROW_LIMIT = 5000

SUPERSET_WEBSERVER_PORT = os.getenv("PORT")
SUPERSET_WEBSERVER_ADDRESS = "0.0.0.0"

# Flask App Builder configuration
# Your App secret key
SECRET_KEY = os.getenv("SECRET_KEY")

# The SQLAlchemy connection string to your database backend
# This connection defines the path to the database that stores your
# superset metadata (slices, connections, tables, dashboards, ...).
# Note that the connection information to connect to the datasources
# you want to explore are managed directly in the web UI
SQLALCHEMY_DATABASE_URI = os.getenv("POSTGRESQL_ADDON_URI")

# Flask-WTF flag for CSRF
WTF_CSRF_ENABLED = True
# Add endpoints that need to be exempt from CSRF protection
WTF_CSRF_EXEMPT_LIST = []
# A CSRF token that expires in 1 year
WTF_CSRF_TIME_LIMIT = 60 * 60 * 24 * 365

# Set this API key to enable Mapbox visualizations
MAPBOX_API_KEY = ''

Superset build scipt

#!/bin/bash

# Load configuration file in PYTHONPATH
mkdir $APP_HOME/config
cp clever_config.py $APP_HOME/config/superset_config.py

# Create an admin user (you will be prompted to set username, first and last name before setting a password)
superset fab create-admin \
    --username admin \
    --firstname admin \
    --lastname admin \
    --email admin@admin.com \
    --password ${ADMIN_PASSWORD}

# Upgrade the database
superset db upgrade

# Loading examples data set (optional)
superset load_examples

# Create default roles and permissions
superset init

# Build frontend
cd superset-frontend 
npm install -f --no-optional
npm run build

Superset requirements

echo "psycopg2>=2.7 --no-binary psycopg2" >> requirements/base.txt

pip3 freeze > requirements.txt 

echo "psycopg2>=2.7 --no-binary psycopg2" >> requirements.txt

# gunicorn valid versions are located in setup.py file
echo "gunicorn>=20.0.2, <20.1" >> requirements.txt

Deploy

chmod u+x init.sh

# Add your files
git add .

# Create the first commit
git commit -m "clever init"

# Deploy the application
clever deploy

# Run the application
clever open

At Clever Cloud, we want developers to be happy. To achieve that we try to adapt as quickly as we can to users needs. The other day, someone asked for Sidekiq support for their Rails applications.

What's Sidekiq

Sidekiq is a solution to run long background tasks in ruby. It works by running a background server and treating jobs stored in a Redis. It's helpful to delay tasks, take a long API call to the background while still be able to answer to a request in a timely manner.

How do I use Sidekiq on Clever Cloud?

First, you need a Redis instance. Start by provisioning a redis addon on Clever Cloud. Remember to link the redis add-on to your application in the Console.

The redis addon provides a REDIS_URL environment variable which will be used by sidekiq to connect to the redis addon.

• The next and only step is to add/modify the clevercloud/ruby.json file to add the "sidekiq" flag:

{
   "deploy": {
      "sidekiq": true
   }
}

Then commit and push your application, and you should be all set!

At Clever Cloud, we use a lot of Java. A large part of our API is written using JEE (aka Java Enterprise Edition). Here is how we implemented the http client we use for our Github integration.

The standard HTTP implementation in JEE is Jersey, so that's what we'll use here.

A bit of background first

Our API is designed in a "micro-services" way. There's a central part and various external components.

Here, we'll be dealing with the central part, which itself is internally designed in a sort-of "micro-services" way. JEE offers a powerful dependency injection mechanism called CDI, which we use to build a lot of small tools we can then use just as we need.

Serialization

When you're building APIs and clients, one of the problematics is about serialization. We made a tool to do just that which you'll see in the code as MapperHelper, namely mh. It's basically a wrapper around jackson's ObjectMapper.

Let's get to work

We'll be working on an internal tool called GithubAPIHelper. We'll not get through the whole of it, but you'll quickly get the concept.

First, some standard JEE boilerplate injecting required tools using CDI.

@Stateless
public class GithubAPIHelper {

   @Inject
   private MapperHelper mh;
   @Inject
   private LogsHelper lh;

   private final static String GITHUB_URL = "https://api.github.com";
}

Now, we want to build an HTTP client using jersey-client, this is pretty straightforward. Btw, we create a method to make the client directly point at Github.

private Client getClient() {
   if (client == null) {
      client = ClientBuilder.newClient(new ClientConfig().property(ClientProperties.FOLLOW_REDIRECTS, true));
   }
   return client;
}

private WebTarget getTarget() {
   return getClient().target(GITHUB_URL);
}

What about authentication? Let's create a new method to get an authenticated request to Github:

private Invocation.Builder getBuilder(WebTarget target, String token) {
   return target.request().header("Authorization", "token " + token);
}

Ok, now that we have those basic pieces in place, we can start requesting for real.

Let's get the basic user profile, for starters

First, we create a basic data structure that we'll use to deserialize the payload from Github. I'll skip most of the available fields here as they're not needed.

public class GithubOwner implements Serializable {

   public int id;
   public String name;
   public String login;
   public String email;
   public String avatar_url;
}

Now, you should all be yelling at me about those public fields. Let me remember you that this data structure is only used for deserializing purpose, it won't ever be used for anything else, so what's the point in complicating things?

Now let's get back to business, we have a way to construct authenticated requests, and a model in which to store the data. So let's do it!

First, let's get to the right endpoint: /user

   getTarget().path("/user");

Now, let's actually issue the request, and get a response

   Response r = getBuilder(getTarget().path("/user"), token).get();

Let's deserialize the data from github

   GithubOwner owner = mh.readValue(r.readEntity(String.class), GithubOwner.class);

And here we are, with some integrity checks added:

   public GithubOwner getSelf(String token) {
      Response r = getBuilder(getTarget().path("/user"), token).get();
      if (r.getStatus() < 300) {
         try {
            GithubOwner owner = mh.readValue(r.readEntity(String.class), GithubOwner.class);
            return owner;
         } catch (IOException e) {
            lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
         }
      }
      return null;
   }

Dealing with pagination

Getting the user's profile was quite easy and straightforward, but everything isn't that smooth.

Let's say we now want all the repositories, instinctively, here is what we'd do:

public class GithubRepository implements Serializable {

   public long id;
   public String name;
   public String description;
   public GithubOwner owner;
   @JsonProperty(value = "private")
   public boolean isPrivate;
   public String ssh_url;
   public String git_url;
}

   private TypeSafeList<GithubRepository> getSelfRepositories(String token) {
      TypeSafeList<GithubRepository> repos = new TypeSafeArrayList<>();
      Response r = getBuilder(getTarget().path("/user/repos"), token).get();
      if (r.getStatus() < 300) {
         try {
            List<GithubRepository> ms = mh.readValue(r.readEntity(String.class), new TypeReference<List<GithubRepository>>() {
            });
            repos.addAll(new TypeSafeArrayList<>(ms));
         } catch (IOException e) {
            lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
         }
      }
      return repos;
   }

That will actually work, but you won't get all of them, only the first 20 or so. If you want all of them, you'll have to follow the pagination.

First, we want to get the link marked as rel=next in the Link http header.

   private String getNextLink(Response r) {
      String link = r.getHeaderString("Link");   /* <http://foobar>; rel="next", <http://blah/>; rel=last */
      if (link != null) {
         String[] links = link.split(",");
         for (String l : links) {               /* <http://foobar>; rel="next" */
            if (l.contains("rel=\"next\"")) {
               String[] tmp1 = l.split("<", 2);
               if (tmp1.length == 2) {
                  return tmp1[1].split(">", 2)[0];  /* http://foobar */
               }
            }
         }
      }
      return null;
   }

Next, we want to request this url and get the next data to our initial request

   private Response getNextData(Response r, String token) {
      String link = getNextLink(r);
      if (link == null)
         return null;
      return getBuilder(getClient().target(link), token).get();
   }

Now let's put all the pieces together, addind a loop to retrieve all the data

   private TypeSafeList<GithubRepository> getSelfRepositories(String token) {
      TypeSafeList<GithubRepository> repos = new TypeSafeArrayList<>();
      Response r = getBuilder(getTarget().path("/user/repos"), token).get();
      while (r != null) {
         if (r.getStatus() < 300) {
            try {
               List<GithubRepository> ms = mh.readValue(r.readEntity(String.class), newhTypeReference<List<GithubRepository>>() {
               });
               repos.addAll(new TypeSafeArrayList<>(ms));
            } catch (IOException e) {
               lh.log(GithubAPIHelper.class.getName(), Level.SEVERE, null, e);
            }
         }
         r = getNextData(r, token);
      }
      return repos;
   }

And voila, you now have a working Github HTTP client. Once you got these, the other endpoints are really easy.

Hi, I’m Leonard. I have been an intern developer at Clever Cloud for one month now. When I came in, the CEO put me on a Scala project the first day. As a software engineer, there’s often a pressure to know everything, and I had never done scala before. My only experience in programming was html, php, javascript, and a little bit of java. So I had to dive into this new language quite quickly.

The thing is, Scala is a functional programming language. And the most brutal change was to learn this new paradigm. If you don't know what functional programming is, there is something you need to understand. In FP, the trick is to do everything only with functions (as in math function). Define the purest function that you can.

But you'll ask me, what is the interest of a pure function? A pure function is very useful because it can be tested separately, and especially because it's very reusable!

What’s a pure function?

A pure function takes a value, returns an other value and does nothing else. For a given argument, it always returns the same result. You don't want any side effect like database calls, etc. But there’s some help from the compiler: Scala has a strong static type system. Be sure the compiler will yell at you if you try to bypass it.

At first sight, Scala can look like an awful programming language, especially if you only code in javascript or php. But after a couple of weeks, by trying hard and being bashed a lot by the compiler, your code gets more and more structured. You finally get used to pure functions, and employ less and less side effects and mutating values.

After one month, I did not become a professional in Scala, but I’m now more and more comfortable with it and I find pleasure in coding. I began to integrate the abilities of FP principles and their great potential. Think about all the possibilities it can offer: it is almost endless.

Getting started with Scala

To learn scala, it’s useful to have some tooling in order to evaluate your code. Check the download page to get typesafe activator. If you have a favorite IDE, I recommend to keep it. Else I personally use vim or sublime text, where Scala syntax highlighting works out of the box.

First, there is more than one way to learn scala. A good way is to look at concrete exercises. Try to rewrite a basic mathematical function or some exercises like this one.

"I have x employees in my office. I want to have the sum of the salary of all Senior employees"

First we need to create the Employee class

case class Employee(name: String, title: String, salary: Int)

Then we create a list of these salary

val employees: List[Employee] = List(
  Employee("Smith", "Junior Developer", 1250),
  Employee("George", "Senior Analyst", 2000),
  Employee("David", "Cadet Developer", 1000),
  Employee("Bernard", "Senior Developer", 2100),
  Employee("Clement", "Senior Developer", 1850)
)

I then define a function which takes an Employee as a parameter, and returns true if the employee title begins with "Senior".

def isSenior(employee: Employee) = employee.title.startsWith("Senior")

Finally, we just apply isSenior on the list of employees to keep the senior employees, and then sum their salaries.

val seniorSalaries =
   employees
     .filter(isSenior _)
     .map(_.salary)
     .sum

To explain, we apply the filter method to our list. filter is a native function which selects all elements which satisfy a predicate (here the isSenior function). Then we map over it. map transforms a list of elements of type A into a list of elements of type B, by applying a function from A to B to all its elements. Here, we transform a List[Employee] into a List[Int] by getting the salary of each employee. Finally, we do a sum on the list of salaries.

As you can see, we solved the problem with small pure functions. They are reusable regardless of the context and we were able to compose them with functions provided by the scala standard library.

If you want to dig deeper into Scala, check this book

If you have any questions, don’t hesitate to send me an email at leonard.imbert@clever-cloud.com