Earlier this week, we released some changes in how we present our invoices. The result: invoices that help you understand what you pay and what to expect next time.

As an introduction, let's describe our problem: Clever Cloud bills you for the services you used over a given period, and asks you to credit your account in advance to cover the next billing period. The next invoice will consume your prepaid credits and ask for more credits for the next period, and so on. Up until now, your credits balance at a given time was not clearly displayed. That was a problem. The other problem lies in the management of free credits. These free credits have an expiration date. We had to write this explicitly in the invoices.

Now that we have expressed our problem, let's dive into the solution!

A clearer first page

First, we reworded items on the first page. In part to please our new legal service, but mostly because the existing terms were confusing. E.g. where you had "Clever Cloud consumption", you get "Platform usage fees". Add a few tweaks to make this first page easy to understand: we display a subtotal of everything you owe us for using the service (that's the new "Total fees" line). After that we display how much credits where removed or need to be added to your prepaid account. This gives us the grand total to pay.

Explicating free credits

The next change appears on the second page: we give you a breakdown of all the free credits consumed / expired this month and how we compute the credits to remove or to add to the invoice.

On this second page you have: all the free credits you are entitled to, which ones have come to expire, how much you still have left to cover the next billing period. You also have all the variables we use to compute the added or consumed credits that show on the first page.

This second page will help you keep track of your prepaid credits balance and have a better understanding of why we charge you this specific amount.

With these changes, your accountant should be happy! Hopefully, this will help them understand our invoices and give them time they can spend trying to figure out invoices of your other providers! 😇

In this article, we will learn how to configure JAAS authentication in a JVM application without resorting to configuration files. This is motivated by the fact that, in a cloud setup, persisting a configuration file to an ephemeral instance/container is not a good practice.

The problem

You are writing an application that depends on a ZooKeeper cluster. The moment comes when you need to put your code into production. During the early development of your application you were running a ZooKeeper instance on your laptop. Now that you are deploying to production, you need a secure connection between your app and the production ZooKeeper cluster you just set up.

Reading the ZooKeeper documentation you learn that you can achieve that through multiple ways: X509, SASL Kerberos, SASL Digest. Since you just want to demonstrate your POC, you have no wish to take on the burden of using X509 client certificates or to deploy a Kerberos architecture. So you fallback to SASL Digest. It's simple and easy, it's a user/password pair like you are used to.

Reading some more documentation you learn that you have to put the credentials in a jaas.conf file. You then look into the internet to find a way to bypass this requirement.

Well, your search is over, your quest ends with this very article!

You just want to copy some code and don't care about the whole "let's talk about X" part? Jump to the "How to get rid of jaas.config file" section!

Some explanations first

Since Java 1.4, a security framework has been added to the JRE: the Java Authentication and Authorization Service (JAAS).

JAAS is a quite comprehensive and pluggable framework. The only downside is that it's thought as a service meant to authenticate "the user running the code". So it has a tendency to use configuration files or even system JVM configuration.

Configuration based on files containing secrets is a growing no-go in the cloud era. Since most clouds rely on disposable systems, using a JAAS file means one of the following:

1. commiting the file, with the secrets in it;
2. commiting a file and use pre-run hooks to set the right value in it;
3. running an external service that would provide this file.

Solution 1 is a strict no-go. Solution 2 is doable, but as a developer, I don't like having to resort to such practices. However, it's still better than solution 1, I'm just stating my very personal point on view! Solution 3 is a bit like solution 2. It needs some kind of pre-run hook. In addition, it needs to trust the external source.

In this article, we will see how to abstract ourselves from the file-based JAAS login configuration. I will use a ZooKeeper Java/Scala client as an example.

What's in the box JAAS file?

First, what is a JAAS file made of?

As the Oracle documentation states, it is made of entries. Each entry is composed of multiple modules. Each module provides a login/authorization mechanism with all its options. An authentication/authorization process has to satisfy all modules of the entry to succeed.

Example

We assume that the server part is already configured. Here we focus on the client part only.

Let's say we want to authenticate a Java application against a ZooKeeper cluster using the DIGEST-MD5 SASL mechanism. We write a jaas.conf file somewhere on the filesystem. Here is what the file content will look like:

Client {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   username="buffysummers"
   password="vampiresbeware";
};

The entry name is Client (the default client session name for ZooKeeper). There is one module identified by the FQDN of the LoginModule class that will process the authentication/authorization. This module is required and has two parameters: username and password. No spaces around the =, the semicolon separates the modules. So only one is needed at the end of the module. One is also needed after the entry.

"Executing" this entry is quite straightforward: Java instanciates the org.apache.zookeeper.server.auth.DigestLoginModule class and calls initialize("subject", callback, sharedState, Map(username=buffysummers, password=vampirebeware)) on the instance. After that, the actual client will use the instance's methods to perform authentication/authorization.

It's the developer's responsibility to ensure the module class is present in the classpath. In the case of ZooKeeper, the library provides the org.apache.zookeeper.server.auth.DigestLoginModule class.

Oh, one last thing about the JAAS file syntax: for each module you must say if the module is required, sufficient, optional, etc. That's what the required keyword is about! That way you can require multiple authentication methods to pass or you can just ask for one of them to pass.

How does it work in the JVM?

When running your application, the client code will use the javax.security.auth.login.Configuration static class to access your configured JAAS. Actually, it will use the only JRE provided implementation com.sun.security.auth.login.ConfigFile, which parses your provided jaas.conf file.

The Configuration singleton provides one method, getAppConfigurationEntry(String entryName). In our example, by default, the ZooKeeper client calls the method with the string "Client".

In most JREs, there is no default JAAS file. That means Configuration has no entries.

How do I tell the JVM where to find jaas.conf?

According to the javadoc, there are multiple ways to do that:

• edit the java.security system file (in $JAVA_HOME);
• run your app with the java option -Djava.security.auth.login.config=path/to/jaas.conf;
• put the file in the default location: $HOME/.java.login.config.

The next section will show us how to setup your JVM using Java code, without relying on the filesystem. It will keep on using the ZooKeeper client example.

How to get rid of jaas.config file

The answer to that is quite simple: override Java's Configuration default implementation!

Disclaimer

Before we dive in the details, please note that There is only one Configuration object installed in the runtime at any given time. When we override it, we remove all previous login configurations. Most JREs do not have any default configuration, so you should be safe. However, if you deploy your application in your company's VMs or container, they may already have specific configuration. So ask around!

Implementing a new Configuration

First, we define a class that implements javax.security.auth.login.Configuration:

import javax.security.auth.login.AppConfigurationEntry;
import javax.security.auth.login.Configuration;
import java.util.HashMap;
import java.util.Map;

public class MyConfiguration extends Configuration {
   AppConfigurationEntry[] getAppConfigurationEntry(s: String) {
      Map<String,String> options = new HashMap<>();
      options.put("username", "buffysummers");
      options.put("password", "vampiresbeware");

      return {
         new AppConfigurationEntry(
               "org.apache.zookeeper.server.auth.DigestLoginModule",
               AppConfigurationEntry.LoginModuleControlFlag.REQUIRED,
               options
               )
      };
   }
}

Of course, you are highly encouraged to get the values for username and password from the environment, from a config database, or whatever you want! You might even add a constructor and feed it the credentials. You do you!

The next step is to instanciate it. You need to do it before you create a new ZooKeeper client instance:

// 1. Do that somewhere *before* creating the ZooKeeper instance.
// I would say do it close enough so other people can understand why you do it, or
// document this call because it is kind of cryptic.
Configuration.setConfiguration(new MyConfiguration());

// 2. Just create the zookeeper instance as usual!
ZooKeeper zookeeper = new ZooKeeper("localhost:2181", 3600000, watchCallback);

// 3. ?

// 4. PROFIT!

And that's it.

Alternative method

When writing this article, I read the documentation a bit more in depth and found out that I could use Java properties in the JAAS configuration file!

So instead of the code-only solution, we could actually just commit a jaas.conf file containing:

Client {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   username="${app.zookeeper.username}"
   password="${app.zookeeper.password}";
};

Then run your Java application with the Java options: -Dapp.zookeeper.username=buffysummers and -Dapp.zookeeper.password=vampirebeware.

I tried it, it works!

Depending on your application, using this solution might be easier than the full-code one.

Conclusion

Now that we understand a bit more about JAAS and how to avoid using the file format, there is actually only one advice to keep in mind: DO NOT STORE CREDENTIALS IN YOUR GIT REPOSITORY! By using one of the two proposed solutions, you should be able to have the secrets provided by the environment.

Besides, both of them should work on Clever Cloud. Your environment variables are injected in your application both as environement and as Java properties! You can also define a variable named app.zookeeper.username. It will only appear as a Java property, due to limitations in environment variables names.

In November 2018, we released the Cellar C2 cluster. Every new cellar account since then uses the new cluster. Previously created add-ons remain on the old one.

Cellar?

Cellar is our S3 add-on. You get the exact same API than S3, but the service is hosted by Clever Cloud. So you can use it with any S3 client. Even the official Amazon ones.

To achieve that, we use an open source implementation.

The differences between the old and the new Cellar

That's a good question. See, at the beginning, we created the first Cellar add-on using Riak CS (then called Riak S2). Riak CS is an S3 API implementation on top of the Riak distributed NoSQL database by Basho.

We used it for quite some time. Then Amazon changed its authentication model, deprecating its Signature v2 algorithm in favor of the new Signature v4. We waited for Basho to update its implementation. Then Basho decided to quit maintaining Riak CS. So no new fancy Signature v4 for us.

With time, Amazon killed Signature v2 by rolling out new AWS APIs clients. It became clear we had to do something.

We had a look at different OSS project and decided to use Ceph.

Ceph is a distributed storage platform that comes with many interfaces:

• Object Store, supporting the S3 and Swift APIs.
• Network Block Device, something we were already interested in.
• Ceph FileSystem, a feature we used at the very beginning of Clever Cloud. It left us with some mixed feeling. We replaced it quickly.

Cellar C2 is now running on Ceph Object Store.

What does that mean, feature-wise? Well, let's compare it together:

The most important feature we gained was obviously the Signature v4 one.

My Cellar add-on is old. Do you plan to migrate it?

For the moment, no. There is no easy way to move from Riak to Ceph. We plan to do it when we get the time. Until then, here is the way to do it:

s3cmd -c old_s3cfg sync s3://bucket/ tmp/
s3cmd -c new_s3cfg sync tmp/ s3://bucket/

Feel free to ask the support (by email or via the console chat) for help and advice about moving your Cellar Bucket!

So, we have been a bit late when it came to supporting PHP 5.6. But we had a good reason for that! We were reworking the PHP scalers to create a single PHP scaler that could handle any PHP versions we would install on it. So we are finally getting PHP 5.6 and 7.0 out in the open.

But that's not all! Previously, selecting your PHP version was done upon application creation once and for all. Changing the PHP version required you to create a new application and to migrate your code (and data).

Now, you can change the PHP version by setting an environment variable and redeploying your application! Just set the PHP_VERSION variable to one of the following values:

• 5.4
• 5.5
• 5.6
• 7.0

Then redeploy your application, et voilà!

You can now test different versions of PHP as much as you want without setting up your environment every time! In the console, select the PHP (beta) application type when creating a new app. Once the beta will be over, you will see only one PHP option instead of PHP 5.4 and PHP 5.5.

To protect your web app or API, there is almost only one way at this time: TLS. But users and browsers don't always use TLS by default. So what you want is to redirect them to a TLS encrypted version of your site if they try to connect via plain http.

Here is how to do it with Play Framework 2.4 in scala:

Play! and HTTP filters

We will start by creating a "TLSFilter.scala" file and write a TLSFilter class in it:

class TLSFilter extends Filter {
  def apply(nextFilter: RequestHeader => Future[Result])
    (requestHeader: RequestHeader): Future[Result] = {
      if(!requestHeader.secure)
        Future.successful(Results.MovedPermanently("https://" + requestHeader.host + requestHeader.uri))
      else
        nextFilter(requestHeader).map(_.withHeaders("Strict-Transport-Security" -> "max-age=31536000; includeSubDomains"))
  }
}

This part is easy: we implement the apply function by just checking the secure value of RequestHeader. If the connection is not secure, we need to redirect the client to the same url only with "https" as the protocol. If the connection is secure, we pass the request to the next header. Nothing simpler.

Note that we use requestHeader.host instead of requestHeader.domain because the host value is actually the value of the Host header as set by the client, with optional port and stuff.

Note that we create a Filter implementation and not an EssentialFilter one because we do not care about the body.

Next, you need to create a HttpFilters implementation that will hold the instance of your TLSFilter:

// In TLSFilter.scala
class MyFilters extends HttpFilters {
  val filters = Seq(new TLSFilter)
}

And finally, you need to tell Play! to use your Filters class:

# In conf/application.conf
play.http.filters=my.package.MyFilters

Now it will check your requests and permamently redirect the clients to HTTPS.

But, how does Play! know that the request is secure?

Reverse proxies: where did the 's' go?

Now, we need to ensure that Play! knows to differentiate a secured connection from a plain one. If you configured HTTPS in your application, it's quite simple to understand. But it is not always the case:

You most probably did not, configure TLS in your application. And that is because you deployed it on a very powerful and developer-friendly PaaS. So, chances are your Play! application is getting requests in plain HTTP, because the TLS encryption ended at the front reverse-proxy that's piping the request towards your app.

How then is your application going to know that the connection is secured? Enter the non-standard and the standard ways.

X-Forwarded-Proto

The first, non-standard but widely used (e.g. at Clever Cloud) way to know if the request handled by the reverse proxy in front came in a secure channel is to check the X-Forwarded-Proto HTTP header. Like all non-standard headers, you can recognize it by the X- at the beginning of the name.

This header describes how the final client is communicating with the reverse-proxy.

It takes two values: http and https. You can check for that header in your application. But we will see below that Play! can do it by itself.

RFC 2739

Also called the Forwarded HTTP Extension, it standardize the way that a proxy tells the final endpoint what is going on between the final client and itself. It's been published in June 2014.

You can read it here: https://tools.ietf.org/html/rfc7239. But the only thing that is relevant for us is the proto parameter. Like X-Forwarded-Proto earlier, its values that interest us are http and https. Like for the other one, Play! can handle those values for you, if you ask nicely.

How to make Play! handle Forwarded headers?

At the time of this writing, Play! framework support for Forwarded headers have known many states:

• In Play! 1.x, you need to add XForwardedSupport=all in your application.conf
• In Play! 2.0 to 2.3, you need to add trustxforwarded=true in your application.conf

Both these ways only support the X-Forwarded-Proto header.

Now, in Play! 2.4.x, the philosophy is different:

• Define the version of the Forwarded header you want to use: play.http.forwarded.version=x-forwarded|rfc7239
• Set the proxies you trust: play.http.forwarded.trustedProxies=["proxy_ip1","proxy_ip2",…].

Of course proxy_ipX can be an actual IP or a subnet mask, like "0.0.0.0" or "::" to trust every IPv4 or v6, respectively. Defaults are "127.0.0.1" and "::FF".

Also, as the X-Forwarded-Proto header is the one that is widely used in the world, the version default value is "x-forwarded".

What the hell is Strict-Transport-Security?

As you read the filter code, you must have seen that in the case the request is already in HTTPS, we still add a header to the response: Strict-Transport-Security: max-age=31536000.

This is the HTTP Strict Transport Security (HSTS) header. What it does is basically telling the client (most likely a browser): "Next time (and for the next 31536000 seconds), if your user tries to load the unencrypted version of the site, don't wait for me to redirect you and use https already".

The browser (meaning: chrome >= 4.0.211.0, firefox >= 4.0, Opera >= 12, IE >= 11) will then save the website and automatically replace "http" by "https" in the requests the next times.

This mechanism is documented here: https://www.rfc-editor.org/rfc/rfc6797.txt.

You MUST set the max-age value. You can also add includeSubDomains (after a ";" of course), which means "if you get that header while requesting domain.com, please use HTTPS when requesting *.domain.com too". It is a good practice to always add includeSubDomains just in case.

Please note that the STS header can only be set if the website is already TLS protected. You MUST NOT set this header on a non-TLS response.

If you want the browsers to use HSTS before the first request, you can register your domain to be included in browsers preload lists. To achieve that, register your domain here: https://hstspreload.appspot.com/. Also add the preload value to the header, like that: Strict-Transport-Security: max-age=31536000; preload.

At Clever Cloud, we want developers to be happy. To achieve that we try to adapt as quickly as we can to users needs. The other day, someone asked for Sidekiq support for their Rails applications.

What's Sidekiq

Sidekiq is a solution to run long background tasks in ruby. It works by running a background server and treating jobs stored in a Redis. It's helpful to delay tasks, take a long API call to the background while still be able to answer to a request in a timely manner.

How do I use Sidekiq on Clever Cloud?

First, you need a Redis instance. Start by provisioning a redis addon on Clever Cloud. Remember to link the redis add-on to your application in the Console.

The redis addon provides a REDIS_URL environment variable which will be used by sidekiq to connect to the redis addon.

• The next and only step is to add/modify the clevercloud/ruby.json file to add the "sidekiq" flag:

{
   "deploy": {
      "sidekiq": true
   }
}

Then commit and push your application, and you should be all set!

Change is happening in the Node landscape.
A group of rebels forked the main Node project, tired to wait for new versions and ES6 features. Their goal is to provide a npm compatible framework that always runs over the latest V8 version. The underlying goal is to profit from ES6 features that are being implemented in V8.

Today, Clever Cloud gives you the possiblity to run your io.js app in the cloud.

PRESS START

A soundtrack is available for this post, in order to cope with its visual artwork:

Here comes the rain again

It is inevitable. Agent Smith, Matrix Reloaded (2003)

Open Source generates diversity. Node.js fork io.js hits its version 1.0 (1.0.4 now). After a slight clash with Joyent last year, one of the Node.js developers, Fedor Indutny, decided to go on his own to maintain a new version of Node.js.
As mentionned above, the goal it to support the features of version 6 of the ECMA-262 specification [PDF], which implementation timeframe was judged too slow on Node.js.

Why is supporting io.js great?

While it is great to have a production-ready environment, addressing stability and security concerns, we are still nerds. We love to use cutting edge technology. The io.js framewok just released version 1.0.4. This is not a stable version. It is just a version different enough from the node.js project that io people just felt they needed to express it in the version number.

Is io going to replace Node.js on Clever Cloud?

No. We will go on supporting Node.js and io together. For now, the default is the latest Node.js version. In the future, the standard for server-side javascript might be io, or Node.js, or another framework that does not exist yet. It is still a young ecosystem.
Even if the entire community switches to another Node.js fork, we will still support Node.js for compatibility reasons.

How do I use io.js on Clever Cloud?

It's fairly simple: as you specified the Node.js version in package.json, you just prefix the version with io.js:

{
  "engines": {
    "node": "iojs-v1.0.3"
  }
}

And that's it! The "node" executable will be the one provided by io.js.

What can I do with io.js?

First, you can do everything you could do with Node.js. In addition to that, io.js is mostly about new features. Check the "ES6 support" page: https://iojs.org/es6.html.
You can use Map instead of Object, you have out-of-the box Promises/A+, you can (and will) use the "let" keyword…

Try it now on Clever Cloud!

Credits & Thanks to nervewax for its great background image used for this post.

Requirements

To follow this post, we assume that you already know and use SBT. Nothing else is needed.

What is / Why use Artifactory?

Artifactory Open Source is an open source proxy and cache server for build automation and dependencies manager tools in the JVM world. It is what I call a "lazy mirror". It acts like a Maven (or Ivy) server, and caches the artifacts "for ever". It is not a proxy as described in RFC 2616. Eventually it becomes a mirror of the repositories it serves.

As the Clever Cloud scalers are stateless, they don't keep the dependencies of a project between two deployments. Therefore, every dependency has to be downloaded for each deployment. Maven having the (almost justified) reputation to download half the internet on a first run, a deployment can take ages because of the dependencies.

The first step we did was to initialize a local Maven (or Ivy, or SBT) cache with common plugins and dependencies. But maintaining an up-to-date image represents a lot of work. So, to speed up the download of dependencies, we needed a Maven (resp. Ivy) repository next to the scalers; that means in the same data center.

After considering various possibilities, the Artifactory solution was the best one:

• (Ridiculously) easy to set-up;
• The open-source (and free) version matches our needs without extra
  complexity;
• Supports high concurrency;
• No need to watch for desynchronisation;
• Can proxy any given repository (Maven, Ivy);
• Does not act like a HTTP proxy but like a maven repository (that's the
  important point);

So, eventually, we started the server

The bad part: client configuration

Setting up the server was as easy as dropping a war in an application server. Actually, it just consists of dropping a war in an application server, and following the (crystal clear, well written) documentation.

Setting up the client wasn't that easy.

Maven client configuration

Maven was not a huge problem to set up. Some clicks in the artifactory public (anonymous) interface give you the following (you can happily remove the servers section):

<?xml version="1.0" encoding="UTF-8"?>
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0
http://maven.apache.org/xsd/settings-1.1.0.xsd"
  xmlns="http://maven.apache.org/SETTINGS/1.1.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <profiles>
   <profile>
    <repositories>
      <repository>
       <snapshots>
        <enabled>false</enabled>
       </snapshots>
       <id>central</id>
       <name>libs-release</name>
       <url>http://maven.mirror.clvrcld.net:8080/artifactory/libs-release</url>
      </repository>
      <repository>
       <snapshots />
       <id>snapshots</id>
       <name>libs-snapshot</name>
       <url>http://maven.mirror.clvrcld.net:8080/artifactory/libs-snapshot</url>
      </repository>
    </repositories>
    <pluginRepositories>
      <pluginRepository>
       <snapshots>
        <enabled>false</enabled>
       </snapshots>
       <id>central</id>
       <name>plugins-release</name>
       <url>http://maven.mirror.clvrcld.net:8080/artifactory/plugins-release</url>
      </pluginRepository>
      <pluginRepository>
       <snapshots />
       <id>snapshots</id>
       <name>plugins-snapshot</name>
       <url>http://maven.mirror.clvrcld.net:8080/artifactory/plugins-snapshot</url>
      </pluginRepository>
    </pluginRepositories>
    <id>artifactory</id>
   </profile>
  </profiles>
  <activeProfiles>
   <activeProfile>artifactory</activeProfile>
  </activeProfiles>
</settings>

And that's it. We just add repositories that will be tried before the user defined repositories. If an artifact is on a private repository, it will not be downloaded through nor cached by our artifactory instance.

SBT client configuration

Add ivy typesafe and SBT repositories to Artifactory

For the SBT instance, there is two things to do: add the ivy typesafe repositories in the Artifactory and configure SBT to use our Artifactory instance.

First, in Artifactory, we add the following remote repositories:

• sbt-plugin-releases => http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/
• typesafe-ivy-releases => http://repo.typesafe.com/typesafe/ivy-releases/
• typesafe-maven-releases => http://repo.typesafe.com/typesafe/maven-releases/

Then we put the first two in a new virtual repository (for example ivy-remote-repo), then we add the third one to the remote-repos virtual repository.

Configure SBT

Here we come to the core of this post: setting up SBT to use our proxy server while keeping the users out of the trouble of setting a specific Clever Cloud configuring for their applications.

Rummaging through the documentation, we come across a "Proxy" section.

Let's follow it and define a ~/.sbt/repositories file:

[repositories]
  local
  maven-local
  clever-ivy-proxy-releases: http://maven.mirror.clvrcld.net:8080/artifactory/ivy-remote-repos, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  clever-maven-proxy-releases: http://maven.mirror.clvrcld.net:8080/artifactory/libs-release
  clever-maven-proxy-snapshots: http://maven.mirror.clvrcld.net:8080/artifactory/libs-snapshot

But it's a trap! If we had used the -Dsbt.override.build.repos option we would have overriden all the project-defined repositories, even the private ones, which would have led to build failures because of unretrievable dependencies.

So, at this point, no easy configuration was possible. It was either not use the ~/.sbt/repositories file, either override user-defined repositories.

So let's be clever, and read about global settings. We can put a global.sbt file in ~/.sbt/ (and ~/.sbt/0.13/ because of the new 0.13+ global files versioning) that will be used each time we start SBT.

We keep our repositories file, and in the global.sbt file we put the following:

externalResolvers <<= (bootResolvers, externalResolvers) map (
    (boot: Option[Seq[sbt.Resolver]], ext: Seq[sbt.Resolver]) =>
        (boot.getOrElse(Seq.empty[sbt.Resolver]) ++ ext).distinct
)

Here, the bootResolvers value represents the content of the repositories file, and the externalResolvers value reflects the default SBT repositories plus project-defined repositories.

For curiosity sake, I added some printlns in global.sbt:

externalResolvers <<= (bootResolvers, externalResolvers) map (
    (boot: Option[Seq[sbt.Resolver]], ext: Seq[sbt.Resolver]) => {
        boot.getOrElse(Seq.empty[sbt.Resolver]).foreach(b => println("Boot::: " + b.toString))
        ext.foreach(e => println("External::: " + e.toString))
        (boot.getOrElse(Seq.empty[sbt.Resolver]) ++ ext).distinct
    }
)

Then I ran sbt compile in a play project with additional resolvers:

...
[info] Set current project to theproject (in build file:/theproject)
Boot::: FileRepository(local,FileConfiguration(true,None),Patterns(ivyPatterns=List(${ivy.home}/local/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]), artifactPatterns=List(${ivy.home}/local/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]), isMavenCompatible=false))
Boot::: Maven2 Local: file:/home/judu/.m2/repository/
Boot::: URLRepository(clever-ivy-proxy-releases,Patterns(ivyPatterns=List(http://maven.mirror.clvrcld.net:8080/artifactory/ivy-remote-repos/[organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]), artifactPatterns=List(http://maven.mirror.clvrcld.net:8080/artifactory/ivy-remote-repos/[organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]), isMavenCompatible=false))
Boot::: clever-maven-proxy-releases: http://maven.mirror.clvrcld.net:8080/artifactory/libs-release
Boot::: clever-maven-proxy-snapshots: http://maven.mirror.clvrcld.net:8080/artifactory/libs-snapshot
External::: FileRepository(local,FileConfiguration(true,None),Patterns(ivyPatterns=List(${ivy.home}/local/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]), artifactPatterns=List(${ivy.home}/local/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]), isMavenCompatible=false))
External::: Typesafe Releases Repository: http://repo.typesafe.com/typesafe/releases/
External::: Typesafe Snapshots Repository: http://repo.typesafe.com/typesafe/snapshots/
External::: Couchbase Maven Repository: http://files.couchbase.com/maven2
External::: Local Maven: file:/home/judu/.m2/repository
External::: public: http://repo1.maven.org/maven2/
...

We can see that bootResolvers contains the resolvers defined in the ~/.sbt/repositories file, and externalResolvers contains the default + project defined repositories. As long as SBT will use the repositories in the given order, our proxy repository will be used before the external ones. (Because of the boot ++ ext order.)

Sources

• http://blog.dlecan.com/configurer-scala-sbt-repository-artifactory/ (in french) for the artifactory ivy configuration,