What is Hexatrust?

Hexatrust is an association of French and European leaders in cybersecurity and the trusted cloud. Its mission is to promote excellence in the digital domain by bringing together start-ups, SMEs and ETIs innovating in the cybersecurity and cloud security sectors. The association strives to defend the interests of its members in dealings with public authorities, and to promote the French and European cybersecurity ecosystem.

Hexatrust's role

Hexatrust's mission is multi-faceted:

1. Advocacy: Representing industry players to the authorities.
2. Visibility: Increase members' recognition on the national and international markets.
3. Innovation: Leading a community of entrepreneurs to promote the exchange of ideas and the development of innovative solutions.

Why is joining Hexatrust important for Clever Cloud?

Collaboration and synergy

Unity is strength. Being a member of Hexatrust enables Clever Cloud to collaborate with other French and European industry leaders. This cooperation is crucial to developing even more robust cloud security solutions that meet today's cybersecurity challenges. We also share a common vision of the European digital age, and of the importance of digital sovereignty in protecting the data of our businesses and citizens.

Technology watch and peer-to-peer exchanges

Hexatrust has set up a number of working groups and organizes events enabling its members to exchange and share feedback on tools and issues. These actions are vital for Clever Cloud and its teams, to ensure the necessary monitoring of our products' security.

In line with our strategy

Clever Cloud has designed its platform to be secured by design: immutable architecture, avoidance of trusted networks (each peer on the same network is identified, authenticated and communicates in encrypted form), development or participation in the development of open source tools (Reverse Proxy Sozu and Token Biscuit). In addition to these best practices, we are ISO 9001 and ISO 27001:2022 certified. We are also in the process of obtaining HDS (Healthcare Data Hosting) certification and SecNumCloud certification.

Conclusion

Joining Hexatrust represents a springboard for Clever Cloud on its journey towards excellence in cloud data security. By joining this community of experts, we are reasserting our commitment while contributing to the evolution of the European digital landscape.

When you access a website or an online application, you most often do so in a "secure" way. This is for example the well-known green padlock that symbolizes HTTPS connections in your browser, which has become a standard these years thanks to initiatives like Let's Encrypt. 

This means that the data transferred to the server is encrypted, and that even if they are intercepted, they cannot be read by a third party. This protection has been provided by the TLS (Transport Layer Security) protocol for almost 20 years, whether it’s a personal site, an online shop or an access to your bank's services.

Over time, this critical technical brick on the Internet has evolved to strengthen the level of security it offers. In August 2018, its version 1.3 (the latest) was released. Meanwhile, versions 1.0 and 1.1 were considered to no longer offer a sufficient level of protection. They have been deprecated by the IETF (Internet Engineering Task Force) since March 2021 and have therefore been gradually removed from recent browsers such as Firefox, Chrome and its derivatives or Safari.

At Clever Cloud, we have seen our customers adopt TLS 1.2 and 1.3 gradually. On our load balancers, based on our in-house and open source reverse proxy Sōzu, the latest version accounts for over 90% of the requests processed each day. TLS 1.2 for just under 9%. TLS 1.0 and 1.1 for only a few tens of thousands of requests per day, less than 0.1% of our traffic.

While we have maintained these versions for compatibility reasons, this will no longer be the case as of June 30. We will of course inform the customers affected by this choice, and encourage them to switch to more recent versions, which will have advantages for them in terms of security, performance and SEO.

Several reminders will be sent between now and the final shutdown of TLS 1.0 and 1.1. If you have any questions on this subject, please contact our support team through the Console.

Few days ago, Marak Squires, the developer behind the open-source npm libraries colors and faker, decided to corrupt the libraries, to denounce issues in open-source projects' funding system.

The infinite loop introduced by the developer broke several apps using these libraries by printing the text 'LIBERTY LIBERTY LIBERTY' and non-ASCII characters in the apps' logs.

It causes a lot of trouble as the colors library receives over 20 million weekly downloads on npm alone and has almost 19,000 projects relying on it. Whereas, faker receives over 2.8 million weekly downloads on npm, and has over 2,500 dependents.

How to check if your Node.js app is impacted?

The first thing to do is to check if your app is using the npm libraries 'colors' or 'faker'. To do so, run either:

npm ls colors

Or

npm ls faker

You will get an output like this:

my-project@1.2.3 /home/me/my-project
├─┬ @storybook/addon-docs@5.3.18
│ └─┬ vue-docgen-loader@1.5.0
│   └─┬ jscodeshift@0.7.0
│     └── colors@1.4.0  deduped
├─┬ @storybook/vue@5.3.18
│ └─┬ @storybook/core@5.3.18
│   └─┬ cli-table3@0.5.1
│     └── colors@1.4.0  deduped
└── colors@1.4.0

With this output, we can identify that this project uses 'colors' directly with version 1.4.0 and through transitive dependencies, also in version 1.4.0.

Your app uses 'colors' or 'faker', what can you do?

If your app uses one of these npm libraries, we invite you to check three thing:

Check the version

First of all, you need to check if you're using one of the compromised versions of these libraries:

• colors: 1.4.1, 1.4.2, and 1.4.44-liberty-2
• faker: 6.6.6

Check the package-lock.json

Do you have a package-lock.json? If you don't we invite you to read the documentation and add one to your project.

If you do, you need to force a version which is not compromised (1.4.0 for colors and 5.5.3 for 'faker'). You're using npm? You can try with the module npm-force-resolutions. You're using Yarn? You can use the process described in this documentation.

Update your tools to their latest version

We also invite you to check if the dependencies you use released an update. As an exemple, if you use Storybook, the v6.4.10 released earlier yesterday fixes the issue.

A note for Clever Tools users

By the way, if you use our CLI, the clever-tools, and if you installed it via npm, please upgrade to v2.8.1.

What is Log4Shell?

You probably heard about Log4Shell (or CVE-2021-44228), the vulnerability which impacted log4j, a famous log library written in Java.

This critical vulnerability allows to remotely execute code on the servers of a company or to display the environment variables of an application.

What has been implemented at Clever Cloud?

At Clever Cloud, we worked all weekend to resolve this issue.

All our Elasticsearch add-ons were secured quickly, and many of our customers are secured by the most recent versions of JDK. Edit (13/12 16:41 UTC+1) : Even the most recent versions of Java are now vulnerable to RCE (Remote Code Execution) due to a bypass. The only viable solution is to patch and update log4j directly.

Please also note :

• Java 8 (or later) users should upgrade to release 2.17.0.
• Users requiring Java 7 should upgrade to release 2.12.2.
• Otherwise, remove the JndiLookup class from the classpath in a post build hook (you have to execute the hook in the file where the log4j jar is):

CC_POST_BUILD_HOOK=zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class "

For the others, we have initiated a thorough monitoring and analysis policy.

We are also in the process of updating the Java image with the following Log4j configuration property: Edit (14/12 14:04 UTC+1) : The Java image has successfully be updated and all Java applications have been redeployed with the following Log4j configuration property:

log4j2.formatMsgNoLookups=true

Please note that this flag only work on versions superior or equal to Log4j v2.10.0.

We upgraded the New Relic Java Agent to the 7.4.1 version and the apps on which the agent was deployed have been redeployed.

We also patched the Pulsar cluster.

How to mitigate the risks?

We urge you to update your dependency to Log4j v2.17.0.

Then, depending on the environments and add-ons you work with, here's what you can do as well:

For Docker

If you are using Docker, you can do either :

• Update to Log4j v2.17.0 (recommended)
• Or setup the following Log4j (v2.10.0 minimum only) configuration property: log4j2.formatMsgNoLookups=true

For Jenkins

The Jenkins security team has confirmed that Log4j is not used in Jenkins core. However, it can be used in some Jenkins plugins. You can identify if Log4j is included in a plugin by using the following command in the Script Console:

org.apache.logging.log4j.core.lookup.JndiLookup.class.protectionDomain.codeSource

Support team

Of course, our support team remains available if you have any question regarding the current situation. You can reach them via the chat or send an e-mail at support@clever-cloud.com.

Yesterday two issues affecting CPUs have been released to the public.

TL;DR: the attacks are named Meltdown and Spectre. They allow reading the memory of the OS or of other processes, to steal secrets or get information for other exploits. A part of the solution can greatly affect performance of running code. In particular, this attack allows to easily cross container boundaries, and in some cases (not our case) even VM boundaries.

In addition to servers, consumer machines are affected, especially through browsers, so you should definitely update your operating system as well as your browsers.

What it means for Clever Cloud users

Your applications will be (or already have been) automatically restarted (just like any other maintenance deployments). The addons will be patched and restarted in place in the following hours. This will generate limited downtime on addons (usually around a minute, depending on the addon start up time).

In addition to restarting virtual machines, we will also need to restart physical machines, as the attacks theoretically allows VM boundaries crossing. This attack is not usable (yet?) on Clever Cloud due to our virtualization choices and our OS hardening, but we will deploy patches preemptively. Physical machines updates will take place in the following days and will not impact applications. We are currently working on finding the best solution for addons, but it will definitely incur additional downtime for addons.

The patches, while mitigating the issues, also come with performance regressions. It heavily depends on the workload as well as the exact CPU model. The CPUs we use are among the less affected by the performance issues, but a slowdown of at least 5% is to be expected.

Technical details

The Meltdown attack and the Spectre categories of attack are related to a performance feature of modern processors: branch prediction and speculative execution. Meltdown shows that when an instruction can cause a trap, like the privilege check for user → kernel access), the processor will perform speculative execution: it starts executing the code in case there’s no trap, but rollbacks if there was a trap. This attack happens at the boundary between user code and kernel. Before the processor has completely checked that we have the authorization to run privileged code, it starts executing it. When it turns out we were not authorized, it rolls back the results of that code, but not completely, it can leave some data in the cache. Combined with a technique called “cache timing attack”, it is then possible to guess the content of the data that was loaded in cache, bit by bit. Branch prediction has a related behaviour: when encountering a branch (example: an if/else expression), the processor will start executing one of the branches before it calculates the condition, to avoid waiting too much. It guesses which side of the condition is most likely thanks to its branch predictor. Spectre uses branch prediction to cause speculative execution to read out of a buffer’s bounds (among other consequences) in the kernel or another process, then guess the results from the cache.

The Meltdown attack is specific to Intel processors, it allows reading from the OS’s memory. There are patches available (the kPTI feature, also named KAISER https://lkml.org/lkml/2017/12/4/709). Those patches have a great impact on syscall performance (https://www.phoronix.com/scan.php?page=article&item=linux-415-x86pti&num=1), with programs running 5% to 30% slower depending on the workload. The Intel Haswell processors with the PCID (Process Context Identifiers) feature get the lowest performance hit (5%). We use those processors on Clever Cloud.

Spectre affects processors from Intel, AMD and ARM, it allows reading from the memory of other processes. It looks more like a new attack category, for which we will have to fix the issue individually in each affected software. The only global solution for Spectre is a radical change in processor architecture, and this is unlikely to happen soon. We will follow closely any new related vulnerability and promptly patch our infrastructure.

For further information

• Papers and explanations about Meltdown and Spectre: https://spectreattack.com/
• Proofs of concept from Google’s Project Zero team: https://googleprojectzero.blogspot.fr/2018/01/reading-privileged-memory-with-side.html
• French twitter thread explaining the attacks: https://twitter.com/fenarinarsa/status/948697105996156928
• English twitter thread explaining the attacks: https://twitter.com/nicoleperlroth/status/948684376249962496

This post has been written by @gcouprie and @clementd.
Spectre and meltdown logos of are designed by Natascha Eibl.

As you know, security is a big deal for us. You might have heard that a single password may not be enough those days. Well, here it is! Two factor authentication is now available for everyone.

What is two factor? And how does it work?

When you login with a simple password, you access you account with something you know.

If someone else ever gets your password, your account is compromised. To prevent this situation, the login process should invole something you know, and something you have. This is the second factor.

2FA is pretty simple, once activated you login in with:

• your credentials
• and a third-party app on your phone providing a temporary code

Now having your password known/guessed by a third-party is not enough for them to get into your account.

How to setup?

Login to the Clever Console and head over to your profile. Under "Authentication", you'll see a new button: Activate 2FA.

A QR-Code will show up. Scan it with a 2FA app on your phone (Google Authenticator, Authenticator by Microsoft…)

For future logins, your password and the code from the app will be necessary.

IMPORTANT:

Don't forget to save your recovery codes! Each one can replace a generated 6-digits code (but only once).

• unclear risk and impact on the business
• time spent tracking new vulnerabilities for various applications
• unclear result of updating code (will it stop working? Will it break other applications on the same machine?)

Defining your risk budget

• targets: user data, intellectual property, machines
• entry points: web server, internal WiFi
• weaknesses: unpatched application, SQL injection, key employees victims of phishing

Staying up to date with security news

• avoid news websites. They write long articles, they want you to panic and they rarely provide usable solutions
• Follow security mailing lists. There are generalist ones, like oss-security@lists.openwall.com and cve-assign@mitre.org. There are more specific ones, like debian-security@lists.debian.org (translate to your specific distribution), or rubyonrails-security@googlegroups.com and ruby-security-ann@googlegroups.com. There is also fulldisclosure@seclists.org, where 0-day vulnerabilities are sometimes published
• Twitter is still a good source of information on vulnerabilities, since people easily share. If you see security people suddenly buzzing in your timeline, you should pay attention. There are good lists of people to follow to get you started here and there. They each have their own focus, though, so you may not be interested in everything
• keep up with new versions of your software and their dependencies. Use your package manager, project specific mailing lists, subscribe to their github feed

• check the mailing lists, see if you use any of the applications mentioned
• check your dependencies: anything new? Any security issues mentioned?
• check Twitter: is the world on fire?

Reducing the risk of code updates

• staging environments to test updates
• replacing huge, risky updates with small increments
• applications can be completely independent. Updating the company's WordPress blog will not affect the SaaS application

• get notified of a vulnerability
• see if it applies
• see if there's a patch (or if you can develop one quickly)
• apply the patch
• redeploy the applications
• go make yourself a nice tea

• The recent CVE-2016-0728 is a privilege escalation in Linux, something we need to take seriously. We took a look at the advisory, wrote a patch, tested it and deployed it in a few hours. Most Linux distributions took days to publish updated packages.
• In the same way, the infamous Heartbleed bug was fixed quickly. One of our clients came to us hours later asking if we knew about it: "oh, that's the reason my applications were redeployed in the middle of the night"

To protect your web app or API, there is almost only one way at this time: TLS. But users and browsers don't always use TLS by default. So what you want is to redirect them to a TLS encrypted version of your site if they try to connect via plain http.

Here is how to do it with Play Framework 2.4 in scala:

Play! and HTTP filters

We will start by creating a "TLSFilter.scala" file and write a TLSFilter class in it:

class TLSFilter extends Filter {
  def apply(nextFilter: RequestHeader => Future[Result])
    (requestHeader: RequestHeader): Future[Result] = {
      if(!requestHeader.secure)
        Future.successful(Results.MovedPermanently("https://" + requestHeader.host + requestHeader.uri))
      else
        nextFilter(requestHeader).map(_.withHeaders("Strict-Transport-Security" -> "max-age=31536000; includeSubDomains"))
  }
}

This part is easy: we implement the apply function by just checking the secure value of RequestHeader. If the connection is not secure, we need to redirect the client to the same url only with "https" as the protocol. If the connection is secure, we pass the request to the next header. Nothing simpler.

Note that we use requestHeader.host instead of requestHeader.domain because the host value is actually the value of the Host header as set by the client, with optional port and stuff.

Note that we create a Filter implementation and not an EssentialFilter one because we do not care about the body.

Next, you need to create a HttpFilters implementation that will hold the instance of your TLSFilter:

// In TLSFilter.scala
class MyFilters extends HttpFilters {
  val filters = Seq(new TLSFilter)
}

And finally, you need to tell Play! to use your Filters class:

# In conf/application.conf
play.http.filters=my.package.MyFilters

Now it will check your requests and permamently redirect the clients to HTTPS.

But, how does Play! know that the request is secure?

Reverse proxies: where did the 's' go?

Now, we need to ensure that Play! knows to differentiate a secured connection from a plain one. If you configured HTTPS in your application, it's quite simple to understand. But it is not always the case:

You most probably did not, configure TLS in your application. And that is because you deployed it on a very powerful and developer-friendly PaaS. So, chances are your Play! application is getting requests in plain HTTP, because the TLS encryption ended at the front reverse-proxy that's piping the request towards your app.

How then is your application going to know that the connection is secured? Enter the non-standard and the standard ways.

X-Forwarded-Proto

The first, non-standard but widely used (e.g. at Clever Cloud) way to know if the request handled by the reverse proxy in front came in a secure channel is to check the X-Forwarded-Proto HTTP header. Like all non-standard headers, you can recognize it by the X- at the beginning of the name.

This header describes how the final client is communicating with the reverse-proxy.

It takes two values: http and https. You can check for that header in your application. But we will see below that Play! can do it by itself.

RFC 2739

Also called the Forwarded HTTP Extension, it standardize the way that a proxy tells the final endpoint what is going on between the final client and itself. It's been published in June 2014.

You can read it here: https://tools.ietf.org/html/rfc7239. But the only thing that is relevant for us is the proto parameter. Like X-Forwarded-Proto earlier, its values that interest us are http and https. Like for the other one, Play! can handle those values for you, if you ask nicely.

How to make Play! handle Forwarded headers?

At the time of this writing, Play! framework support for Forwarded headers have known many states:

• In Play! 1.x, you need to add XForwardedSupport=all in your application.conf
• In Play! 2.0 to 2.3, you need to add trustxforwarded=true in your application.conf

Both these ways only support the X-Forwarded-Proto header.

Now, in Play! 2.4.x, the philosophy is different:

• Define the version of the Forwarded header you want to use: play.http.forwarded.version=x-forwarded|rfc7239
• Set the proxies you trust: play.http.forwarded.trustedProxies=["proxy_ip1","proxy_ip2",…].

Of course proxy_ipX can be an actual IP or a subnet mask, like "0.0.0.0" or "::" to trust every IPv4 or v6, respectively. Defaults are "127.0.0.1" and "::FF".

Also, as the X-Forwarded-Proto header is the one that is widely used in the world, the version default value is "x-forwarded".

What the hell is Strict-Transport-Security?

As you read the filter code, you must have seen that in the case the request is already in HTTPS, we still add a header to the response: Strict-Transport-Security: max-age=31536000.

This is the HTTP Strict Transport Security (HSTS) header. What it does is basically telling the client (most likely a browser): "Next time (and for the next 31536000 seconds), if your user tries to load the unencrypted version of the site, don't wait for me to redirect you and use https already".

The browser (meaning: chrome >= 4.0.211.0, firefox >= 4.0, Opera >= 12, IE >= 11) will then save the website and automatically replace "http" by "https" in the requests the next times.

This mechanism is documented here: https://www.rfc-editor.org/rfc/rfc6797.txt.

You MUST set the max-age value. You can also add includeSubDomains (after a ";" of course), which means "if you get that header while requesting domain.com, please use HTTPS when requesting *.domain.com too". It is a good practice to always add includeSubDomains just in case.

Please note that the STS header can only be set if the website is already TLS protected. You MUST NOT set this header on a non-TLS response.

If you want the browsers to use HSTS before the first request, you can register your domain to be included in browsers preload lists. To achieve that, register your domain here: https://hstspreload.appspot.com/. Also add the preload value to the header, like that: Strict-Transport-Security: max-age=31536000; preload.

nom is a parser combinators library witten in Rust that I started about a year ago. Its goal is to let you write parsers that are safe by default, fast, and abstract all of the dangerous or annoying details of data consumption.

During that year, more than 50 projects have started using it; from toy parsers to high performance production code. Their feedback has been invaluable to improve the library, include more and more parsing patterns, and test ideas on what makes a great parser library. The 1.0 version is the result of that feedback. More stable version, but also a few breaking changes to improve the architecture, make it more flexible and easier to use. We now feel it is reliable enough to be used in production at Clever Cloud. We have a lot of data to manage, coming from trusted and untrusted sources, and this is exactly the kind of tool we need to build a safe infrastructure.

The quantity of open source projects using nom has been really helpful in developing that stable release. If you maintain one of those projects, you may have received a pull request from me. That's right: I took care of testing the 1.0 branch on every project I could get my hands on, to see what would break, which features developers were using, and document the upgrade process. This has been a lot of work, but worth it. I'll probably tell more about that in a future blog post, for other library maintainers that want to try the approach.

That's all good, but why would you use nom right now? Let's see!

nom is fast. How fast? A few benchmarks have shown that it is consistently faster than Parsec and attoparsec (Haskell parser combinator libraries), faster than other Rust parser combinator libraries, and even faster than Rust's regular expression library. There is even a benchmark where it beats Joyent's http-parser on parsing HTTP request headers.

Why is it faster? I have a few ideas about this. First, unlike most parser combinators systems, nom does not copy data if it is not needed. It uses the slice heavily, a Rust data structure containing a pointer and a length. Since Rust's compiler manages memory correctly, you can afford to refer to the original input from the beginning to the end of the parser, without copying anything.

Second, nom does not chain parsers at runtime. The macros directly generate the parsing code at compile time. This creates very linear code, something that CPUs find very easy to handle. If you tried to decompile the final binary to C code, you would just see a long list of if-else branches.

It is also a safe alternative to handwritten C parsers. nom bases its memory safety on Rust's compiler: it knows, at any moment, which part of the code owns which part of the memory, prevents out of bound accesses, automatically manages memory allocation and deallocation. And since that is not enough, some nom parsers were fuzzed to hell with American Fuzzy Lop, just to verify those claims.

The result? The only flaws that were found appeared, not in nom generated code, but in code written manually outside of nom: index calculations that could overflow if a specific value appeared in the input. And those could not result in memory corruption, just crashes.

you can quickly write a parser that will be safe by default

This has awesome implications: you can quickly write a parser that will be safe by default. This lets you test ideas, experiment with your design, without fear for your security.

You should now see where I'm going: with parsers that are easy to write, as fast or faster than handwritten C, and safe by default; you can replace old and vulnerable C parsers. Rust can work without a runtime, and is easily embedded in C code. It has already been used to write extensions for Ruby, Python, NodeJS and others. It is only a matter of time until it replaces the vulnerable parts of current C projects.

This is one of my long term goals: making reliable, safe building blocks to build our systems. Not only new bricks, but also replacing the old ones. This will require a tremendous effort, and nom is just the first step, but a big one.

To get started using nom, you can include it in your Rust projects from crates.io. Here are a few links you will find useful:

• Github repository Geal/nom
• Reference documentation
• Upgrading to nom 1.0
• Gitter chat room. You can also go to the #nom IRC channel on irc.mozilla.org, or ping 'geal' on Mozilla, Freenode, Geeknode or oftc IRC
• Tutorial about parsing ISO8601 dates
• Making a new parser from scratch (general tips on writing a parser and code architecture)
• How to handle parser errors
• How nom's macro combinators work

Also, if you have existing code running older versions of nom, please take a look at the upgrade documentation

The attacker is only one wrong click on a lovingly crafted PDF file away from your network.

Assuming that servers will be safer if they are on your own network leads to a false sense of security.