Back to basics with this first article in a new series. Let's take a look at how cloud computing works, its different architectures (public, private, hybrid), its many benefits for businesses, and the differences between cloud computing, edge computing and on-premise to help you choose the solution best suited to your needs.

Cloud computing: definition and origins

Cloud computing has profoundly transformed the way businesses and individuals access IT resources. The idea is simple but innovative: rather than owning and managing their own servers and data centres, users can now access IT services on demand, via a network. These services include computing power, data storage and sometimes even database management, offering unprecedented flexibility and accessibility. It's not an idea that dates back to yesterday...

How cloud computing has evolved over time

The concept of cloud computing, although widely popularised in the early 2000s, has its roots in the 1960s. At that time, computer pioneers were already considering the idea of making computing power accessible as a public utility. During the 1990s, network diagrams began to use the image of a ‘cloud’ to represent the Internet, symbolising the complexity and interconnectedness of the networks that today enable cloud computing to function as a global and flexible architecture.

The real strength of the cloud lies in its ability to provide remote and scalable computing resources, enabling businesses to simplify the management of their infrastructure, avoid wasting resources, and improve energy efficiency through three pillars:

• Unified billing: The cloud offers a single billing system that simplifies cost management for businesses. This transparency makes it possible to accurately track expenditure and optimise costs;
• Centralised, secure access: The cloud offers a single interface that simplifies access to services, while centralising management. IAM (Identity and Access Management) systems provide enhanced security, with auditability tools for improved compliance;
• A rich and diverse marketplace: The cloud provides a vast marketplace, giving businesses access to a wide range of products and services. This diversity promotes the flexibility and scalability of the solutions deployed.

How do you simply define cloud computing?

In simple terms, cloud computing is defined as remote access to shared computing resources, without the complexity of managing the underlying physical infrastructure. This approach enables businesses to make a long-term investment in the long-term management and hosting of their data, while giving them the flexibility to rapidly adjust their capacity as required. Clever Cloud, for example, automates the scalability, optimising resources in real time to meet demand.

How does cloud computing work?

Based on a network-based computing services model, the cloud enables businesses to access resources such as data storage, computing power and access to a catalogue of online applications, without the need to manage the hardware infrastructure directly. This approach offers greater flexibility, cost optimisation and operational efficiency, all of which encourage innovation.

The key elements of cloud operation

There are several fundamental elements that ensure the efficiency and scalability of the cloud. Here's an overview of the key elements.

The physical infrastructure

At the heart of the cloud is a physical infrastructure made up of servers, storage systems and interconnected networks. These infrastructures are located in data centres around the world, promising very high availability, enhanced security and optimum performance. Cloud providers like Clever Cloud invest heavily to maintain high quality standards, verified through various certifications.

Virtualisation

This fundamental principle enables the creation of virtual versions of physical resources, such as servers or storage systems. Virtualisation improves efficiency by enabling several users or applications to use the same resources in an optimised way: the famous Virtual Machines (VMs). This technology also simplifies management and reduces operating costs by disconnecting the hardware infrastructure from the user experience.

Access interfaces and APIs

Cloud service interfaces are designed to be secure and high-performance, making it easier to manage resources and deploy applications. APIs offer system administrators, DevOps and developers the possibility of automating and integrating cloud services into their own systems, maximising customisation, efficiency and flexibility and extending the range of possible functionalities.

The different cloud architectures: public, private and hybrid

Each cloud architecture has its own advantages and addresses different needs. The key is to choose the one that best aligns with your organisation's strategic objectives and operational constraints.

Public Cloud: simplified accessibility and scalability for data hosting

The public cloud is the most common architecture. Here, IT resources are hosted on servers belonging to cloud service providers, such as Clever Cloud. These services are accessible via the network, providing a scalable solution that can adapt quickly to changing business needs.

The public cloud offers different levels of service, from Infrastructure as a Service (IaaS) to Platform as a Service (PaaS), via Software as a Service (SaaS). This diversity means that businesses can choose the solution best suited to their needs, while benefiting from the fact that there is no initial investment in infrastructure.

Private cloud: controlled, secure data hosting

By definition, the private cloud is a solution dedicated to a single organisation. Unlike the public cloud, the customer's servers are dedicated to that organisation. This is often a hosting zone behind the customer's firewall, made available by a cloud provider. This architecture offers total control over data and enhanced security, which is ideal for sectors subject to strict regulations (financial, health, pharmaceutical, public, defence, legal, industrial, etc.). The private cloud allows greater customisation of the data hosting architecture, tailored to the specific needs of the business.

Hybrid cloud: don't put all your eggs in one basket

The hybrid cloud offers greater resilience by combining the advantages of the public and private clouds. By diversifying environments, businesses avoid putting all their eggs in one basket. For example, they can store sensitive data in a private cloud while using a public cloud for backups or less critical workloads. This means they can switch from one environment to the other depending on their needs in terms of performance, cost or security. However, with the high egress fees charged by hyperscalers, this approach can become costly.

In reality, the hybrid cloud is already common practice, often without companies being fully aware of it. Since the democratisation of SaaS, the majority of applications are no longer solely on-premises, but are based on a mix of public cloud, private cloud and SaaS services.

What are the different types of cloud computing?

Cloud computing is divided into several main categories: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS) and Function as a Service (FaaS). Each type offers different levels of control, flexibility and management, so you can choose the one that best suits your needs.

Infrastructure as a Service (IaaS) for maximum customisation

Infrastructure as a Service (IaaS) is historically the most widespread model of cloud computing. With IaaS, businesses rent IT resources such as servers, virtual machines, storage and networks directly from a cloud service provider.  Customers take care of maintaining their OS, databases, runtimes, etc. in operational conditions, using system administrators, either in-house or from an outsourcer. This allows companies to have a specific architecture that meets their needs. 

Platform as a Service (PaaS) for rapid deployment and automatic updates

Platform as a Service (PaaS) goes a step further than IaaS by providing an OS, middleware and a console for easy deployment of runtimes and even databases or third-party services, known as add-ons. The customer no longer has to worry about maintaining the system in operational condition (MCO), which is the responsibility of the cloud provider.

PaaS solutions make it easy to deploy applications quickly, while automatically maintaining and updating the infrastructure. With integrated development tools and a modular architecture, Clever Cloud simplifies the work of developers and DevOps teams by offering them a ready-to-use environment to accelerate innovation: deploy code, we'll take care of the rest! 

Software as a Service (SaaS) for greater simplicity

Software as a Service (SaaS) is the most common cloud computing model. SaaS applications are hosted and managed by service providers, enabling users to access them via the Internet, usually on a subscription basis. This model eliminates the need to install and maintain software locally, reducing costs and technical complexity.

SaaS is particularly useful for businesses looking to standardise their software tools without investing in IT infrastructure. For example, collaboration tools, customer relationship management (CRM) systems and office suites are often deployed as SaaS services for maximum simplicity.

Functions as a Service (FaaS) for greater agility

Functions as a Service (FaaS) is an evolution of the PaaS model that allows developers to create and deploy individual functions without worrying about server management. The cloud provider takes care of all the infrastructure, scaling and resource management, charging only for the time it takes to run the functions.

The functions are triggered by events defined in advance, enabling the function to launch its action (processing an image, converting a file, etc.) and therefore to switch on, be active for the duration of the action and then switch off. services for economic reasons. FaaS is often complementary to PaaS.

What are the advantages of cloud computing?

Whether in terms of scalability or security, the cloud offers a number of benefits for organisations that decide to adopt it.

Cost optimisation: a strategic approach rather than just savings

Cloud computing can be part of a financial strategy, in particular by transforming capital expenditure (CAPEX) into operational expenditure (OPEX). This budgetary transition enables companies to outsource their initial IT infrastructure costs (purchase of servers, data centre maintenance, fibre management, recruitment of a dedicated team, etc.). Organisations can adopt a pay-as-you-go model, adjusting their spending according to their real needs. Some hyperscalers will still offer to pay for reserved instances, which will have to be paid for even if they are not used, but this is not the majority of usage. 

Scalability: the cloud adapts to your needs

Whether you need to increase processing capacity during peak periods or reduce resources during off-peak periods, the cloud is flexible and scalable on demand, thanks to probes and orchestrators. This responsiveness ensures optimum use of resources and gives businesses greater agility to respond to changes in the market and in demand. At Clever Cloud, scalability is automatic, which means it adapts in real time to your needs, so you only pay for what you actually use.

Security: a key pillar for businesses

Cloud computing greatly improves the security of your data. Cloud providers invest, keep a constant watch and have the role of warning their customers or updating exposed areas. Storage in the cloud and its backups provide great resilience. Backed-up data can easily be accessed or a disaster recovery plan (DRP) or business continuity plan (BCP) can be launched. At Clever Cloud, we go even further: our Materia databases are automatically replicated to our three Paris data centres, giving you peace of mind. Another key point is that many security certifications may be required when hosting critical or personal data. Cloud providers undertake to comply with them. 

Accessibility: instantaneous response to new work trends

Cloud computing makes your data more accessible, enabling your teams to collaborate effectively, regardless of their location. Data and applications hosted in the cloud can be available 24/7. This facilitates teleworking, speeds up decision-making processes and improves team productivity, while guaranteeing real-time synchronisation of information.

Automatic updates and operational maintenance

With cloud computing, software updates and security patches are automatically managed by the supplier, without the intervention of the customer's IT team. This not only reduces the internal workload, but also ensures that the systems have no security holes.

Who uses cloud computing?

Cloud computing is now an essential component for businesses of all sizes and in all sectors. Its massive adoption is driven by the outsourcing of management and MCO, the unification of costs and access to a marketplace.  Whether it's for data storage and backup, Big Data analysis, mail server hosting or web application development and deployment, the cloud has become essential for meeting the growing demands of the market.

For example, organisations in the financial sector are using cloud computing to detect and prevent fraud in real time, using machine learning algorithms and predictive analysis. Healthcare companies, meanwhile, are exploiting these technologies to offer more personalised treatments, thereby improving the quality of care for their patients. The entertainment sector and video game creators are relying on the cloud to deliver their content to millions of users around the world, guaranteeing a fluid and immersive experience.

But this technology is not just for large companies: SMEs, start-ups, freelancers and private individuals are also adopting the cloud to benefit from a scalable infrastructure without having to bear the high costs associated with physical data centres. By opting for these solutions, they can concentrate on what they need.

What are the uses of cloud computing?

Cloud computing offers a multitude of uses for businesses and individuals, including the following.

Storing and backing up data

The cloud enables businesses to protect their information while benefiting from a large storage capacity, all the while retaining control over billing. Data can be easily accessed from any connected device, making it easier to recover in the event of a technical problem.

Hosting applications and websites

Hosting applications and websites is another key use of cloud computing, enabling applications to be created and deployed with a high degree of scalability. Developers benefit from a reliable solution that delivers a fluid, high-performance user experience.

Distributing software and content as a service

Cloud computing also facilitates the development of software and content as a service (SaaS). This model reduces the need for local installations and enables companies to offer regular updates without service interruption, accessible at any time and on any type of device. They can focus on the continuous improvement of their products while simplifying the management of software versions.

Data analysis and artificial intelligence (AI)

Data analysis and the integration of artificial intelligence (AI) are made possible by the capabilities and performance of cloud computing. Companies can centralise their data in cloud systems, such as datalakes or data warehouses, and use and train machine learning services to extract the desired elements. These technologies help to develop predictive models, automate processes, generate text or images, etc.

Application testing and development

The cloud is also a powerful tool for application testing and development. Thanks to the standardisation of test environments, it is now possible to obtain dedicated spaces in just a few clicks, saving time for teams of developers. Cloud infrastructures offer a high degree of flexibility for creating, testing and deploying applications, while facilitating continuous integration and continuous deployment (CI/CD). This approach accelerates the development cycle and enables companies to reduce their time to market.

Edge computing and cloud computing: the differences

Cloud computing and edge computing are two distinct architectures that meet different needs. Whereas cloud computing centralises resources and processing in large data centres, edge computing moves these operations closer to the data sources using small-scale infrastructures.

This approach enables edge computing to reduce latency and improve processing speed, which is essential for applications such as the Internet of Things (IoT), autonomous vehicles and augmented reality. The choice between these two technologies often depends on the context and on performance, security and connectivity requirements. Edge computing also makes it possible to offer a hybrid architecture, which can take over in the event of availability problems.

Cloud computing is distinguished by its ability to offer scalable, pooled resources, accessible via the Internet, making it an ideal solution for applications requiring high computing and storage power. By contrast, edge computing favours local processing for real-time responses, reducing dependency on network connectivity.

When should you choose edge computing rather than cloud computing?

Choosing between edge computing and cloud computing depends on several key factors:

• Reactivity and low latency: Edge computing is the best solution when instant response is crucial. For example, in industrial control systems or autonomous vehicles, every millisecond counts. Local processing of data by edge computing reduces delays and improves the reliability of operations;
• Volume of data to be processed: When the volume of data generated is too large to be transferred quickly to a central data centre, edge computing becomes essential. It enables data to be filtered and processed at source, minimising network traffic and optimising bandwidth use;
• Isolated environments or those with limited connectivity: In areas where the Internet connection is unstable or non-existent, edge computing ensures that data is processed without interruption. This makes it an ideal solution for remote sites, such as industrial facilities or critical infrastructures;
• Data security and confidentiality: Edge computing provides an extra layer of security by processing data locally, limiting its exposure to potential cyber threats. This is particularly important for sensitive sectors such as healthcare or financial services.

So, while cloud computing is suited to tasks requiring vast resources and global accessibility, edge computing is essential in contexts where speed, locality and data security are paramount.

On-premise and cloud computing: the differences

The evolution of IT infrastructures has led to two main approaches: on-premise and cloud computing. These two solutions differ fundamentally in terms of deployment, management, cost and scalability. On-premises involves purchasing, installing and managing hardware and software directly on the company's servers by a trained team, offering total control over data, network and security. Cloud computing, on the other hand, is based on outsourcing these tasks so that the company can concentrate on its core business.

When should you choose on-premise rather than cloud computing?

Opting for an on-premise infrastructure can be a strategic choice, from a budgetary, technical and HR point of view. If the company wants to pay for and amortise its investments, control the entire value chain and be able to recruit and retain its talent, the on-premise option may be just what it needs. If it wants to concentrate on its core business, it will find it easier to turn to cloud computing to outsource this work to a service provider whose job it is. 

The future of cloud computing

Cloud computing continues to evolve at lightning speed, and its potential continues to grow. We are at the dawn of an era where cutting-edge technologies such as artificial intelligence (AI), quantum computing and edge computing will redefine current cloud paradigms. This future promises increased integration of low-code and no-code platforms, as well as massive adoption of the Internet of Things (IoT), enabling businesses and developers to design innovative solutions without necessarily benefiting from advanced technical expertise.

They “push” their code, and we deploy it, while ensuring that the environment in which they operate is kept up to date and secure, with one clear objective: maintaining it in operational condition. Whatever the situation, their service responds.

Designing simple, robust and innovative services

We do this whilst minimizing the footprint on our servers, offering on-demand or automatic horizontal/vertical scaling. This means we can multiply instances or scale them up / down according to the load of your sites. This commitment to simplicity extends beyond applications, as we also offer a wide range of “as-a-service” database (DB) services, enabling you to deploy MongoDB, MySQL, PostgreSQL or Redis environments in a matter of seconds, with the same tools and guarantees.

However, over the past few years, we faced a number of recurring requests: not only did some of our customers wish to access other services, but they also regularly asked to take benefit of our infrastructure, spread over three datacenters, to obtain high-availability services. But when it comes to DB, this isn't always easy to implement while ensuring a high level of automation and data consistency. So we decided to develop our own range of products to achieve this.

The “Materia” project was born, based on the FoundationDB open-source transactional engine used, for example, by Apple for its iCloud services and by Snowflake. We're contributing to it and adapting it to our needs, while taking advantage of its distributed and resilient aspect, notably due to its significant simulation capacity. It allows us to imagine a whole universe of serverless services, very simple to use, invoiced on a pay-per-use basis, with compatibility layers we're developing to enable our customers to “talk” to it using existing protocols such as DynamoDB, GraphQL or PostgreSQL, for example.

Materia KV available to all

We started with the Redis protocol to introduce Materia KV, currently in Alpha testing phase. It's a key-value database synchronously replicated across our three Paris datacenters. When our servers respond, your data has been written three times. Above all, it's very easy to use: go to our Console, create a Materia KV add-on, and you'll instantly obtain a token that can be used with any Redis client or by sending commands to our servers directly through the TCP protocol.

Thus, you can now benefit from a high-availability, key-value data storage solution, without having to configure a single instance, and with no minimum monthly fee: you simply pay for what you use. Above all, we provide this service with compatibility to existing protocols. Today Redis, tomorrow DynamoDB and GraphQL. The list of supported commands is available here.

Launched at Devoxx France mid-April, Materia KV made a big splash on site and during our demonstrations since then. We've therefore decided to make it available to all, to enable our customers and the interested public to give us feedback on how it works for them, and what their needs are. While billing will be on a per-use basis starting with the Beta phase, right now it's free! So give it a try and let us know what you think.

Already built-in our tools and interfaces

Materia KV is of course available as an addon-provider in our API, Terraform (or OpenTofu, for example), but also in our CLI, Clever Tools. To create an access token, simply run the following command:

clever addon create kv NAME

This add-on can be linked directly to a Clever Cloud application, injecting its environment variables. But you can also use it from any third-party service, command-line client (CLI) or with graphical user interface (GUI).

While Materia KV and its various interfaces will be developed over time, we've started to think about integrating it into Clever Tools. A test branch, available as a binary and documented, offers a clever kv command.

With it, you can directly send requests to Materia KV by having the environment variable configured locally, or by specifying an ID in your commands:

# With Bash and Zsh
source <(clever addon env ADDON_ID --export)
# With Fish
clever addon env ADDON_ID --export | source

clever kv PING # PONG
clever kv set yourKey theValue
clever kv get yourKey

As this demo version doesn't handle all the commands managed by Materia KV, you can send a “raw” command and it will return the answer :

clever kv redis_raw 'hset key field1 value1 field2 value2' # 2
clever kv redis_raw 'hget key field2'                      # value2
clever kv redis_raw 'hgetall key'                          # the full hash

To demonstrate the possibilities of such integration, you also have the option of directly querying a JSON object stored in a key :

clever kv set simpleJson '{"key": "value"}' # the set data
clever kv getJson simpleJson key            # value
clever kv set jsonKey '[{"key": "value"}, {"bigKey": {"subKey1": "subValue1","subKey2": "subValue2"}}]'
clever kv getjson jsonKey bigKey.subKey2    # subValue1
clever kv getjson jsonKey ''                # the full JSON

To show just how simple such implementations can be, we've written a Materia KV client in Go using Go-Redis. If you'd rather understand how you can send commands and receive responses from Materia KV directly in TCP, you'll find examples in both V or Ruby.

Feel free to create your own and let us know ;)

A demonstration of Materia KV on Twitch, a community space

Want to find out more? Horacio González, our head of DevRel, will be giving a demonstration on Twitch on Tuesday June 18 at 13h15 (Paris Time). Pierre Zemb, creator of Materia at Clever Cloud, will also be discussing with our CTO Steven Le Roux in a video published at the end of June on Clever Cloud's YouTube channel.

And to share your views, questions and requirements, go to the Materia-dedicated section of our GitHub community space.

Clever Cloud was founded nearly fifteen years ago, out of a desire to innovate in the field of web hosting and deployment, by offering a reliable range of services in a simple, straightforward way. As a result, we became Europe's leading provider of PaaS services, attracting developers and key accounts alike.

More importantly, we are unique in offering these solutions on our own infrastructure, as well as on-premise or through various partners around the world. We now do this with ISO 9001 and 27001:2022 certification, with HDS and SecNumCloud as our next targets.

Over the years, our company and its product range evolved and expanded. But we've never compromised with our original vocation: innovation. That's why, alongside the continuous improvement of our services, interfaces and integrations, we are massively investing in the development of our future, in solutions that will meet our customers' needs ever more effectively, and over which we have full control.

This is where the independence and sovereignty of our offering comes in. That's why we've decided to build our own infrastructures, to choose a GNU/Linux system to which we actively contribute, and to use it on both our servers and our network equipment. It's also why we've spent the last few years developing a range of serverless, easy-to-use, pay-as-you-go products.

• What’s Clever in our cloud ?

Clever Functions: WASM stands for compatibility and efficiency

To achieve this, we didn't hesitate to create our own low-level virtualization and function execution layer. We wanted to maintain our strong isolation criteria by setting up virtual machines that we activate as close as possible to the CPU. We also wanted to be able to achieve good performance while opening up to a wide range of languages, rather than simply relying on v8 at the risk of focusing almost exclusively on JavaScript.

So we opted instead for a solution that leverages the capabilities of the WebAssembly standard (WASM/WASI), which can be used with a wide range of languages, from Go to Zig as well as C, Rust or AssemblyScript... and of course JavaScript.

After a successful first demo of our Tech Preview in Barcelona at the WASM.io conference last month, we'll be demonstrating again at Devoxx Paris, which takes place at the Palais des Congrès from April 17 to 19. Feel free to come and talk to us about it, and let us know if you're interested in taking part in the next stages of development of this product, which will be progressively opened up over the course of the year.

Materia KV: our distributed, multi-protocol, serverless revolution

Another major innovation from our team is the new Materia offering. To achieve this, we have designed our own serverless databases, built on the FoundationDB open source transactional engine, used, for example, by Apple for its iCloud services. We are also an active member of its community, participating in the development of its ecosystem.

Distributed and robust, notably thanks to its high simulation capacity, it has enabled us to design and build a whole range of services to meet the different needs expressed by our customers in recent years. Our approach is always open and resilient, with deployment across multiple availability zones, compatibility with existing protocols and clients, and pay-as-you-go billing.

The first to be publicly announced is Materia KV, which, as its name suggests, is a key-value database. To create one within your Clever Cloud account, you have no instance size to choose, no storage capacity to worry about. We simply provide you with a host address, a port and a token (based on the Biscuit technology to which we actively contribute), and you're ready to go!

You can then send us your keys and their contents. They will be synchronously replicated in our three Paris datacenters. So you don't have to set up any complex configurations to ensure high availability, or even activate any options. Materia KV is resilient by design! And it adapts to individual needs, as it's billed on a pay-as-you-go basis. It's the perfect solution for hobbyists, seasoned developers and key accounts alike.

Redis API compatibility… to start with

We didn't want this innovation to come at the cost of complex configuration, requiring the use of special clients and ORMs. That's why we've developed Materia KV's own compatibility layers.

To "talk" to it, you don't need a special API or tools specific to Clever Cloud. You can use it with existing solutions for DynamoDB, GraphQL or Redis. The latter is the first of the compatibility layers available in our test phase. It enables customers using a Redis instance for their application to migrate to Materia KV with just a few clicks in their interface or via Clever Tools.

This solution will be publicly demonstrated for the first time at Devoxx Paris, which takes place at the Palais des Congrès from April 17 to 19. It will then be rapidly made available to all our customers in alpha. Feel free to come and talk to us about it, and the other serverless services we'll be offering later this year.

We'll also be demonstrating other upcoming new features, such as our Keycloak-as-a-Service solution in partnership with Please Open It, and celebrating the release of version 1.0 of our in-house, open source load balancer, developed in Rust: SŌZU!

To keep up to date with our regular developments, be sure to follow our Changelog (via its RSS feed), or this blog. But also follow us on LinkedIn or X/Twitter.

When you deploy an application with Clever Cloud, it's not just delivered online and maintained in operational condition. It also comes with a range of services and dedicated features to help you operate it all along its lifecycle. One of these is our log stack.

We upgraded it last year to rely on Vector for emission, Apache Pulsar for ingestion and consumption, while ensuring isolation between each user (multi-tenant) and each application. The previous solution had shown its limitations and prevented us from delivering some enhancements. Our main objective was to improve our users' experience on many different levels.

A new Logs experience at Clever Cloud

We've been using it successfully in our Clever Tools command line interface (CLI) for several months now. You can view your logs live, download those for a given time window, or even filter them, with the results available in a variety of formats. Documentation is available here. Of course, this feature is based on our APIv4, which is also documented there.

The next step was to integrate this new technical brick into the Console. This was an opportunity for us to rethink the section dedicated to logs so that it exploited our Web Components and met our criteria in terms of usability, functionality and accessibility. Our team spent several months rethinking it from top to bottom.

After an internal testing phase, it is now available in beta to all our customers. They can choose it by clicking on a simple link in the interface, and return to the old one whenever they want. Our objective is to continuously improve this section, and then gradually extend it to all our services. To share your feedback, we've created a community space on GitHub for the purpose:

• Go to the Clever Cloud Console
• Give your opinion on our new Logs interface

A myriad of details…

The previous interface allowed you to browse your latest deployments and their various instances one by one. The new interface goes one step further, listing by default all deployments and instances of an application over the last 7 days. The logs of the most recent deployment will be loaded, but you can also add or remove those of the instance of your choice. Each deployment is identified by the hash of the related code commit, with instances numbered (in the case of horizontal scalability).

You can reduce the time window to a few days or a few hours, or even manually indicate start and end times, which is useful when you're looking for specific information, for example, when tracking down a bug. You also have a text filter above the main area. It will evolve with the introduction of an indexing mechanism that will open up new possibilities in this area.

Of course, you can also follow the publication of your logs live. This will be the section displayed by default when a deployment is in progress.

... and of features

The main area lets you select one or more lines with the keyboard and mouse via the gutter on the left, with the option of copying the result to the clipboard just as simply by pressing CTRL+C/Cmd+C. At the top right you'll find a button to set the area to full screen, another to return to the bottom of your logs and a settings area.

Although it's still new, it's already quite complete and uses our new API for user settings. It lets you choose between different light/dark themes, remove colors and ANSI codes, line wrap or not, define a format for date and time, choose whether to display in your time zone or in UTC, distinguish instances by name, etc. Of course, this is subject to change, so don't hesitate to make suggestions.

We'll keep you posted on future improvements in our Changelog (via its RSS feed), or on this blog for the next big steps between now and general availability. Don't forget to follow us on LinkedIn or X/Twitter to keep up to date.

To ensure a Clever Cloud application has enough resources allocated in our infrastructure, we use metrics to know when to scale up/down.Those metrics, along with those from our internal systems, are stored into a time series database which is growing at a pace of 2 TB per week. As we want to keep them for any further analysis, we had to set up a solution to automatically offload this data every month into our Cellar object storage.

Our metric infrastructure

With thousands of customer applications, and hundreds of internal applications, we needed a time series database capable of ingesting hundreds of thousands data points per second.

That’s why our customer-facing cluster,warp10-c2, in production since October 2023, is using Warp10. It effortlessly satisfies our performance requirements thanks to the underlying storage layer, FoundationDB. In its distributed version, it can handle ingestion spikes of over 500,000 data points per second while processing reads of over 5,000,000 data points per second.

Continuously growing cluster

Even though we could handle metric data efficiently, the growing rate of our database (2 TB per week) has put us in a challenging situation.At first, we were handling stability issues. To focus on one of them without having to deal with storage problems, we added more SSD nodes to our FoundationDB cluster.

Once this situation was handled, we had to deal with computing resources needs, find a good balance and how to gain storage capabilities without losing data.

In the end, we found a good way to achieve this: offloading those metrics in our object storage (HDD) service (Cellar) and once done, delete them in the hot (NVMe SSD) storage.

Offloading data can be done through various ways

First and naive way would be to select data points and send them into buckets in our own format. Our query system can handle millions of reads per second, but we are talking about billions of data points here. Even with dedicated  servers, it would have led to performance issues coming from our storage layer. In addition, we would have disk usage similar to  data in the hot storage.

A second way would have been to reduce the resolution of metrics, instead of having data points with seconds or minutes precision, we could re-process with their average value per hour or per day. This solution would leave us with more questions than solutions: “How to handle metrics extreme values ? An application running only a few seconds or minutes and not a day? What about our internal applications, do we re-process metrics on a case-by-case basis?” It would have been too much effort put in a flawed system.

The last and final way would have been to use Warp10 HFiles proprietary extension. It can be added on internal machines of our c2 cluster, and used to generate compact and data-efficient files, with encryption capabilities. This solution could lead us to a compression ratio that can take TB of data into a few GB, this is due to metrics that are mostly the same (e.g a CPU usage percentage has a finite number of values).

In February, we’ve chosen this approach. Still, we should implement this system, with automatic processing, so that our data team doesn't have to deal with this tedious task every month.

Building the automated process

Before diving into how we made this possible, a small introduction into HFiles is necessary to better understand why we did it this way. Then, we’ll dive into a workflow orchestrator we are using offload automatically. And finally, the figures from our first batch.

Introduction to Warp10 HFiles

HFile is part of the Warp10 HFiles extension. Its process is fairly simple. At first, it will fetch all `s matching a criteria, and for each of them, try to gather all the data points in a specified time range, while continuously compressing the data. This can take a lot of time, within our experiments, some generations could take days to be done (yes, really). But with proper optimisation, days could become hours. Those generations are synchronous, meaning we need to keep an open HTTP connection with the machine running the generation in order to succeed.

We mostly adjusted three parameters in order to optimize the processing time:

• Data points time range: it allows to only fetch data points between two dates
• Concurrency: running a single HFile generation is time consuming, so running multiple generations in parallel can save a lot of time
• Number of series: for each series, it will try to fetch its data points (even if it’s zero in the specified time range). Thus, optimizing the number of series can save a lot of requests into our storage layer, and then save time

What do "series" mean?

A series, called GTS (“Geo Time Series”) in Warp10, is composed of a metric name and one or multiple labels which gives context. For instance, a GTS to know the CPU usage of an application would be cpu.usage_guest{app_id='app_87e33fea-9dde-4a99-b347-674de382ff7b'}. In our c2 cluster , we got over 150 million of them.

Scheduling the offloading

Those generations required a tool that met the following constraints:

• Scheduling capabilities: to prevent having to trigger the action manually each month
• Retries and fail over: in case of failure of a HFile generation, it should be able to retry, or if a scheduling node fails, to run it again on another node
• Alerting, when retries aren’t enough, we must be aware of the failure
• Handle long tasks: as our generations can take several hours

We went through several workflow orchestration solutions, but our final bet went to Kestra. It’s an open source tool that answers the above constraints. Deployed as a Clever Cloud application, it helped us to create workflows that minimise generation time by using our optimised parameters for HFile generation.

At the end of February, we finished our HFile generation workflow (see above), and ran it for the first time on November metrics, without deletion. This test-run, once successful, gave us enough confidence in Kestra. We ran the deletion manually afterwards.

However, starting in March, the whole HFile generation and data deletion from hot storage has been automated. Although our need for this scheduler is limited to this use case, we intend to apply it for different scenarios in the future.

November's metrics offload

As said earlier, we ran our first month of offloading successfully, based on November data (we want to continuously have at least 3 months of metrics available from hot storage). This first month was special, as c2 entered into production at the end of October, and all Clever Cloud customers applications had to be re-deployed using it. It means that this month has a higher number of series and data points than any other.

Thus, last November, we offloaded over 22 billions data points from 33 millions series. These needed approximately 20 TB of data from hot storage. Once compressed, it fits in only…20 GB. An important thing to note here is that the data in the hot storage is encrypted, meaning the size is several times the actual data point or series size.

In conclusion, managing metric data efficiently, especially at a growth rate of 2 TB per week, presented challenges that required careful planning. By leveraging Warp10 HFiles' data compression capabilities and building an automated offloading system to Cellar object storage, we successfully optimised our storage resources.

As you know, you can deploy your applications and add-ons via Clever Cloud, on Health Data Hosting (HDS) certified servers, on tiers 1 and 2 of the standard, provided by OVHcloud in Roubaix. We also provide an object storage service (Cellar) from a cluster of such servers in Roubaix and Gravelines (separated by ~100 km), available as a fr-north-hdsregion.

However, we had a growing demand for an HDS zone in Gravelines. independently for applications and add-ons. So our teams worked to enable you to deploy on HDS-certified servers in this zone, from clever-tools or the Console as grahds.

Perfect for providing redundancy for your healthcare data applications!

We'd like to take this opportunity to remind you that Clever Cloud's services and infrastructure were awarded ISO 9001 certification last year. We are also awaiting final approval for ISO 27001. Finally, HDS certification and SecNumCloud qualification will be our next projects.

If AI has been in the news recently, it's mainly through large language models (LLMs), which require substantial computing power for training or inference. Thus, it's common to see developers using them via turnkey APIs, as we did in a previous article. But this is changing thanks to more open and efficient models and tools, which you can easily deploy on Clever Cloud.

Indeed, while our platform was designed for a variety of uses, we always paid particular attention to its flexibility and the freedom it gave our customers. This means they can easily exploit it to address new markets, with minimum effort spent on adaptation.

To confirm this, we recently tried to deploy open source AI models such as those derived from LLaMA and Mistral. Their particularity is that they exist in lightweight versions, which don't necessarily require a GPU to run. As a result, they can be hosted as simple Clever Cloud applications, as long as they have been allocated enough memory.

Let's ollama (with a web UI) !

To do it, we've chosen to use ollama, an MIT-licensed project that lets you download models from a registry, or even your own. It works as a server that communicates via an API, and is very easy to host on our infrastructures.

• Create a Clever Cloud account (and get 20 euros in credits)

To get started, you’ll need a Clever Cloud account. You can create the application and follow the next steps using our web interface, the Console, but also through our open source CLI: Clever Tools. That’s what we’ll describe below.

We’ll assume that your machine is running git and a recent version of Node.js. If you haven’t yet installed Clever Tools on your system, type (with system administrator rights, or sudo if necessary):

npm i -g clever-tools
clever login

Once logged in, you can check that everything has gone smoothly with the following command:

clever profile

Next, we create a Clever Cloud application using Node.js. It will install ollama to download and run our models, adding a web interface (chatbot-ollama, also under MIT license) that will allow us to query the server from any browser, run from a computer or a simple smartphone.

This solution also offers local storage of chats in the browser. It therefore requires no database, but offers an import/export function. You can also choose between different models, set some of their parameters, configure system instructions, various prompts, and so on.

mkdir myollama && cd myollama
git init
curl -L https://github.com/ivanfioravanti/chatbot-ollama/archive/refs/heads/main.tar.gz | tar -xz --strip-components=1

clever create -t node myollama && clever scale --flavor L
clever env set CC_POST_BUILD_HOOK "npx next telemetry disable && npm run build"
clever env set CC_PRE_BUILD_HOOK "./ollama_setup.sh"
clever env set CC_PRE_RUN_HOOK "./ollama_start.sh"

echo "orca-mini" > models.list
echo "mistral" >> models.list
echo "codellama" >> models.list

The code above creates the application. We add a file containing the list of models we wish to fetch during startup. We've chosen an L instance, the minimum size for loading such models into memory. We also declare several environment variables for running scripts to install ollama, start the server and disable telemetry for the Next.js framework used to build the web interface after the dependencies have been installed.

Now add some scripts

Our application is almost ready. We still have to create two executable files:

touch ollama_setup.sh && chmod +x ollama_setup.sh
touch ollama_start.sh && chmod +x ollama_start.sh

ollama_setup.sh download the ollama binary and the models from our list. Open it with your favorite editor and insert the following content:

#!/bin/bash

# We define the folder (in $PATH), where the ollama binary will be placed
BIN_DIR=${HOME}/.local/bin
mkdir -p ${BIN_DIR} 

ollama_start() {
    echo "Downloading and launching ollama..."
    curl -Ls https://ollama.ai/download/ollama-linux-amd64 -o ${BIN_DIR}/ollama
    chmod +x ${BIN_DIR}/ollama
    ollama serve &> /dev/null &
    echo -e "Finished: \033[32m✔\033[0m\n"
}

get_models() {
    MODELS_FILE="models.list"
    MODELS_DEFAULT="orca-mini"

    # We check if the models file exists
    if [ ! -f ${MODELS_FILE} ]; then
        echo "File ${MODELS_FILE} not found, ${MODELS_DEFAULT} used by default."
        echo ${MODELS_DEFAULT} > ${MODELS_FILE}
    fi

    # Loop through each line and pull model
    while IFS= read -r line; do
        echo "Pulling ${line} model..."
        ollama pull "${line}" > /dev/null 2>&1
        echo -e "Finished: \033[32m✔\033[0m\n"

    done < ${MODELS_FILE}
}

# We start ollama, and wait for it to respond
ollama_start
if [ $? -ne 0 ]; then
    echo -e "\nError during ollama setup."
    exit 1
fi

count=0
MAX_TIME=10

while ! pgrep -x "ollama" > /dev/null; do
    if [ ${count} -lt ${MAX_TIME} ]; then
        sleep 1
        count=$((count+1))
    else
        echo "Application 'ollama' did not launch within 10 seconds."
        exit 1
    fi
done

get_models
if [ $? -ne 0 ]; then
    echo -e "\nError during models downloading."
    exit 1
fi

This code may seem a little long, but that's mainly because we've made it modular, logging only successful steps and any errors.

ollama_start.sh detects whether the ollama server is already present before starting the web interface. If it isn't, we launch it in the background (we'll see why below).

Open it with your favorite editor and insert the following content:

if ! pgrep -f "ollama" > /dev/null; then
    # If 'ollama' is not running, start it in the background
    ollama serve &
    echo "Application 'ollama' started."
else
    # If 'ollama' is already running
    echo "'ollama' application is already running."
fi

Once these two files have been created, we can deploy the application:

git add . && git commit -m "Init application"
clever deploy
clever domain

You'll then see the domain to access it (which you can customize).

Cache and quick (re)start

Using Clever Cloud not only means deploying such an application easily, with logs and monitoring, and being able to instantly change its horizontal and/or vertical scaling. It also means being able to turn it into an image ready for a fast redeployment.

In fact, every application we host is stored as an archive just before dependencies are installed. This means it can be stopped and restarted in this form, so we don't have to re-download the models every time. To do this, simply specify the folders to be added to the archive:

clever env set CC_OVERRIDE_BUILDCACHE "/:../.ollama/:../.local/bin/"
clever restart --without-cache --follow

The application will restart, but this time it will take the built web interface, ollama and models from its cache, as the CC_PRE_BUILD_HOOK and CC_PRE_BUILD_HOOK steps are not executed. This is one of the reasons why we need to make sure that we start the ollama server with CC_PRE_RUN_HOOK, but check that this hasn't already been done in a deployment where it is used to download models before running them.

Once this step is complete, you can stop the application and restart it whenever you like - it'll be much faster from now on. And you'll only be charged for the time you use it, by the second. If you wish to modify the list of models to use, do so in the dedicated file. A git push will be enough to rebuild the new cache.

# Stop/Restart the application
clever stop
clever restart

# After a modification in models.list, git push to recreate the cache
git add models.list && git commit -m "New models" 
clever deploy

You'll find the files you need for this project in this GitHub repository, with the environment variables ready to import (clever env import < .env).

And if you want to push things further, deploy other models and tools for your applications that require more power and/or GPUs, let us know!

At the beginning of the Internet as we know it, about thirty years ago, the primary choice of a developer when he wanted to put a website online was to choose between a dedicated server where he was the only one in charge, and a hosting shared between several customers, more or less numerous depending on the offers.

If the weak isolation provided at the time in the shared hosting model pushed the most careful to use dedicated servers, economic factors also made the choice: not everyone could afford his own machine. Nevertheless, with time, low-cost offers came to shake up the market, then it was the turn of the "Cloud" and "as-a-service" instances that are no longer rented monthly but per hour or per second. This has changed practices in depth.

Reduce resource waste (and headaches)

This is the origin of Clever Cloud, founded on a conviction: we had to take advantage of these new opportunities to reinvent shared hosting practices, which were still based on old transfer protocols, by adding real isolation and ease of use through automated deployments and load adaptation. Create tools to improve the life of developers and ops (professions sometimes combined from now on), take benefit of the ongoing revolutions caused by the emergence of Git and platforms such as GitHub or GitLab, of continuous development and integration (CI/CD).

• What is Clever Cloud ?

Both then and now, it is common to see companies continuing to rent "old-fashioned" dedicated servers to deploy their applications using more or less automated tools, but this often leads to a waste of resources. The reason for this is that these machines are dimensioned to meet any peaks that may occur, and therefore are rarely used to their full potential.

An oversizing which is not really encouraging energy efficiency. In March, The Register reported that according to the US Government Accountability Office (GAO), only 5% of the server resources available to the US government were used. A situation that shouldn't happen, in the US or in Europe.

Part of the hosting industry is now built around this lack of efficiency. The datacenter players are decreasingly technical and increasingly on a model borrowing from the real estate market: you have to use the available space and first of all fill square meters (no matter if it means reserving large spaces for crypto-currency miners). But whether the servers are used or not makes no difference: it's rented. By the way, if they are running at a low level, it relieves the electrical infrastructure, the lessor is all the more winning.

One solution could be to use instances (IaaS), which can be turned on or off, customized as needed, with better resource management by the cloud service provider (CSP) since they are shared between different customers. But this requires developers and ops to constantly fit the infrastructure to the load, take care of updates and security, be the ones on whom everything relies.

But in practice, this essential work is not always done. This is a problem that we tried to solve, to pool efforts and avoid scares and headaches for our customers' teams.

Applications up and running…

Clever Cloud's pledge is that "you develop, we deploy". So we are more than a Platform-as-a-Service (PaaS) solution as some hear it, rather a kind of "Software Defined Hosting", "Serveless" style, in the way you never have to worry about the infrastructure. Only your code and where you want to deploy it.

In fact, our job is to maintain your applications in operating conditions. This means that we make the deployment process as simple as possible, making it fully automated and part of your team processes. We keep your applications up and running, whether you need to upgrade or experience peak loads, we also manage backups, disaster recovery, and more.

All of this with the flexibility to scale horizontally (more instances available) or vertically (higher performance instances). A choice that can be made in a fully automated way (auto-scaling) or manually through our web interface and our tools (CLI, SDK, etc.): you remain in full control of your choices.

We also update the system and the software bricks that we operate, restricting them as strictly as possible to what is necessary to achieve maximum efficiency, performance and security for our customers. Thus, we do not neglect software isolation and all applications run in their own virtual machines.

… in a flexible and secure environment

It is important to understand that the security attacks and vulnerabilities are mostly a financial issue. For those who exploit them, it is a market. If there will always be methods to attack a company and a service, our challenge is to reduce as much as possible the attack surfaces and to break the potential profitability of attacking Clever Cloud.

This is why we compile the Linux kernel and all our binaries from source, using the Exherbo distribution. This allows us to reduce them to a bare minimum, as well as the attack surface, and to have constantly updated software without having to wait for the maintainers to take care of it.

This offers another advantage, as our CEO Quentin Adam regularly says: "any compiled binary released is a binary waiting to be exploited". Indeed, they are used by a large number of users and are a target of choice, which can be easily detected. Compiling our own binaries allows us to protect ourselves from such risks "by design".

What if a datacenter has a problem? Simply deploy elsewhere. This resilience is the basis of our technical decisions over the last few years, with a strong multi-cloud approach, whether in the development of our own infrastructure or by allowing you to take advantage of those of our partners. And if you want to have your own Clever Cloud "on-premise" or in a SecNumCloud qualified infrastructure? That's possible. That's what openness is all about.

Highly utilized servers, designed to last

This freedom of choice will enable us to offer deployments that can be driven by geographical considerations, commitment to diversification or eco-design. We are now switching to servers based on the work of the Open Compute Project (OCP). With a centralized power supply within the rack, they are designed for greater energy efficiency.

Consume less for the same level of service is an obsession of our teams, a strategy we apply both in the short and long run, through optimization of software and hardware, better use of existing resources, and the introduction of better components when they are added or renewed. This strategy is at the heart of our business model, but also part of our efforts to reduce our environmental footprint, and therefore the one of our customers.

We recently chose to use Capri servers based on AMD EPYC 7003 processors with 64 latest generation cores, offering a very good level of performance and density. They are now installed in our third point of presence in Paris, Green Data Nanterre. The Flashbox will provide us with remote storage within the network, in CPU-less servers, using chips from the French company Kalray, very energy-efficient.

These innovations and this density level allow us to strengthen the Clever Cloud commitment: keep as many applications up and running as possible, with a good level of resilience but the least infrastructure required. Because the best way to avoid wasting resources is to consume as little as possible "by design", which we allow our customers to do on a daily basis without having to worry about anything. To better inform them, we are also working on delivering metrics such as PUE, but also the consumption and impact of our servers, their applications, etc.

Our servers are designed to operate in our infrastructure for many years, to meet needs that may change over time, and will be replaced only when necessary. By relying on standards such as OCP, which are used by a growing number of players in the market, we can ensure that they will be easily reusable for a second life cycle by third parties.

For our future deployments, we will explore alternative ways to further improve the efficiency of our servers and their components, including cooling, which is a key issue in data centers, with the rise of direct liquid cooling and immersion, two technologies that can be implemented through Green Data. We are also in discussions with various partners about the reuse of waste heat produced by servers.

Coming next? Network Groups, Tasks, GPU, FaaS, IaaS

Clever Cloud matches its company logo: multi-faceted. While we've long been focused on building our platform, over the last few years our customers have asked us to go further. We listened: the team has doubled in the recent months to increase our core strength: innovation through software, using the hardware at its best.

Thus, in addition to the continuous improvements of our Console, we are currently finalizing a project that has been one of our biggest achievements: the Network Groups, allowing you to link your applications, but also your machines through a private network (VPN). For this, we rely on the open source solution Wireguard. The feature is finalized and is currently being tested internally, it will be released in public beta in a few weeks.

We will soon offer our VM Tasks, with the ability to run tasks on demand, which can be activated in different ways. We've also started to rework our GPU computing offering, so don't hesitate to ask us if you have any needs in this area, as preliminary tests can be conducted. Later this year, we'll offer access to our Function-as-a-Service (FaaS) solutions for large-scale, on-demand function processing and IaaS, so you can deploy the operating system and tools of your choice via Clever Cloud, without relying on solutions such as Docker.

And for Kubernetes enthusiasts, in addition to the recent launch of our operator, a managed service will also be available… as well as other surprises that we'll talk about in the near future. Don't hesitate to submit your requests and suggestions, which can help us refine our priorities and developments. Our team is listening to you.

Dans cet épisode il on chasse les trolls avec 1xEngineer avant de rendre possible l'impossible, puis de typer avec Rust. Nous parlons également de CockroachDB, de battements de coeur, de casseurs de circuits, de transformation de paquets et de Pixelation pour finir comme des voyous en musique.

👋 Venez discuter avec nous sur @clever_cloudFR pour nous dire ce que vous avez pensé de ce nouvel épisode.

➡️ Pour découvrir ou réécouter d’anciens épisodes c’est par ici !

Timecode & Liens

00:00:00 Introduction

00:03:15 1xEngineer
https://1x.engineer/
De la bonne bienveillance

00:09:00 "Making Impossible States Impossible" by Richard Feldman (Hubert)
https://www.youtube.com/watch?v=IcgmSRJHu_8

00:22:35 Le typage en Rust (Yannick)
https://fasterthanli.me/articles/the-curse-of-strong-typing

00:34:50 Enabling the Next Generation of Multi-Region Applications with CockroachDB (PZ)
https://www.cockroachlabs.com/blog/sigmod-2022-cockroachdb-multi-region-paper/

00:42:50 Phi φ Accrual Failure Detection (François)
https://medium.com/@arpitbhayani/phi-%CF%86-accrual-failure-detection-79c21ce53a7a
https://www.researchgate.net/profile/Xavier-Defago/publication/29682135_The_ph_accrual_failure_detector/links/0a85e53ce412e3b069000000/The-ph-accrual-failure-detector.pdf

00:47:20 Will circuit breakers solve my problems? (François)
https://brooker.co.za/blog/2022/02/16/circuit-breakers.html
https://radlab.cs.berkeley.edu/people/fox/static/pubs/pdf/c18.pdf

00:50:47 Comment transformer un paquet de bytes en un double (Yannick)
https://blog.m-ou.se/floats/

01:00:00 Never, Ever, Ever Use Pixelation for Redacting Text (Hubert)
https://bishopfox.com/blog/unredacter-tool-never-pixelation

01:05:44 Musique de fin
Rogue Legacy 2 OST - Axis Mundi 1
https://www.youtube.com/watch?v=f_QIN57e48A

Keycloak is a modern and efficient way to deal with identity and access management. It allows for single sign-on, user federation across multiple user directories. It is also compliant with standard authentication protocols like OpenID Connect, OAuth2 or SAML. All details are available on its website. This blogpost will go through steps to deploy Keycloack and Grafana on Clever Cloud.

How to deploy Keycloak

Setup

Keycloak is written in Java, and gives a JAR file in their release, which means we will use a a JAVA runtime with a JAR deployment. It also requires a PostgreSQL Database, which means we will deploy a PostgreSQL addon.

# Get lastest Keycloak release
wget https://github.com/keycloak/keycloak/releases/download/17.0.1/keycloak-17.0.1.zip

# Extract the archive
unzip keycloak-17.0.1.zip

# Go to the folder
cd keycloak-17.0.1

# Create the jar application
clever create --type jar keycloak-server
export KEYCLOAK_DOMAIN=$(clever domain | xargs)

# Create the PG addon
clever addon create postgresql-addon --plan s_sml --addon-version 10 keycloak-postgres

# Link the addon
clever service link-addon keycloak-postgres

Configure

For now we will configure Keycloak through environment variables. Note that you can also do configure Keycloak through its CLI or through a file, as stated in its documentation.

Note: For the configuration through environment variables; the pattern to use is KC_ and the uppercase name of any configuration key, separated by underscores.

# Export env for templating purpose
export $(clever env | sed '/^#/d;s/\"//g')

clever env set CC_JAR_PATH lib/quarkus-run.jar
clever env set CC_JAVA_VERSION 11
clever env set CC_RUN_COMMAND "bin/kc.sh start --auto-build"

clever env set KC_DB postgres
clever env set KC_DB_USERNAME $POSTGRESQL_ADDON_USER
clever env set KC_DB_PASSWORD $POSTGRESQL_ADDON_PASSWORD
clever env set KC_DB_URL "jdbc:postgresql://$POSTGRESQL_ADDON_HOST:$POSTGRESQL_ADDON_PORT/$POSTGRESQL_ADDON_DB"
clever env set KC_HOSTNAME $KEYCLOAK_DOMAIN
clever env set KC_HTTP_ENABLED true
clever env set KC_PROXY passthrough
clever env set KC_PROXY_ADDRESS_FORWARDING true

Note: Consider setting KC_DB_POOL_MAX_SIZE at a lower value (default is 100) regarding the database size you ordered earlier. You can find the connection limit here. If you used the S_SML size that is in the command earlier, you can ignore this message. You can also fine tune the database connection pool using KC_DB_POOL_INITIAL_SIZE and KC_DB_POOL_MIN_SIZE

Now, you need to define KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD :

clever env set KEYCLOAK_ADMIN <choose an admin username>
clever env set KEYCLOAK_ADMIN_PASSWORD <choose an admin password>

Deploy

Deploy Keycloak on Clever Cloud:

# Init a git repository
git init

# Add your files
git add .

# Create the first commit
git commit -m "clever init"

# Deploy the application
clever deploy

# Open Keycloak in your browser
clever open

Validate the configuration

You can follow the Keycloak documentation starting here to ensure Keycloak works as expected. It will take you through a realm (tenant) creation, user creation and client creation/test.

Create an Initial Keycloak configuration

In the keycloak UI, ensure you create a Realm called "myrealm" (hover "Master" on the top left corner). Ensure the realm "myrealm" is selected.

Then, go to Users > Add a User. Fill the Username, Email, First Name and Last Name fields, then hit save.

In the credentials tab of the user, add the password of your choice and deselect "Temporary", the set password.

You're all set.

A real world scenario

Let's see how we can use Keycloak in the real world: as an SSO Proxy for Grafana, using Oauth 2.

Setup Grafana on Clever Cloud

If you need a in-depth guide on how to use or deploy Grafana please read this article, the following is a helper to illustrate the usages of Keycloak.

Let's use the grafana-example on Clever Cloud's Github repository.

# Clone the repository
git clone https://github.com/CleverCloud/grafana-example.git

# Enter the directory
cd grafana-example

# Create a NodeJS Application
clever create --type node grafana
export GRAFANA_DOMAIN=$(clever domain | xargs)

# Configure Grafana
clever env set GRAFANA_VERSION 8.5.2
clever env set GRAFANA_PLUGINS "grafana-worldmap-panel"
clever env set GF_PLUGIN_DIR ./data/plugins

clever env set GF_SERVER_HTTP_PORT 8080
clever env set GF_SERVER_ROOT_URL "https://$GRAFANA_DOMAIN"

Note: In this section you need to set the GRAFANA_SHA_256 variable yourself. You can find the SHA256SUM required here, next to "Standalone Linux Binaries"

clever env set GRAFANA_SHA_256 <SHA256SUM>

Configure Keycloak for OAuth 2

Note: In this section, you need to replace $GRAFANA_DOMAIN with the value you exported in the last section. You can find this value by running the following command: clever domain

In the Keyloak UI, Go to Configure > Clients > Create

Create a new client with these configurations:

Client ID: $GRAFANA_DOMAIN
Client Protocol: openid-connect
Root URL: https://$GRAFANA_DOMAIN

Further make these configurations:

Access Type: confidentials // The OAuth client must use a client id and secret.
Root URL: ${authBaseUrl}
Valid Redirect URIs: https://$GRAFANA_DOMAIN/login/generic_oauth
Base URL: /login/generic_oauth

Clear Admin URL and Web Origins.

Click save and open the Credentials tab. Copy the Secret into a separate note, we will need it in the second and third part of this tutorial.

Open the tab Roles and click Add Role. Create a new role with name admin. This role defines the access level for Grafana.

Head over to Scope tab and set Full Scope Allowed to OFF. We do not want to share any other details about the realm in the client token.

Then, we are going to configure a client mapper for the roles property. We must ensure that Grafana can extract the access role from the JWT token. Open the Mappers tab and click on Create. Create an entry with these options:

Name: Roles
Mapper Type: User Client Role
Client ID: $GRAFANA_DOMAIN
Token Claim Name: roles
Claim JSON type: string

Finally, assign the client role to your Keycloak user: Go to Users > View All Users and click on the ID of your user. Click on the Role Mapping tab, then in the Client Roles selector, select $GRAFANA_DOMAIN. In the Available Roles section, select admin and click Add selected.

Configure Grafana for OAuth 2

# Configure appropriate env vars
clever env set GF_SERVER_DOMAIN $GRAFANA_DOMAIN
clever env set GF_SERVER_ROOT_URL "https://$GRAFANA_DOMAIN"
clever env set GF_AUTH_GENERIC_OAUTH_ENABLED true
clever env set GF_AUTH_GENERIC_OAUTH_NAME Keycloak
clever env set GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP true
clever env set GF_AUTH_GENERIC_OAUTH_CLIENT_ID $GRAFANA_DOMAIN

clever env set GF_AUTH_GENERIC_OAUTH_SCOPES profile
clever env set GF_AUTH_GENERIC_OAUTH_AUTH_URL "https://$KEYCLOAK_DOMAIN/realms/myrealm/protocol/openid-connect/auth"
clever env set GF_AUTH_GENERIC_OAUTH_TOKEN_URL "https://$KEYCLOAK_DOMAIN/realms/myrealm/protocol/openid-connect/token"
clever env set GF_AUTH_GENERIC_OAUTH_API_URL "https://$KEYCLOAK_DOMAIN/realms/myrealm/protocol/openid-connect/userinfo"
clever env set GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH "contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'"

Note: In this section, you need to define GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET yourself, using the value in Keycloak UI, under Client > $GRAFANA_URL > Credentials

clever env set GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET <KEYCLOAK_CLIENT_SECRET>

You can now deploy Grafana:

# Deploy grafana
clever deploy

# Open grafana and try the Login with Keycloak button !
clever open

Now hit login with Keycloak, and use the username and password you defined for the user you created Earlier in Keycloak.

Using the same procedure describe earlier to create the first user, you can now create more users and roles. Create the roles "editor" and/or "viewer" and assign them to your new users within Keycloak. Try to connect with those on Grafana and you will see the roles are propagated to Grafana.

Sources:

• https://janikvonrotz.ch/2020/08/27/grafana-oauth-with-keycloak-and-how-to-validate-a-jwt-token/
• https://www.keycloak.org/getting-started/getting-started-zip