In my last article I created a simple Canvas workpad in my Kibana. To do so I already had meetup data indexed in my Elastic instance. Today I am going to tell you how I did gather data and added it to Elasticsearch.

Selecting meetups I want to track

The first thing to do is to select a list of meetup groups you want to keep track of. There are many ways to do that using the Meetup API. For instance, you can search to similar groups using the similar_groups endpoint. I let you read the API documentation to find a way to select the events you will track. You just need to format the response to extract the cities and the names of the Meetups groups in a JSON file formatted as follows:

{
  "City name 1": [
    "meetup group name 1",
    "meetup group name 2",
    ...
  ],
  "City name 2": [
    "meetup group name 1",
    ...
  ]
}

Getting data about meetup events

Once you have this, you can use the node.js application we created. Of course it has the Elasticsearch dependency in package.json. You are strongly invited to check out the source code at this point. You can see in the index.js many parts of interest:

• The creation of an express server listening on port 8080. We need it, so Clever Cloud will know our app is up running.

app.get('/', (req, res) => {
    res.send('Hello !');
});
app.listen(8080, () => console.log('Listening on port 8080!')); 

• The meetup API call:

axios.get(`https://api.meetup.com/${meetupName}/events/?status=past,upcoming\&fields=comment_count`)

• the creation of Elasticsearch indexes:

await client.indices.create({
  index: "meetup",
  body : {
    "mappings": {
      "properties": {
        "time":  {"type": "date", "format": "epoch_millis"},
        "group.name": {"type": "keyword"},
        "yes_rsvp_count" : {"type": "integer"},
        "grouploc": {"type": "geo_point"},
        "venueloc":{"type": "geo_point"}
      }
    }
  }
});

• The creation of another server where our meetup API calls happen, listening on port 8081. We can also notice that we've restricted our server to allow only localhost connections.

localapp.listen(8081, 'localhost', function() {
  console.log("... port %d in %s mode", 8081, localapp.settings.env);

Now in ./clevercloud/cron.json you can notice a cron task, wich will trigger a curl on http://localhost:8081/ every night at 1 AM:

"0 1 * * * /usr/host/bin/curl http://localhost:8081/"

It is this cron that will call our second server to trigger the meetup API calls.

True fact: to use it in its current version, you must keep your application running all the time for one hour of usage maximum. A way to improve the application regarding this issue would be to implement authentication to our application, so we still are the only one having access.

Then remove the cron from this project to have it running in your main application instead. Your main application will be the one consuming this indexed data. Taking advantage on the fact that every virtual machine running on Clever Cloud already has the Clever Tools CLI installed, we could improve our cron to start the application for an hour then stop it when it has finished its indexation job.

So you will end up with two machines, one with your main application, and the second one running for one hour each night.

We must also know that Clever Cloud does not monitor what's going on on port 8081. You could add a logging system or use Elastic APM to monitor your application during its execution time.

This is an approach among many others, do not hesitate to talk with us about your own implementation.

Okay, let's go back to our main goal, and to do so, you can use our sample data meetup list or use your own by replacing the json in the meetups.jsonfile.

Try it out

You can $ git clone the repo in your console, and go into your Clever Cloud console.

Under the organization of your choice, select New, Application, Node. When prompted if you need add-ons, select Elastic Stack, select the plan you need and enable Kibana as an option.

In the environment variables menu of your application, add NODE_ENV=production and add the provided clever remote to your local git folder. Then push using git push -u clever master.

Your deployment will start and thanks to the ES_ADDON_URI we provided in our index.js file, we have nothing else to configure, our application will start sending data to elastic.

Visualize your data and go further

Either in your Kibana or Elastic instance menu in the Clever Cloud console, in the information page you will find a Open Kibana button. Click it and login using your Clever Cloud credentials.

Into Kibana click on the Management (gear) icon in the left side menu. Under the Kibana title, select Index Patterns, then meetup* to see how the data is indexed.

Of course at this point, you are able to do the exact same as I did in the previous article video.

Here is the ElasticSQL query I used in the Canvas demonstration:

SELECT AVG("yes_rsvp_count") AS average, "group.name" FROM "meetup*"
GROUP BY "group.name"
ORDER BY average DESC
LIMIT 5

Happy indexing!

Good news everyone, we are really excited to offer the Elastic Stack — 🔥Platinum Version 🔥 — on Clever Cloud. It's the first as a service offer that is officially supported by Elastic from a French company, with datacenters located in France.

If you are looking for an Elastic Stack provider dealing with the American Cloud Act problematics and the GDPR regulations, look no further. :)

We are proud to partner with Elastic and offer you the full Elastic Stack on Clever Cloud.

Am I talking about X-Pack? Yes and no, I am talking about so much more. You will have access to Kibana Canvas & Lens, Kibana Spaces with full security (encryption, RBAC, field and doc-level security), Alerting, Elasticsearch SQL, Machine Learning, Metrics, logs, ... If you want the full details about what is available on Clever Cloud, take a look at the Platinum column on this page.

Our starting price is 17 euros per month. With this you can get the full extent of the Elastic Stack, at a very small scale. On the other hand, you can go all the way up to 64 CPUs and 256Go of RAM per node. Our support team will manage the first levels of support while being able to escalate to Elastic's team when needed. You are in good hands!

And of course we worked on an integration with the rest of Clever Cloud. Please have a look at our documentation for the details. Here's a quick glance at what we did.

Specific Clever Cloud Integrations

We have worked on our Elastic Stack integration on several fronts. When you provision the Elastic add-on, we allow you to provision Kibana and an APM server as traditional Clever Cloud applications. It means that they benefit from all the goodness that we bring to applications. You can turn them off if you want to, you can enable auto scalability, you can link them to other applications, really anything you would do with traditional applications. Let's see each integration a bit more in details.

Authentication

Authenticating to Kibana is available through an automatically configured SSO. Every member of the organisation the addon has been deployed to can use their Clever Cloud account to authenticate. No configuration is required on your part.

Elastic APM

If you link the APM server application to any of your application, the right environment variables will be injected and automatically picked by the APM agent in your dependencies. Then simply authenticate to Kibana and start setting up your APM! This feature will be showcased in a dedicated blog post in the coming days.

Backups

We are introducing a new way to manage your add-on backups. When we create your new add-on, we also create a Cellar add-on (our S3-compatible object storage solution) named Backups. All your backups will be stored there. We are starting with Elasticsearch but other databases will soon follow.

What's next?

We plan to provide an even better integration with the Elastic Stack. We are currently thinking about the best way to integrate Beats to our applications. You can currently do it manually but wouldn't it be nice if this was automated? This should really ease the usage of Elastic SIEM for instance.

And of course we are working on automatic cluster provisioning, not just for Elastic. In the meantime you can provision as many nodes as you need and contact our support team to put them in the same cluster.

We are supper excited about working with Elastic and hope you will be as excited to try it on Clever Cloud! We will publish more blog posts in the coming days, highlighting some of the awesome capabilities of our Elastic Stack integration.

➡️ https://console.clever-cloud.com