Previously called the ELK Stack (an abbreviation for Elasticsearch, Logstash, and Kibana), the addition of other tools to the product group (such as Beats and APM) invalidated the acronym and necessitated the name change. Curiously, there wasn’t a security plugin in the free tier distribution of Elasticsearch and Kibana until very recently—and even now, only the very basic security features are included. 

The Elasticsearch Security Paradigm

We all know that Elasticsearch is a distributed search engine built on top of Lucene. Yet while the authors of Elasticsearch aimed to make data exploration quick and easy, they didn’t consider security until very late in its development. Authentication and access control functionalities have been missing from Elasticsearch for a great part of its history. Yet eventually, because of the growing demand and varied use cases, security is a foremost business requirement wherever Elasticsearch or its stack of technologies are used.

After introducing security as a non-free, paid for plugin in 2015, it took about four years for Elastic to release the Elastic Stack 6.8 and 7.1, which for the first time included a number of basic security features as part of the default distribution. These features include TLS encryption and user authentication with locally configured username and password. But there are many enterprise-grade features—such as LDAP and Active Directory-based authentication, SSO/SAML, and encryption at rest—that are still only available for high-ticket paying subscribers of Elastic gold and platinum support tiers.

What Is ReadonlyREST?

ReadonlyREST offers a lightweight Elasticsearch plugin that adds encryption, authentication, authorization, and access control capabilities to Elasticsearch-embedded REST APIs. This plugin is an alternative to the Elastic’s X-Pack security plugin and provides high-performance access control for Elasticsearch-native REST APIs.  

The ReadonlyREST Free for Elasticsearch security plugin is released under the GPLv3 Free Software license, and it’s been available since 2013 (two years before the official Elastic’s security plugin).

The ReadonlyREST family of security plugins extends also to Kibana: the free 1 month trial packages of ReadonlyREST PRO and Enterprise Kibana plugins are free to be downloaded on the website.
These Kibana plugins are commercial products that require the free Elasticsearch plugin to be already installed.

The last addition to ReadonlyREST products is ReadonlyREST Embedded (a commercial dual-license for the Elasticsearch plugin that involves a yearly licensing fee). It’s substancially the same code of ReadonlyREST Free for Elasticsearch, but the nature of the commercial license makes it possible to use ReadonlyREST technology in commercial products (whitelabeling). A detailed documentation of this plugin can be found here.

Design and Architecture

The core of the ReadonlyREST Free plugin for Elasticsearch is an ACL engine that checks each incoming request through a sequence of rules, similar to a firewall. There are multiple rules that can be grouped in sequences of blocks to form a powerful representation of a logic chain.

The following diagram models an Elasticsearch instance with the ReadonlyREST Free plugin installed, configured with SSL encryption and an ACL with at least one “allow” type ACL block.

Elasticsearch request flow with ReadonlyREST Free plugin
Elasticsearch request flow with ReadonlyREST Free plugin
  1. The user agent (i.e., cURL or Kibana) sends a search request to Elasticsearch using the port 9200 and the HTTPS URL schema.
  2. The plugin’s HTTPS filter unwraps the SSL layer and hands over the request to the Elasticsearch HTTP stack.
  3. The HTTP stack in Elasticsearch parses the HTTP request.
  4. The HTTP handler in Elasticsearch extracts the indices, action, and request type, and creates a search request (internal Elasticsearch format).
  5. The search request goes through the ACL; external systems such as LDAP can be asynchronously queried; and an exit result is eventually produced.
  6. The exit result is used by the audit log serializer to write an index record and/or Elasticsearch log file.
  7. If an ACL block wasn’t matched, or if a type: forbid block was matched, ReadonlyREST does not forward the search request to the search engine. Rather, it creates an “unauthorized” HTTP response.
  8. If the ACL did match with the type: allow block, the request is forwarded to the search engine.
  9. The Elasticsearch code creates a search response containing the results of the query.
  10. The Elasticsearch code converts the search response to an HTTP response.
  11. The HTTP response flows back to ReadonlyREST’s HTTPS filter and to the user agent.

Key Features

In addition to Elasticsearch’s capabilities, ReadonlyREST provides the following:

  • Encryption: Transforms the Elasticsearch API from HTTP to HTTPS.
  • Authentication: Requires credentials.
  • Authorization: Declares groups of users, permissions, and partial access to indices.
  • Access control: An ACL written in YAML can be used to model complex logic.
  • Audit logs: A trace of the access requests can be logged to file or index (or both).

Advantages

When it comes to securing Elasticsearch, there are not many choices. Apart from the ReadonlyREST Free plugin for Elasticsearch, users have two other options:

  • Elastic’s  “xpack.security” module in the non-free Elastic Stack (extremely expensive, and complex)
  • Reverse proxy using NGINX with basic HTTP authentication

Note that the Nginx reverse proxy is no longer recommended because:

    • Your cluster is still insecure on the server it’s running on, and anyone who has direct access to the node can use the Elasticsearch port to perform all the operations. 
    • Misconfiguring a node’s firewall can still leave it exposed and lead to vulnerabilities.
    • Elasticsearch has a vast number of non-standard REST API endpoints, and getting the right configuration for access control is almost impossible.

In contrast to these solutions, ReadonlyREST offers several advantages, including: 

  • Freely available: All the Elasticsearch security features—such as encryption, authentication, authorization, and access control—are free.
  • Easy setup: The ReadonlyREST Free plugin is by far the easiest to install and configure, and there’s great support available if you run into difficulties.
  • Easy integration into an existing stack: It is very easy to integrate the ReadonlyREST Free plugin into an existing stack because you can get the basic security features even without installing the plugin in all nodes, just add an extra small data-less, non master-eligible node and you’re up and running without messing with the production cluster. In addition, it’s simple to integrate with LDAP, or JSON microservices for authentication and authorization.
  • Robust features: The ReadonlyREST Free plugin’s security features are more robust than the free features provided by Elasticsearch itself.
  • Less expensive: The ReadonlyREST Enterprise plugin (which is needed for Kibana) is much less expensive than the commercial Elasticsearch module. In addition, ReadonlyREST’s fixed pricing model is not correlated to the amount of nodes or clusters, so as you grow, it becomes more and more price-efficient.
  • Customization options: ReadonlyREST plugins for Kibana (PRO and Enterprise) can  seamlessly inject custom CSS and Javascript into the look and feel of the whole Kibana UI. This is very important because the other options involve patching and forking the Kibana source code.

Integrating ReadonlyREST into ELK Clusters 

Points to Consider Before Starting

The most important thing to consider before integrating ReadonlyREST into an ELK cluster is the version compatibility between Elasticsearch and the ReadonlyREST plugin you are using. So, before downloading a plugin, choose the correct version of the ELK stack you are running. If you have a cluster with more than one node of Elasticsearch, there are two options for integrating the plugin:

  1. Install the plugin on a dedicated client node that exposes the HTTP interface over port 9200. You don’t need to install the plugin on all of the nodes in the cluster, and for security hardening, you can even disable the HTTP port on all other nodes. The only drawback of this approach is that document-level security and field-level security are not going to work.
  2. Install the plugin on all client and data nodes in the cluster. This approach is necessary when you need document-level and filter-level security. Document-level security helps you ensure that only the documents that satisfy the query provided as an argument are returned. On the other hand, field-level security ensures that only certain fields from queries are returned.

Prerequisites

Download the ReadonlyREST Free plugin for your Elasticsearch version, click here. After selecting the product type and Elastic Stack version, simply enter your email address to receive the download instructions. The same steps are required for the PRO and Enterprise Kibana plugins.

Important: If you are using ELK Stack version 6.3.x or greater, you need to disable Elasticsearch’s default X-Pack security module. This is because it can’t run together with ReadonlyREST. To do this, just add the following parameter inside both elasticsearch.yml and kibana.yml:

xpack.security.enabled: false

Installing and Configuring the ReadonlyREST Free Plugin for Elasticsearch

In this section, we will show you how to configure basic HTTP authentication in Elasticsearch using the ReadonlyREST Free plugin. Once the settings are applied, Elasticsearch will return a “401 Unauthorized ” error message in the case of wrong or missing credentials.

1. Navigate to your Elasticsearch home directory and install the plugin using the following command:

bin/elasticsearch-plugin install "https://api.beshu.tech/download/es?versions=7.6.0"

2. Accept the warning message and continue installation. 

During installation, the plugin will show a warning message and ask for additional permissions. Continue the installation by pressing the “y” key, as shown in the following screenshot.

3. Disable the X-Pack security plugin by adding the following parameter inside the elasticsearch.yml file:

xpack.security.enabled: false

4. Create and edit a file inside the Elasticsearch config directory, right next to elasticsearch.yml:

vi readonlyrest.yml

5. Now write some basic settings, as shown below:

readonlyrest:
    access_control_rules:
    - name: "Require HTTP Basic Auth"
      type: allow
      auth_key: elasticuser:somerandomepassword

6. Start Elasticsearch.

bin/elasticsearch

7. Test if the authentication was successful by executing a GET request.

curl -vvv -u elasticuser:somerandomepassword "http://localhost:9200/_cat/indices"

If the authentication was successful, the request will return a list of indices. If the authentication was not successful, it will return a “401 forbidden error” message.

Understanding ReadonlyREST’s Authentication Settings

All of the authentication parameters and rules are written in the readonlyrest.yml file. In the previous steps, we created a basic authentication using a username and password in plain text format. If you want to create multiple users in a similar way, simply add the similar settings to the file. It will look like this: 

readonlyrest:
    access_control_rules:

    - name: "Require HTTP Basic Auth"
      type: allow
      auth_key: elasticuser:somerandomepassword

    - name: "Require HTTP Basic Auth second block"
      type: allow
      auth_key: seconduser:password123

In this case, make sure that the name parameter is unique for each block.

Adding SSL Encryption

The ReadonlyREST Free plugin requires your certificates in Java KeyStore (.jks) format. You can also use Let’s Encrypt certificates by converting them into Java KeyStore. For testing purposes, you can generate a self-signed certificate using the following command:

keytool -genkey -keyalg RSA -alias selfsigned -keystore keystore.jks -storepass readonlyrest -validity 360 -keysize 2048

Here is how to enable encryption:

1. Open the elasticsearch.yml file and add the following:

http.type: ssl_netty4

2. Copy your keystroke.jks file inside the Elasticsearch config directory, parallel to the elasticsearch.yml file.

3. Add the SSL settings to the readonlyrest.yml file. The complete file should now look like this: 

readonlyrest:

    ssl:
      keystore_file: "keystore.jks"
      keystore_pass: qwerty123
      key_pass: qwerty123

    access_control_rules:

    - name: "Require HTTP Basic Auth"
      type: allow
      auth_key: elasticuser:somerandomepassword

4. Restart Elasticsearch.

5. Test the encryption using a cURL command:

 curl -vvv -u user:password "https://localhost:9200/_cat/indices"

Important: You may get an error that says “curl: (51) SSL: no alternative certificate subject name matches target host name ‘localhost.’”

This means that the hostname of the certificate file does not match with localhost. To remedy this, try using the correct domain name server on which you are running Elasticsearch. Otherwise, you need to fix the certificate by adding the correct hostname to it. For testing purposes, you can turn off cURL’s certificate verification by using the -k (or –insecure) option.

For example:  

curl -k -vvv -u user:password "https://localhost:9200/_cat/indices"

Note that as the error message said, access to Elasticsearch is insecure. Therefore,  you shouldn’t use this option in production, since it allows self-signed certificates.

Sending Data from Logstash to ReadonlyREST-Enabled Elasticsearch

We assume that you already have SSL and basic authentication available inside Elasticsearch (as explained in the previous section), and that you are now ready to ship data from Logstash to Elasticsearch.

To do this, edit the Logstash configuration file and fix the output block as follows:

output {
elasticsearch {
ssl => true
ssl_certificate_verification => false
hosts => ["YOUR_ELASTICSEARCH_HOST:9200"]
user => elasticuser
password => somerandomepassword
}
}

Sending Data from Metricbeat to ReadonlyREST-Enabled Elasticsearch

Similar to the Logstash process, here you just need to enable the HTTPS protocol and add the Elasticsearch username and password inside Metricbeat’s output block:

output.elasticsearch:
username: elasticuser
password: somerandomepassword
protocol: https
hosts: ["YOUR_ELASTICSEARCH_HOST:9200"]
index: "log_metricbeat-%{+yyyy.MM}"

Accessing Password-Protected Elasticsearch Using Kibana

If Elasticsearch is protected with basic authentication, you need to add a username and password in the kibana.yml file using the following parameters:

elasticsearch.username: "elasticuser"
elasticsearch.password: "somerandomepassword"

The Kibana server requires these settings in order to perform maintenance on the Kibana index at startup. Your Kibana users still need to authenticate with Elasticsearch while accessing the Kibana UI in the browser, which is proxied through the Kibana server.

Summary

In this post, we discussed what Elasticsearch offers in terms of security. We also explored the many features of ReadonlyREST’s plugins, with a special focus on how to set up the ReadonlyREST Free plugin for Elasticsearch.

In the  next one, we will explore how to write sophisticated ACL rules for securing indices and Elasticsearch APIs, along with document-level and field-level security. We will also discuss how to avoid using a plain text username and password by utilizing industry standard encrypted authentication, such as SHA-256. Additionally, we’ll look into advanced authentication configurations such as LDAP and SAML.

 

Related content

HOW CERN SAVES MONEY WITH READONLYREST

This year, CERN (The European Organization for Nuclear Research) optimized the usage of computing resources by consolidating 30+ Elasticsearch clusters into a handful of multi-user clusters.

Watch the presentation CERN organized to understand the guiding principles behind ReadonlyREST.