Configure Kafka TLS Encryption

By default, Redpanda data is sent unencrypted. A security best practice is to enable encryption with TLS or mTLS.

  • Transport Layer Security (TLS), previously SSL, provides encryption for client-server communication. A server certificate prevents third parties from accessing data transferred between the client and server. By default, Redpanda clusters accept connections from clients using TLS version 1.2 or later, but this value is configurable.

  • mTLS, or 2-way TLS, is a protocol that authenticates both the server and the client. In addition to the server certificate required in TLS, mTLS also requires the client to give a certificate. This involves more overhead to implement, but it can be useful for environments that require additional security and only have a small number of verified clients.

Prerequisites

TLS certificates are required for encryption. You can use your own certificates, either self-signed or issued by a trusted CA.

You’ll need the following files:

  • A private key file (broker.key) for each broker.

  • A certificate file (broker.crt) for each broker.

  • A truststore file (ca.crt). All brokers can have the same ca.crt file.

Ensure these files are readable by Redpanda and protected against unauthorized access:

chmod 400 broker.key broker.crt ca.crt
chown redpanda:redpanda broker.key broker.crt ca.crt
bash

For mTLS, client certificates signed by the same CA are also required.

If you don’t already have these files, you can learn how to generate them in Create a local CA for self-signed certificates.

If you enable TLS encryption, you can also specify a certificate revocation list (ca.crl) so that Redpanda can check and reject connections from entities using certificates already revoked by a certificate authority (CA). All brokers can have the same ca.crl. The file must contain a single, concatenated list of certificate revocation lists (CRLs) for all issuing certificates in the truststore file.

Generate certificate files

This section shows you how to generate self-signed certificate files for your Redpanda brokers. If you already have your own, you can skip this step.

Self-signed certificates are useful if you want to generate multiple certificates all signed by the same root. For example, you want to use mTLS but issue different certificates to multiple Redpanda brokers and clients.

Create a local CA for self-signed certificates

  1. Create a self-signed certificate and private key for the CA:

    openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -keyout broker.key -out broker.crt -subj "/CN=redpanda" -addext "subjectAltName = DNS:localhost, IP:127.0.0.1"
    bash

    Ensure that subjectAltName is accurate for your setup.

    This command generates:

    • broker.key: Private key

    • broker.crt: Self-signed certificate

  2. Create a configuration file for the CA, ca.cnf, and include your organization’s details. Ensure the subjectAltName extension is set correctly for broker certificates.

    [ ca ]
default_ca = CA_default

[ CA_default ]
default_days    = 365
database        = index.txt
serial          = serial.txt
default_md      = sha256
copy_extensions = copy
unique_subject  = no
policy          = signing_policy

[ req ]
prompt             = no
distinguished_name = distinguished_name
x509_extensions    = extensions

[ distinguished_name ]
organizationName = <organization-name>
commonName       = <common-name>

[ extensions ]
keyUsage         = critical,digitalSignature,nonRepudiation,keyEncipherment,keyCertSign
basicConstraints = critical,CA:true,pathlen:1

[ signing_policy ]
organizationName = supplied
commonName       = optional
    ini

  3. Create a private key for the CA and set its permissions:

    openssl genrsa -out ca.key 2048
chmod 400 ca.key
    bash

  4. Use the CA configuration to generate a public certificate:

    openssl req -new -x509 -config ca.cnf -key ca.key -days 365 -batch -out ca.crt
    bash

    where:

    Inputs Description

    -new

    New request.

    -x509

    Create an X.509 certificate, instead of a certificate signing request (CSR).

    -config ca.cnf

    Configuration file to use when generating certificates (created above).

    -key ca.key

    Private key of the CA (created above).

    -days 365

    Number of days signed certificates are valid.

    -batch

    Batch mode, where certificates are certified automatically.

    The output ca.crt is the CA’s public certificate, which you’ll use in the truststore.

Create broker certificates and certificate signing requests (CSRs)

To issue certificates for brokers, create a certificate signing request (CSR) and sign it with the CA.

  1. Define the broker’s Subject Alternative Name (SAN) entries in broker.cnf under alt_names.

    A subject alternative name (SAN) indicates all domain names and IP addresses secured by the certificate. Depending on the address the client uses to connect to Redpanda, you might need to create a CNF file for each broker to modify the alt_names section with organizational details. For production usage, edit alt_names with DNS resolutions and/or the IP addresses.

    For example:

    broker.cnf
    [ req ]
prompt             = no
distinguished_name = distinguished_name
req_extensions     = extensions

[ distinguished_name ]
organizationName = <organization-name>

[ extensions ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = localhost
DNS.2 = redpanda
DNS.3 = console
DNS.4 = connect
DNS.5 = ec2-3-15-15-272.us-east-2.compute.amazonaws.com
IP.1  = 10.0.8.1
    ini

    You could configure alternative names with a single version of broker.key/broker.crt, as long as you update the certificate for all brokers in the cluster any time you edit an entry. For example:

    [ alt_names ]
DNS.1 = broker1.example.com
DNS.2 = broker2.example.com
DNS.3 = broker3.example.com
    ini

    Additionally, you can configure alternative names using the public or private IP addresses of all your brokers. For example:

    [ alt_names ]
IP.1 = 10.0.8.1
IP.2 = 10.0.8.2
IP.3 = 10.0.8.3
    ini

  2. Create a 2048-bit private key for the broker:

    openssl genrsa -out broker.key 2048
    bash

  3. Create the broker’s certificate signing request (CSR):

    openssl req -new -key broker.key -out broker.csr -nodes -config broker.cnf
    bash

    where:

    Inputs Description

    -req

    Input is a certificate request. Sign and output.

    -signkey ca.key

    Private key of the CA (created above).

    -days 365

    Number of days signed certificates are valid.

    -extfile broker.cnf

    Configuration file for CA.

    -extensions extensions

    Section in broker.cnf to use when applying extensions.

    -in broker.csr

    Broker certificate signing request (CSR generated above).

    The output broker.crt is the signed public key certificate for the broker.

  4. Sign the CSR with the CA’s private key:

    touch index.txt
echo '01' > serial.txt
openssl ca -config ca.cnf -keyfile ca.key -cert ca.crt -extensions extensions -in broker.csr -out broker.crt -outdir . -batch
    bash

    If generated by a corporate CA, these certificate signing requests must be signed with the following extensions:

    keyUsage         = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth
    ini

  5. Set ownership and permissions:

    chown redpanda:redpanda broker.key broker.crt ca.crt
chmod 400 broker.key broker.crt ca.crt
    bash

Configure TLS

To configure TLS, in redpanda.yaml, enter either the standard PEM configuration files or the PKCS#12 bundle configuration.

Choose PEM files when:

  • You are using FIPS mode compliance.

  • You prefer file-based configurations with a separate key, certificate, and truststore file.

Choose PKCS#12 bundles when:

  • FIPS mode is not required in your environment.

  • You want a single, password-protected file that contains all certificates and keys.

Configure TLS with PEM files

If you have separate files for key_file, cert_file, and truststore_file, use the following configuration in redpanda.yaml:

redpanda.yaml
redpanda:
  rpc_server_tls: {}
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: tls_listener
  kafka_api_tls:
    - name: tls_listener
      key_file: broker.key
      cert_file: broker.crt
      truststore_file: ca.crt
      crl_file: ca.crl # Optional
      enabled: true
      require_client_auth: false
  admin_api_tls: []
pandaproxy:
  pandaproxy_api_tls: []
schema_registry:
  schema_registry_api_tls: []
yaml

The following files must be readable by Redpanda, either through 444 permissions or chown to Redpanda with 400 permissions:

  • broker.crt

  • broker.key

  • ca.crt

  • ca.crl

Because the keys and certificates are only read at startup, you must restart Redpanda services after updating redpanda.yaml. TLS-related changes to redpanda.yaml will not be known to Redpanda until after this restart:

systemctl restart redpanda

If you replace a working ca.crl file with a file that contains an invalid certificate revocation list, such as an unsigned list, Redpanda will reject all connections until you either:

  • Replace the file.

  • Remove the crl_file: ca.crl line from redpanda.yaml and restart Redpanda.

To set the RPC port to encrypt replication, add:

redpanda.yaml
redpanda:
  rpc_server_tls:
    enabled: true
    require_client_auth: false
    key_file: broker.key
    cert_file: broker.crt
    truststore_file: ca.crt
    crl_file: ca.crl # Optional
yaml

Schema Registry and HTTP Proxy connect to Redpanda over the Kafka API. If you configure a TLS listener for the Kafka API, you must add schema_registry_client::broker_tls and pandaproxy_client::broker_tls. All APIs, except the internal RPC port, support multiple listeners. See:

Configure TLS with PKCS#12 bundles

You can simplify certificate management by generating a password-protected PKCS#12 bundle from your broker.key and broker.crt files.

PKCS#12 keys are not supported when FIPS mode is enabled in Redpanda. The PKCS12KDF algorithm used in PKCS#12 is not FIPS-compliant. To use Redpanda in FIPS mode, configure your certificates and keys in PEM format instead.

  1. Run this command to create a PKCS#12 file from your broker.key and broker.crt files:

    openssl pkcs12 -export -inkey broker.key -certfile broker.crt -passout pass:<password> -out broker.p12
    bash

    Replace <password> with your own password.

  2. Update redpanda.yaml with the path to the PKCS#12 bundle:

    redpanda:
  kafka_api_tls:
    - name: tls_listener
      p12_path: <path-to-bundle>
      p12_password: <password>
      enabled: true
      require_client_auth: false
    yaml
Do not configure both both key_file/cert_file and p12_path/p12_password together, as this will cause a startup error.

Configure mTLS

To enable mTLS, add require_client_auth set to true. Configure either the standard PEM files or the PKCS#12 bundle.

Configure mTLS with PEM files

For the Kafka API, in redpanda.yaml, enter:

redpanda.yaml
redpanda:
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: mtls_listener
  kafka_api_tls:
    - name: mtls_listener
      key_file: mtls_broker.key
      cert_file: mtls_broker.crt
      truststore_file: mtls_ca.crt
      enabled: true
      require_client_auth: true
yaml

See also: Configure Listeners

Configure mTLS for a Kafka API listener

To enable mTLS for a Kafka API listener, edit redpanda.yaml:

redpanda.yaml
redpanda:
  kafka_api:
  - name: internal
    address: 0.0.0.0
    port: 9092

  advertised_kafka_api:
    - name: internal
      address: <port-clients-connect-to>
    port: 9092

  kafka_api_tls:
  - name: internal
    enabled: true
    require_client_auth: true
    cert_file: <path-to-pem-cert-file>
    key_file: <path-to-pem-key-file>
    truststore_file: <path-to-pem-ca-file>
yaml

  • Remember to replace placeholders in brackets.

  • kafka_api is the listener declaration. This name can have any value.

  • advertised_kafka_api is the advertised listener. This name should match the name of a declared listener. This address is the host name clients use to connect to the broker.

  • kafka_api_tls is the listener’s TLS configuration. This name must match the corresponding listener’s name.

See also: Configure Listeners

Configure mTLS with PKCS#12 bundles

  1. Update redpanda.yaml with the path to the PKCS#12 bundle:

    redpanda:
  kafka_api_tls:
    - name: mtls_listener
      p12_path: <path-to-bundle>
      p12_password: <password>
      enabled: true
      require_client_auth: true
    yaml
Do not configure both the key_file/cert_file and the p12_path/p12_password together, as this will cause a startup error.

Manage the minimum TLS version

Redpanda sets the minimum TLS version for all clusters to 1.2, using the tls_min_version cluster configuration property. This prevents client applications from negotiating a downgrade to the TLS version when they make a connection to a Redpanda cluster.

You can update the minimum TLS version of your clusters to v1.0, v1.1 or v1.3 using rpk. Replace the placeholder in brackets.

rpk cluster config set tls_min_version <version-number>
bash

You must restart Redpanda for the new setting to take effect:

systemctl restart redpanda
bash

Use rpk with TLS

If you’re using rpk to interact with the Kafka API using mTLS identity (for example, to manage topics or messages), pass the --tls-key, --tls-cert, and --tls-truststore flags to authenticate.

To create a new topic called test-topic, run:

 rpk topic create test-topic \
--tls-key <path-to-PEM-formatted-key-file> \
--tls-cert <path-to-PEM-formatted-cert-file> \
--tls-truststore <path-to-PEM-formatted-CA-file>
bash

Replace placeholders in brackets.

To check the configuration of the topic, run:

rpk topic describe test-topic <tls-flags-from-above>
bash

To interact with the Admin API (for example, to manage users), pass the --admin-api-tls-key, --admin-api-tls-cert, and --admin-api-tls-truststore flags.

By default, rpk connects to localhost:9092 for Kafka protocol commands. If you’re connecting to a remote broker or if you configured your local broker differently, use the -X brokers=<address:port> flag.

Monitor TLS certificates

Redpanda exposes several metrics to help administrators manage their installed certificates. When queried, these metrics list details for all resources that have an installed certificate. This may include APIs, storage, or other assets. These metrics also support labels so that you can more readily report statistics on single resources.

Configuring alerts on these metrics is a critical tool for managing certificate expiration and avoiding surprise outages. The public metrics reference contains a full list of available TLS metrics. You can refer to the monitor Redpanda guide for full details on configuring Prometheus to monitor these metrics. This guide also explains how to create a Grafana dashboard for visualizations and alerting.

Alternatively, you can choose to specify a certificate revocation list to reject connections from entities using certificates already revoked by a certificate authority.

Suggested reading

Suggested labs

Search all labs