Upgrading Elastic Stack From 8.x to 9.x on Ubuntu 24.04

Prerequisites

Before upgrading to 9.x, you need to be running 8.19.x. See the official upgrade path for details.

Check your current version:

curl -s localhost:9200 | grep number

If you are not on 8.19.x, upgrade within the 8.x repo first before switching to 9.x.

Step 1: Fix GPG Key

The 9.x packages use an updated GPG key. Re-import it in the dearmored keyring format:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
  sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

Step 2: Switch Repo to 9.x

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] \
https://artifacts.elastic.co/packages/9.x/apt stable main" | \
sudo tee /etc/apt/sources.list.d/elastic.list

sudo apt update

Step 3: Upgrade Elasticsearch

sudo systemctl stop elasticsearch
sudo apt install --only-upgrade elasticsearch
sudo systemctl daemon-reload
sudo systemctl start elasticsearch
curl -s localhost:9200 | grep number

Step 4: Upgrade Kibana

sudo systemctl stop kibana
sudo apt install --only-upgrade kibana

Fix Missing Encryption Key

!!! danger Kibana 9.x requires encryption keys to be set. Without them Kibana will return 500 errors on startup.

Generate the keys:

sudo /usr/share/kibana/bin/kibana-encryption-keys generate

Add the three generated lines to the bottom of /etc/kibana/kibana.yml:

xpack.encryptedSavedObjects.encryptionKey: "your-generated-key-here"
xpack.reporting.encryptionKey: "your-generated-key-here"
xpack.security.encryptionKey: "your-generated-key-here"

Then start Kibana:

sudo systemctl daemon-reload
sudo systemctl start kibana

Step 5: Upgrade Logstash

sudo systemctl stop logstash
sudo apt install --only-upgrade logstash
sudo systemctl daemon-reload
sudo systemctl start logstash

Step 6: Upgrade Other Components (if installed)

Filebeat

sudo systemctl stop filebeat
sudo apt install --only-upgrade filebeat
sudo systemctl daemon-reload
sudo systemctl start filebeat

Unique id Required for Every filestream Input

From Filebeat 9.x onwards, every filestream input must have a unique id. If two or more filestream inputs share the same id (or any of them omits id entirely), Filebeat will refuse to start and log an error similar to:

filestream input with ID 'my-id' already exists, this will
lead to data duplication, please use a different ID

Review /etc/filebeat/filebeat.yml and every file under /etc/filebeat/inputs.d/ (or whichever filebeat.config.inputs.path points at) and make sure each - type: filestream block has its own distinct id:

filebeat.inputs:
  - type: filestream
    id: nginx-access
    paths:
      - /var/log/nginx/access.log

  - type: filestream
    id: nginx-error
    paths:
      - /var/log/nginx/error.log

After fixing the IDs, restart Filebeat and confirm it is running:

sudo systemctl restart filebeat
sudo systemctl status filebeat

Step 7: Verify

curl -s localhost:9200 | grep number
curl -s localhost:9200/_cluster/health?pretty

References