Elasticsearch : Empty reply from server

 

Like many, I had the "Empty reply from server" error !

curl -X GET "http://XXX.XX.XX.XX:9200"
curl: (52) Empty reply from server

There were many answers on the web ( see https://stackoverflow.com/questions/35921195/curl-52-empty-reply-from-server-timeout-when-querying-elastiscsearch).

In my case, it was quite simple. It was missing the credentials :

curl -u [user(elastic)]:[PASSWORD] -k -X GET "https://XXX.XX.XX.XX:9200"

Hope it helps !

Elasticsearch : cerebro, une solution web pour gérer votre cluster facilement

Dans le cadre du travail, nous étudions actuellement la solution ElasticSearch et tout son écosystème.

Je suis tombé sur une vidéo intéressante qui décrit l'outil Cerebro, un outil pour gérer les clusters Elasticsearch.

Reminder : Not enough space due to Elasticsearch & GC log files

I had a partition full in /var/

It was a problem with my Elasticsearch configuration. Elasticsearch produces gc.log files :

15M     /var/log/elasticsearch/gc.log
488K    /var/log/elasticsearch/gc.log.00
65M     /var/log/elasticsearch/gc.log.01
65M     /var/log/elasticsearch/gc.log.02
...

I found the solution in the Elasticsearch documentation 

By default, GC logs are enabled in Elasticsearch. The settings are configured in jvm.options and the logs are written in the same location as other Elasticsearch logs. The default configuration rotates the logs every 64 MB and can consumer up to 2 GB of disk space

You can change this behaviour by editing /etc/elasticsearch/jvm.options :

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=5  ==> here
8:-XX:GCLogFileSize=64m
# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=5,filesize=64m  <== here

Note :

If you update Elasticsearch you may encountered a jar Hell exception. The update brings a new version of log4j.So, you have two version of log4j and Elasticsearch refuse to start.

[2022-01-10T09:15:31,653][INFO ][o.e.n.Node               ] [k9CXYZR] stopped

[2022-01-10T09:15:31,658][INFO ][o.e.n.Node               ] [k9CXYZR] closing ...

[2022-01-10T09:15:32,229][INFO ][o.e.n.Node               ] [k9CXYZR] closed

[2022-01-10T09:16:19,481][ERROR][o.e.b.Bootstrap          ] [unknown] Exception

java.lang.IllegalStateException: jar hell!

class: META-INF.versions.9.org.apache.logging.log4j.core.util.SystemClock

jar1: /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar

jar2: /usr/share/elasticsearch/lib/log4j-core-2.17.0.jar

        at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:282) ~[elasticsearch-core-6.8.22.jar:6.8.22]

        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:195) ~[elasticsearch-core-6.8.22.jar:6.8.22]

        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86) ~[elasticsearch-core-6.8.22.jar:6.8.22]

To solve it, just remove a version of log4j :

rm /usr/share/elasticsearch/lib/log4j*2.16*

TODO : check version of log4j

Faille log4j & elasticsearch 6.8.*

Pour corriger la faille de sécurité dans Elasticsearch, nous avons mis à jour le paquage Debian et ainsi passer de la version 6.8.20 à 6.8.21. Nous nous sommes rendus comptes qu'il restait un jar impacté par la faille : /usr/share/elasticsearch/lib/log4j-core-2.11.1.jar 

A priori, d'après les discussions, il n'y a pas de risques (https://discuss.elastic.co/t/zero-day-exploit-in-log4j2-which-is-part-of-elasticsearch/291439/63)

Toutefois afin de prendre toues les précautions, nous avons remplacé les versions impactés par la bonne version de log4j. Le seul effet de bord possible serait un problème de permission : “Probably permissions issue. That's why Elasticsearch is not starting. If you have removed class from the jar, check the permissions of the jar and make sure it is elasticsearch:elasticsearch or whatever it was before.”

Voici les commandes utilisées :

cd /tmp/
wget https://dlcdn.apache.org/logging/log4j/2.16.0/apache-log4j-2.16.0-bin.tar.gz
tar xvzf apache-log4j-2.16.0-bin.tar.gz
mkdir /tmp/log4j/old
cd /usr/share/elasticsearch/lib/
mv log4j-1.2-api-2.11.1.jar /tmp/log4j/old/
mv log4j-api-2.11.1.jar /tmp/log4j/old/
mv log4j-core-2.11.1.jar /tmp/log4j/old/
cp /tmp/apache-log4j-2.16.0-bin/log4j-1.2-api-2.16.0.jar .
cp /tmp/apache-log4j-2.16.0-bin/log4j-api-2.16.0.jar .
cp /tmp/apache-log4j-2.16.0-bin/log4j-core-2.16.0.jar .
systemctl stop elasticsearch
systemctl stop elasticsearch
tail -f /var/log/elasticsearch/elasticsearch.log
systemctl restart apache2