Reminder : Not enough space due to Elasticsearch & GC log files

I had a partition full in /var/

It was a problem with my Elasticsearch configuration. Elasticsearch produces gc.log files :

15M     /var/log/elasticsearch/gc.log
488K    /var/log/elasticsearch/gc.log.00
65M     /var/log/elasticsearch/gc.log.01
65M     /var/log/elasticsearch/gc.log.02
...

I found the solution in the Elasticsearch documentation 

By default, GC logs are enabled in Elasticsearch. The settings are configured in jvm.options and the logs are written in the same location as other Elasticsearch logs. The default configuration rotates the logs every 64 MB and can consumer up to 2 GB of disk space

You can change this behaviour by editing /etc/elasticsearch/jvm.options :

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=5  ==> here
8:-XX:GCLogFileSize=64m
# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=5,filesize=64m  <== here

Note :

If you update Elasticsearch you may encountered a jar Hell exception. The update brings a new version of log4j.So, you have two version of log4j and Elasticsearch refuse to start.

[2022-01-10T09:15:31,653][INFO ][o.e.n.Node               ] [k9CXYZR] stopped

[2022-01-10T09:15:31,658][INFO ][o.e.n.Node               ] [k9CXYZR] closing ...

[2022-01-10T09:15:32,229][INFO ][o.e.n.Node               ] [k9CXYZR] closed

[2022-01-10T09:16:19,481][ERROR][o.e.b.Bootstrap          ] [unknown] Exception

java.lang.IllegalStateException: jar hell!

class: META-INF.versions.9.org.apache.logging.log4j.core.util.SystemClock

jar1: /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar

jar2: /usr/share/elasticsearch/lib/log4j-core-2.17.0.jar

        at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:282) ~[elasticsearch-core-6.8.22.jar:6.8.22]

        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:195) ~[elasticsearch-core-6.8.22.jar:6.8.22]

        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86) ~[elasticsearch-core-6.8.22.jar:6.8.22]

To solve it, just remove a version of log4j :

rm /usr/share/elasticsearch/lib/log4j*2.16*

TODO : check version of log4j