Reminder : Not enough space due to Elasticsearch & GC log files

I had a partition full in /var/

It was a problem with my Elasticsearch configuration. Elasticsearch produces gc.log files :

15M     /var/log/elasticsearch/gc.log
488K    /var/log/elasticsearch/gc.log.00
65M     /var/log/elasticsearch/gc.log.01
65M     /var/log/elasticsearch/gc.log.02
...

I found the solution in the Elasticsearch documentation 

By default, GC logs are enabled in Elasticsearch. The settings are configured in jvm.options and the logs are written in the same location as other Elasticsearch logs. The default configuration rotates the logs every 64 MB and can consumer up to 2 GB of disk space

You can change this behaviour by editing /etc/elasticsearch/jvm.options :

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=5  ==> here
8:-XX:GCLogFileSize=64m
# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=5,filesize=64m  <== here

Note :

If you update Elasticsearch you may encountered a jar Hell exception. The update brings a new version of log4j.So, you have two version of log4j and Elasticsearch refuse to start.

[2022-01-10T09:15:31,653][INFO ][o.e.n.Node               ] [k9CXYZR] stopped

[2022-01-10T09:15:31,658][INFO ][o.e.n.Node               ] [k9CXYZR] closing ...

[2022-01-10T09:15:32,229][INFO ][o.e.n.Node               ] [k9CXYZR] closed

[2022-01-10T09:16:19,481][ERROR][o.e.b.Bootstrap          ] [unknown] Exception

java.lang.IllegalStateException: jar hell!

class: META-INF.versions.9.org.apache.logging.log4j.core.util.SystemClock

jar1: /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar

jar2: /usr/share/elasticsearch/lib/log4j-core-2.17.0.jar

        at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:282) ~[elasticsearch-core-6.8.22.jar:6.8.22]

        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:195) ~[elasticsearch-core-6.8.22.jar:6.8.22]

        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86) ~[elasticsearch-core-6.8.22.jar:6.8.22]

To solve it, just remove a version of log4j :

rm /usr/share/elasticsearch/lib/log4j*2.16*

TODO : check version of log4j

apt-get install php7.3-curl failed

 

I had a problem when installing the php7.3-curl dependency :

The package dependies was not satisfied :
 php7.3-curl : Depend: libcurl3 (>= 7.44.0) but not installable

In fact, the problem occurs because I forgot to change stretch to buster in my configuration :

 vi /etc/apt/sources.list.d/php7.3.list
deb https://packages.sury.org/php/ buster main

Then I restarted :

apt-get remove --auto-remove php7.3-curl
apt-get purge --auto-remove php7.3-curl
apt-get install php7.3-curl

Reminder : install Docker

This a reminder to install Docker (20.10.8) on Debian 11 (Bullseye).

Get the key

 curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Add Docker in source list

echo \

  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \

  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker

apt-get update
apt-get install docker-ce docker-ce-cli containerd.io

Final steps

It's better to add a Docker user :

useradd -m -s /bin/bash -G docker oscardocker
su - oscardocker

Test

You can execute the following command  : docker-compose up

Docker version 20.10.8, build 3967b7d

Reminder : upgrade Debian 9 (Stretch) to Debian 11 (Bulleyes)

This is an update to my previous post. I had old system still in Debian 9 version.To upgrade this system, I made two steps : first, I upgraded Debian 9 to Debian 10 then I upgraded Debian 10 to 11.In my mind it's more secure.

Note : to upgrade Debian 9 to Debian 10, I used the following command :

apt-get update && apt-get upgrade && apt-get dist-upgrade
sed -i 's/stretch/buster/g' /etc/apt/sources.list
apt-get update
apt-get upgrade
apt-get dist-upgrade
reboot

WARNING : If you have a POSTGRESQL database, you need to launch the reinxdb command :

su - postgres
reindexdb --all

See https://www.debian.org/releases/buster/amd64/release-notes/ch-information.en.html#postgresql-reindex
See https://wiki.postgresql.org/wiki/Locale_data_changes

The command to upgrade Debian 10 to Debian 11 are quite the same ( See my previous post )

Reminder Upgrade Debian 10 (Buster) to Debian 11 (Bullseye)

A simple reminder to upgrade  Debian 10 (Buster)  to Debian 11 (Bullseye) 

• apt update & apt upgrade
• Change source list :

vi /etc/apt/sources.list

deb http://deb.debian.org/debian bullseye main contrib non-free
deb-src http://deb.debian.org/debian bullseye main contrib non-free
deb http://deb.debian.org/debian-security/ bullseye-security main contrib non-free
deb-src http://deb.debian.org/debian-security/ bullseye-security main contrib non-free
deb http://deb.debian.org/debian bullseye-updates main contrib non-free
deb-src http://deb.debian.org/debian bullseye-updates main contrib non-free

Note : comment other source like Docker.

• apt update
• apt upgrade --without-new-pkgs
• apt full-upgrade
• cat /etc/debian_version

@todo : postgresql migration

Debian 10 : enable unattended upgrade

 This article is a simple reminder to enable unattended upgrade feature quickly under Debian 10.

Configuration

apt install unattended-upgrades
vi /etc/apt/apt.conf.d/50unattended-upgrades

You have to activate update automatically. In that case, we made package update 
and security update :

"origin=Debian,codename=${distro_codename},label=Debian";
"origin=Debian,codename=${distro_codename},label=Debian-Security";
Unattended-Upgrade::Mail mail@toto.fr;
dpkg-reconfigure --priority=low unattended-upgrades

systemctl status unattended-upgrades.service

Note : A mail is sent every week if update occurs.

More details : https://linuxhint.com/managing_debian_unattended_upgrades/

Frequency

cat /etc/apt/apt.conf.d/20auto-upgrades

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

You can check which packages are updated :

tail /var/log/dpkg.log

Set frequency

I didn't configure this feature but you could find more information :

https://unix.stackexchange.com/questions/342663/how-is-unattended-upgrades-started-and-how-can-i-modify-its-schedule

How to add CAS authentication to a PHP website ?

The main purpose of this article is to explain what you have to do in order to activate the CAS authentication. 

We will cover all steps needed to begin quickly with CAS !

My configuration :

• Debian 9 Stretch
• PHP 7.0
• jasig/phpcas 1.4

Installation

You need the following Debian package :

apt-get install wget php-cli php-zip unzip

apt-get install php7.0-curl

Then, you have to install composer :

cd /root

wget -O composer-setup.php https://getcomposer.org/installer

php composer-setup.php --install-dir=/usr/local/bin --filename=composer

The next step is to add  the file composer.json with the referencies of phpcas:

{

    "require": {

        "jasig/phpcas": "^1.4"

    }

}

The composer update command retreives the phpcas library in your project (in vendor directory).So you need to execute the following :

cd /var/www/html/myproject

composer update

If you use git, don't fortget to add these lines in your .gitignore file :

vendor

composer.lock

How to use it in your PHP file ?

Here is a sample :

require __DIR__ . '/vendor/autoload.php';


// Enable debugging
phpCAS::setLogger();
// Enable verbose error messages. Disable in production!
phpCAS::setVerbose(true);
//phpCAS::setDebug("./err.log");

// initialize phpCAS
phpCAS::client(CAS_VERSION_2_0,'monserveur_cas.toto.fr',443,'/cas',FALSE);

// force CAS authentication
//phpCAS::setCasServerCACert($cas_server_ca_cert_path);
phpCAS::setNoCasServerValidation();
phpCAS::forceAuthentication();

// at this step, the user has been authenticated by the CAS server
// and the user's login name can be read with phpCAS::getUser().

if (!phpCAS::checkAuthentication()) {
    die("Echec !");
}
$login = phpCAS::getUser();
print_r(phpCAS::getAttributes());
echo "===> LOGIN : $login";

PostgresSQL administration

In my line of work and also in my personal activity (See OpenLearning Project Page), I need to administrate a Postgresql Database.I'm going to share what I need to do.In this paper, I don't explain each parameters : I only give my configuration for my purpose.If you want details about configuration, see the official website od PostgreSQL.
I encourage you to let me know you advice and what you think of my paper.Please do not hesitate to report.It's important for me and other users.

My server configuration :

• Debian 8.0 x64 (Jessie)
• PostgreSQL 9.4.13

Backup the database

Here is my backup script :

       
#!/bin/sh
###################################################################
# Save [DATABASE_NAME] database
#
# @author  : ...
# @version : 1.0
#
###################################################################


BACKUP_DIR=/opt/[DATABASE_NAME]/postgresql-backup
MAILTO="[MAIL]"

current_time=$(date "+%Y.%m.%d-%H.%M.%S")
backup_file="$BACKUP_DIR/[DATABASE_NAME]-dump_${current_time}.sql"

# Log script output
log()
{
    echo $1
}


##########################################################################################
# MAIN
##########################################################################################


log ""
log ""
log "#####################################################################"
log "START $0 script at $current_time"
log "#####################################################################"

log ""
log "Saving Postgresql database in ${backup_file}"

# Remarque .pgpass present dans /root/
`pg_dump --format=custom --no-owner -U postgres -v -b -f $backup_file -h 127.0.0.1 [DATABASE_NAME]`
if [ -s $backup_file ] ; then
  echo 'Dump succeeded'
  echo 'Dump succeeded'  | mail -s "[DATABASE_NAME] save database SUCCESS" $MAILTO
else
  echo 'Dump failed'
  echo 'Dump failed'  | mail -s "[DATABASE_NAME] save database FAILED" $MAILTO
fi

current_time=$(date "+%Y.%m.%d-%H.%M.%S")
log "#####################################################################"
log "END synchro script at $current_time"
log "#####################################################################"
exit 0
       

Note :
- We use postgres user to avoid permission problem.
- You have to change [DATABASE_NAME] and [MAIL] in order to use this script.
- This script ask you a password.To avoid this, see next section.

Automatic connection

Automatic connection is necessary by the backup script for crontab purpose.First of all, we need to set a password for postgres user :

       
su - postgres
psql
ALTER USER postgres WITH PASSWORD 'MY_PASSWORD';
       
 

To be sure that is work, you have to test it :

       
psql -U postgres -h 127.0.0.1 -W
       

It should ask a password and you will be able to connect at Postgresql.
Then, we create and fill a .pgpass file like the following :

       
vi /root/.pgpass
127.0.0.1:5432:[DATABASE_NAME]:postgres:[MY_PASSWORD]
       

After, make a chmod :

       
chmod 600 /root/.pgpass
       

Now, the following command should work :

       
psql -U postgres -h 127.0.0.1 -W
       

That's all ! Now a script using pg_backup should work.
To finish, we execute the backup script in crontab every night at 22:00 :

       
crontab -e
00 22 * * * /root/app/scripts/crontab/save_db.sh >> /var/app/archive/postgresql/save_db.log
       

Restore the database

Advice : the most important is to test restore of the database !

The following script ask you a password.Usually, it should not because of .pg_pass file ! It is not really important because restore isn't use in crontab script.

       
#!/bin/sh

###################################################################
# Restore [DATABASE_NAME] database
#
###################################################################
clear
current_time=$(date "+%Y.%m.%d-%H.%M.%S")
echo ""
echo "#####################################################################"
echo "START $0 script to RESTORE [DATABASE_NAME] DATABASE  at $current_time"
echo "#####################################################################"
echo ""


filepath=""
echo "Please enter PGSQL's backup file path ( e.g : /opt/[DATABASE_NAME]/postgresql-backup/[DATABASE_NAME]-dump_2017.08.24-10.28.35.sql) : "
read filepath
if [ "$filepath" = "" ];then
        echo "You must write a backup file path !"
        echo "END"
        exit 1
fi

if [ -e "$filepath" ];then
        echo "Restoring Postgresql database from ${filepath}..."
        pg_restore --if-exists -v -c -h 127.0.0.1 -U postgres -d [DATABASE_NAME] ${filepath}
else
        echo "File not found : $filepath"
fi

current_time=$(date "+%Y.%m.%d-%H.%M.%S")

echo "#####################################################################"
echo "END synchro script at $current_time"

echo "#####################################################################"
exit 0     
 

Note : You need to create an empty database before restoring.

Monitoring your database

To improve our application, we need to identify heavy request which have a duration upper than 50 ms.That's why we activate logging :

 vi /etc/postgresql/9.4/main/postgresql.conf
logging_collector = on
log_rotation_age = 1d
log_rotation_size = 100MB

Then we modify following parameters :

       
log_min_duration_statement = 50
log_statement = 'mod'

After, you only need restart the database :

service postgresql restart
ls /var/lib/postgresql/9.4/main/pg_log/

You will log file and you can test logging with simple request : select pg_sleep(0.5);


Source :
https://www.depesz.com/2011/05/06/understanding-postgresql-conf-log/