My VPS is sending SPAM!

[iamthwee]

Hi guys,

Got a bit of a dilemma, I just recently discovered my VPS was sending spam, this was brought to my attention by my hosting company. And sure enough I went to check the mail queue log in my control panel, I'm using plesk btw, and it failed to open a sure sign that someone has hacked us and is sending spam.

Unfortunately, the tech support is non existent, or very unhelpful I should say which is fair enough. The only clues I have is that I have the so called username, and the name of the script that appears to be sending it from.

The script is called config.php, and as far as I can see the only file on my server containing config.php is a codeigniter file.

I've changed my VPN root password, I've changed the ftp passwords they are very strong 20 random alpha numeric characters. I've deleted all the files on one website that the spam header appeared to come from.

Now I don't even use the servers mail service to send emails, I always use gmail. So I was thinking why can't I just turn off sendmail and qmail, which appears to be what my server uses.

I'm really confused on how to start debugging this. Obviously, this is high priority, our hosting has threatened to shut us down if we don't rectify this soon, and I don't want to get blacklisted.

Has anyone had any experience dealing with this sort of thing?

[iamthwee]

Anyone have any idea, I'm stumped :(

[cereal]

Are you sure the spammer is using your PHP scripts to send spam and not accessing directly to your SMTP server?

Usually the config.php file in CI does not execute code, it's a container for variables, can you verify if the file is genuine?

Also CI includes .htaccess files into the application and system directories with the Deny from all directive: it means the config files are not directly reachable by an HTTP request, the server will return status 403, but it can be opened through another script, for example a fake image file.

But why to overwrite the config.php of CI?

I think it's another file, for example, if using PHP-FPM and this is not properly set up, an attacker can upload an image script with embedded code an make the server execute it by appending a fake script name, for example:

/images/smile.jpg

Then from remote the attacker calls:

http://domain.tld/images/smile.jpg/config.php

And the code embedded is executed.

Now, can you check the path of the script from the mail log? Can you scan the system in search of files? If you can open a shell with the server, try this command:

find / -name 'config.php' 2>/dev/null

It should return all the config.php files under the root of the server.

So I was thinking why can't I just turn off sendmail and qmail, which appears to be what my server uses.

Yes, but if there's a backdoor, then they can reactivate it or connect to another SMTP server using your VPS as base. It depends a lot on the kind of infection. Your hosting should provide assistance for this kind of situation.

[iamthwee]

Are you sure the spammer is using your PHP scripts to send spam and not accessing directly to your SMTP server?

I honestly don't know, when I examined the mail queue there were about 29,000 messages queued and all seemed to be sending from one of two domains.

Since these domains are non critical I removed them and all their contents, it is possible the script could have resided there. I also stopped the mail services, and as far as I can tell they haven't restarted.

I've checked my cron jobs and all seem legit.

I believe php-fpm is set up:

ps -ef | grep php-fpm
root     30952 30571  0 14:22 pts/0    00:00:00 grep php-fpm

I did a root find of config.php on my server and all the config files seem legit. But then I did remove all the files from those two domains which may have been where the problem file was.

I have changed all my ftp passwords and I have changed the root password to log onto the vps.

The two domains that were sending out spam coincidentally have the same wordpress themes installed where a major vulnerability was discovered... using the revolution slider. So I'm thinking perhaps this is where the problem was, but again not sure.

Something is hijacking the internal server's mail service.

[cereal]

root 30952 30571 0 14:22 pts/0 00:00:00 grep php-fpm

No, that's the execution of grep, when in use you will see the master process and the workers, as here:

root      1437     1  0 set16 ?        00:00:01 php-fpm: master process (...)                    
www-data  1439  1437  0 set16 ?        00:00:02 php-fpm: pool www                                                       
www-data  1440  1437  0 set16 ?        00:00:01 php-fpm: pool www                                                       
user      8989  5877  0 00:49 pts/1    00:00:00 grep --color=auto php-fpm

The two domains that were sending out spam coincidentally have the same wordpress themes installed where a major vulnerability was discovered... using the revolution slider.

If you have this doubt then change the database password, because they used the database SELECT '<?php /*script*/ ?>' INTO OUTFILE 'config.php' to write to the filesystem, it happens if mysqld (the server process) runs as root, more information here:

• http://dev.mysql.com/doc/refman/5.6/en/select-into.html
• http://blog.sucuri.net/2014/09/slider-revolution-plugin-critical-vulnerability-being-exploited.html

[iamthwee]

Hi Cereal,

Can you tell me what the command is to check if the php-fpm is running?

How do I know if the mysql runs as root, for each database I create a new username with all permissions except 'grant.' Is that the same thing?

I've deleted both those themes as I couldn't afford to take the risk of patching it which is fine. They weren't mission critical like the other sites.

As I've said before, I assume this has solved the issues, but I have turned off my mail server services so as excepted there aren't any spam mails in my mail queue.

Thanks.

[cereal]

for each database I create a new username with all permissions except 'grant.' Is that the same thing?

No, it's not the same, I'm referring to the owner of the process, which is system related, i.e. it's outside of the database environment.

How do I know if the mysql runs as root

Run ps -ef | grep mysqld the output should return:

mysql     1346  1129  0 09:39 ?        00:00:01 /usr/sbin/mysqld
    --basedir=/usr
    --datadir=/var/lib/mysql
    --plugin-dir=/usr/lib/mysql/plugin
    --user=mysql
    --pid-file=/var/run/mysqld/mysqld.pid
    --socket=/var/run/mysqld/mysqld.sock
    --port=3306

As you see here the user is mysql. If you see root, instead:

• make sure mysql user exists and which groups are associated, so run: id -uG mysql it should return only mysql.

• open /etc/mysql/my.cnf and find the [mysqld] section, there you can change the user from user=root to user=mysql

• restart the database: sudo service mysql restart, a reload won't change the owner of the process, a restart yes, but if you have problems then stop and start the process again. An issue can happen here: if the logs in /var/log/mysql/ are owned by root then the server will fail to start, if this happens you have to change the ownership of the directory to the mysql user:

  chown -R mysql /var/log/mysql/
  

then you can start the server daemon again. If in doubt check the syslog file:

    tail -n30 /var/log/syslog

If the installation was assigned to the root account, then there could be some other permission issue to solve, for more information read here:

• http://dev.mysql.com/doc/refman/5.6/en/unix-postinstallation.html

the command is to check if the php-fpm is running?

You can use the ps command, as you have already done, it's not running. The PHP-FPM was only an example, but if you want to check which SAPIs are available then check /etc/php5/.

As I've said before, I assume this has solved the issues, but I have turned off my mail server services so as excepted there aren't any spam mails in my mail queue.

Probably yes, but if MySQL can receive direct connection from remote, then they can still write to the filesystem. Assigning the process to the mysql user should limit the write access, but you have to change or remove the database users involved in the compromised domains, otherwise they will continue to mess up the database.

Good luck! :)

[iamthwee]

ps -ef | grep mysqld
root      1635     1  0 Mar16 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql     1740  1635  0 Mar16 ?        05:22:44 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
root     20135 20121  0 11:04 pts/0    00:00:00 grep mysqld

This is my output for checking mysql. Am I ok?

And Yes I have deleted the databases that I suspected the compromised domains used. As a precaution I may just change the log on details for all my databases.

[iamthwee]

Sorry, I'm a bit confused on this php-pfm, are you saying I don't have it, and is it necessary?

[cereal]

Sorry, I'm a bit confused on this php-pfm, are you saying I don't have it, and is it necessary?

Oh no, it is not necessary, this is just an handler for PHP to communicate with the web server. I'm sorry my example created confusion.

Traditional installations (Apache+PHP) use mod_php to bind the PHP engine to the Apache processes.

PHP-FPM (FastCGI Process Manager) is another handler: PHP starts it's own daemon on port 9000 and waits for a redirect from the web server, which in this case works as a proxy. The traditional mod_php works only with Apache, while PHP-FPM is supported by many web servers.

This is my output for checking mysql. Am I ok?

Yes and no. Yes because the process is assigned correctly to the mysql user, no because if the database user as the FILE privilege, then he can still read and write files. To avoid it you can remove this grant from non-administrative users or apply the secure-file-priv=/defined/path directive in my.cnf, to limit the action range.

A useful read about securing MySQL can be found here:

• http://dev.mysql.com/doc/refman/5.6/en/security-against-attack.html

As a precaution I may just change the log on details for all my databases.

Good!