How To: Restart Failing Services Automatically using Cron

[HostV Harry]

Hi,

Now first we need to create a script to check for services and restart if any of them is down. Then we need to add the script as a cron job to autmatically execute it at any specified time interval. But what's the code to put inside the script ?

Here's it,

/usr/bin/pgrep cpanel || /etc/init.d/cpanel restart
/usr/bin/pgrep httpd || /etc/init.d/httpd restart
/usr/bin/pgrep named || /etc/init.d/named restart
/usr/bin/pgrep ftp || /scripts/restartsrv_ftpserver
/usr/bin/pgrep exim || /etc/init.d/exim restart
/usr/bin/pgrep mysql || /etc/init.d/mysql restart
/usr/bin/pgrep courier || /scripts/restartsrv_courier
/usr/bin/pgrep sshd || /etc/init.d/sshd restart

Thats it. Save it to any filename say /root/check_services.
Give execute permission to the script,

chmod 700 /root/check_services

Then add it as a cron job. You can use the following command to add a line for the new cron job.

crontab -e

OR

Just edit the file /var/spool/cron/root and add the line to it,
for eg: here is a sample line for executing the script every 15 minutes,

*/15 * * * * /bin/bash /root/check_services

save the file and restart cron service.

/etc/init.d/crond restart

Thats it, done.

Now to check whether a vps is down and restart it a script has to be setup on the main hardware node. For that you will need to contact us. Just leave a word at our help desk.

Cheers !

[bdh734]

Interesting......
Very good, I am going to try it.

But...
What are the two pipes for or what do they mean? ( || )
Thanks!

[inteluser]

The pipes mean "or". In other words,
"find the process named <something> OR start the process named <something>".

[bdh734]

Thanks....but there is two of them.
I know what one means...but what about two side-by-side?

[bdh734]

Hello,
Okay I tried this and now I am getting emails every 15 minutes with

18025
1909
7489
9363
16033
17922
21669
22063
22438
22442
28056
30329
31957
Usage: /etc/init.d/named {start|stop|status|restart|condrestart|reload|probe}
26566
7606
28655
28668
29700
9883
28428
28430
28447
28449
28456
28457
28464
28465
28574
9804
13570

How do I stop this, and do you see any errors from this output.

[Parody]

Edit the cron so it sends to an email such as cron@yourdomain.com that way if you need them you get them but if you dont u still have them there just incace

[HostV Harry]

Hi,

@bdh734,

|| or two pipes stand for 'OR' where as the singe pipe just pipes the output of the preceding command to the successive one.

The output of these lines would of course be the process id's of these services OR a service restart. Thats why you are getting these mails with process id's.

@Parody

Just been late to check out the forum, Thanks for replying and sharing info..
After all information sharing is the first step to wisdom. :-)

To be frank , checking for process ID is not that the best idea to determine if a service is up, since there can be situations where the will a lot of pid's, but the service would be in almost a dead state. I have seen this with mysql and apache sometimes.. But thats quite rare and this logic works in most cases.

Cheers !

[bdh734]

Thank you all for your reply.
@Harry, I have learned something today. It is this little bit of knowledge that makes learning awesome.

@Parody, I had to disable the cron job because of the emails. This is a great bit of advise. One thing I have noticed about apache and php, we are always having to "out-think" it. We "out-think" apache into parsing html as php and so on.

@inteluser, I just a little confused at what you were trying to explain. Sorry.

Again...thanks to you all.

[HostV Harry]

Hello,

Even though a process ID does exist for a service, it might still be down. We have had such cases with apache, named and mysql mainly. So here's the altered script which would definitely make it more effective.

/usr/bin/pgrep cpanel || /etc/init.d/cpanel restart
/usr/bin/pgrep httpd || /etc/init.d/httpd restart
/usr/bin/nc -z localhost 80 || /etc/init.d/httpd restart
/usr/bin/pgrep named || /etc/init.d/named restart
/usr/bin/nc -z localhost 53 || /etc/init.d/named restart
/usr/bin/pgrep ftp || /scripts/restartsrv_ftpserver
/usr/bin/pgrep exim || /etc/init.d/exim restart
/usr/bin/pgrep mysql || /etc/init.d/mysql restart
/usr/bin/nc -z localhost 3306 || /etc/init.d/mysql restart
/usr/bin/pgrep courier || /scripts/restartsrv_courier
/usr/bin/pgrep sshd || /etc/init.d/sshd restart

The '-z' option of Netcat ( /usr/bin/nc) allows us to test connectivity to a port and returns success or failure as result.

 /usr/bin/nc  -z 80 || /etc/init.d/httpd restart

The above command tests connectivity to port 80 and restarts apache if it fails and hence makes better sense than looking out whether a process ID for apache exists !!

But for the above script to work netcat should be present on the server which is not the case by default. How ever you can install netcat on the server easily by using yum,

 yum -y install nc

Cheers !