Member Avatar for fo2sh

Hello Everybody,

I am trying to run a simple script to monitor a log file, where I need to send an alarm every time specific line is being written into this log file, currently my script is working fine, the problem is with the miltiple lines that is being written whenever I have a problem with my SW and many lines are logged into this log file which is causing many alarms to be sent, so I am seeking a way to trigger sending the alarm only one time but without existing this script, I need my script to keep running 24/7 to monitor any errors in log's, but when the script captures an error lines in log file, I want to send the alarm one time, for example, it will be fine for me if I can send one alarm for error occurrence in log file for each 1 or 2 minutes, but sometimes I am having more than 100 error written within seconds causing more than 100 alarm to be triggered.

This is my script:

!/bin/sh

tail --pid=$$ -F /usr/fuad/testing_alarms/errors_log.log | while read line ; do
echo "$line" | egrep -v --line-buffered "error"
if [ $? = 0 ];
then

Send an SMS

/usr/fuad/testing_alarms_fuad/send_alarm.pl
fi
done

I have also tried something like the below, but also each line written into the file errors_log.log is triggering the alarm causing many alarms to be sent.

!/bin/sh

tail --pid=$$ -F /usr/fuad/testing_alarms/errors_log.log | egrep --line-buffered "error.Link.[012]." *| /usr/fuad/testing_alarms_fuad/send_alarm.pl

What I really need to do, whenever a line contains the string "error" is being written into the file errors_log.log, I want to trigger the script "send_alarm.pl", but when I have 100 line with string "error" written into errors_log.log within 3-4 seconds, causing 100 alarm to be sent, this is a problem, so how can I avoid duplicates without the option's of exiting the scripts as I need it to keep running and checking for errors? I want to avoid duplicates that appear within X period of time at least, so if alarms is sent due duplicates every 2 minutes for example will be fine but I dont want to trigger alarms for all errors appearence specially when they are a lot within seconds.

Thank you in advance.

Edited by fo2sh
  • 3 Contributors
  • 4 Replies
  • 466 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by rproffitt

Dani AI

Generated

Useful follow-ups and fixes building on and ’s cooldown idea.

Problem summary: your loop is correctly spotting lines but floods occur when hundreds of matching lines arrive in seconds. Don’t solve that by sleeping inside the read loop or by using return (that will stop the loop). Instead, treat alarm-sending as “rate‑limited” (one send per N seconds) or as “batched” (collect matches for N seconds, then send one summary). Also be careful: piping tail | while read can run the loop in a subshell depending on shell, which complicates state — persist the last-alert time to a file.

Example: a small, robust bash pattern that enforces a cooldown (replace paths and cooldown as needed):

LOG=/path/to/errors_log.log
SEND=/path/to/send_alarm.pl
ALERT_TS=/tmp/last_error_alert.ts
COOLDOWN=120   # seconds

while IFS= read -r line; do
  case "$line" in
    *error*)
      now=$(date +%s)
      last=$(cat "$ALERT_TS" 2>/dev/null || echo 0)
      if (( now - last >= COOLDOWN )); then
        "$SEND"
        printf "%d" "$now" > "$ALERT_TS"
      fi
      ;;
  esac
done < <(tail -n0 -F "$LOG")

Practical notes and troubleshooting:

  • Use process substitution (done < <(tail ...)) so your loop runs in the main shell and can update files cleanly.
  • The case pattern is simple and avoids buffering quirks. If you prefer regex, use grep --line-buffered or awk upstream.
  • If your environment may run concurrent instances, protect the timestamp write with flock or an atomic move to avoid races.
  • To send a single summary instead of just throttling, buffer matching lines to a temp file and send the aggregated content when the cooldown expires.

Finally: check the conditional logic in your posted snippet — using inverted matches (e.g., -v) or return inside the loop are common pitfalls. The pattern above implements the same spirit as / but keeps the monitor running 24/7 while preventing alarm floods.

Recommended Answers

All 4 Replies

Member Avatar for rproffitt

Thinking simple here. Try this (pseudocode only)

' Found an alarm. Send SMS
/usr/fuad/testing_alarms_fuad/send_alarm.pl
' Don't send alarms for 10 minutes.
sleep 10 minutes
'Reset alarms, return to loop.

Member Avatar for rubberman

has it right. Set a timer after alarm is raised, or set a number of events to process before raising another alarm within some specified (and adaptable) time frame.

Member Avatar for fo2sh

Thank you rproffitt and rubberman!

As I am not so familiar with this, I have done it with google help :)
I would like to share what I got from your comments, I modified the script to be as the following.

!/bin/sh
tail --pid=$$ -F /usr/fuad/testing_alarms/errors_log.log | while read line ; do
echo "$line" | egrep -v --line-buffered "error"
if [ $? = 0 ];
then
Send an SMS
/usr/fuad/testing_alarms_fuad/send_alarm.pl
fi
sleep 15
return
done

Its what you have suggested, right?

Member Avatar for rproffitt

. That's the idea. You limit the flood in your example to once every 15 seconds. If that works for you, you're done. There are perfectionists that will want more but in many cases you only need to get the tech the message and said tech will log in more than 15 seconds later to read the log and ... see the flood if there was one.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.