log file archivaly utility

[can-mohan]

I am looking for log archival utility or approach which should be able to help me in based on below policy.
(1) whenever log file (system.log) size >30859 or file is 3 day old then it should archive the same with new name system.log1 and again start the feeding the log in system.log file from start after truncating its content.
I am going to write all the debug statements in log file through c++ code in linux environment.
Any immediate help will be appreciated.

[JamesCherrill]

You could simply:

Close the log file
Re-name it to a unique archive log file name
Open a new log file (same name as previous file before it was renamed)

[rubberman]

Look into the logrotate command and options. This is how we do it in enterprise systems. After the log file is rotated, you can copy it to backup media, to analyze at your leasure. Here is the relevant part of the logrotate man page:

DESCRIPTION
       logrotate  is  designed  to ease administration of systems that generate large numbers of log files.  It allows automatic
       rotation, compression, removal, and mailing of log files.  Each log file may be handled daily, weekly, monthly,  or  when
       it grows too large.

       Normally,  logrotate is run as a daily cron job.  It will not modify a log more than once in one day unless the criterion
       for that log is based on the log's size and logrotate is being run more than once each day, or unless the -f  or  --force
       option is used.

       Any number of config files may be given on the command line. Later config files may override the options given in earlier
       files, so the order in which the logrotate config files are listed is important.  Normally, a single  config  file  which
       includes  any  other  config  files  which  are  needed should be used.  See below for more information on how to use the
       include directive to accomplish this.  If a directory is given on the command line, every file in that directory is  used
       as a config file.

[JamesCherrill]

Sure, logrotate is a much more comprehensive solution, but I didn't think my simple suggestion was so bad as to deserve a reputation-reducing downvote.

[rubberman]

@JameCherrill - sorry, I woud not have down-voted your post if I was able to figure out a way to comment without doing that. Apologies. If you could advise me how to do that, I would appreciate it. That said, there are a number of reasons why simply closing and renaming the file and moving it elsewhere is not a (in my opinion) valid option as there could easily be more logs coming it that would still be going to the original filel Logrotate deals with all of that cruft. Having to deal with thousands of log files at Nokia in real time, I quickly discovered the "joys" of the tool! No lost data was one of the bigger benefits. We were collecting 10 billion data points per day from SAR, SNMP, and application log files from 5000+ servers world-wide. I looked at other options and realized that using a proven tool for this was much more efficacious. 10 billion data points and nothing lost!

[rubberman]

I don't suppose there is a "neither up nor down vote" option for comments like mine? :-)

[JamesCherrill]

Hi, OK. AFAIK you can't add a comment without it being a rep-affecting vote. I such cases I just add a new post.

Anyway, I was just responding to a "do I need to archive the file of debug output and delete the content from the original?" kind of question with a " no, there's a simpler way" answer. But there's no way I would suggest such an approach for management of all logs from a large-scale live system

I fully agree with what you said. :)

JC

ps: Just a point of curiosity: how do you know you haven't lost a single log entry in 10,000,000,000?

[can-mohan]

"You could simply:
Close the log file
Re-name it to a unique archive log file name
Open a new log file (same name as previous file before it was renamed)"

i can't open new log file as this file is created by the other process and same file needs to be used for logging. if you truncate or rename the log file , loging activity will be completely stopped.

[JamesCherrill]

OK, not such a good idea then! Nevermind. JC