shell-scripting - Tar-ing and zip-ing files after a specific ...

dannyfang 0 Newbie Poster

18 Years Ago

Hi,

I have a script which is to perform the following task:
1) check for files in a particular directory.

2) The directory should contain 4 files in the form of:
<OSSInstance>_<OSSName>_BSC<Nokia BSC ID>.<counter no>.<YYYYMMDDHHMM>.<gid>

whereby:
OSSInstance - a combination of alphanumeric characters
OSSName -a combination of alphanumeric characters
Nokia BSC ID - a combination of alphanumeric characters
counter no - Digits 1,2, 3 and 72 only
YYYYMMDDHHMM - basically a numeric value denoting timestamp of the file i.e. 200603130500
gid - a numeric value for a particular process

An example of the file is shown below as:
1003_oxnn2_BSC48379.72.200603130500.768751

3)There is a timeout period in the script which checks if these 4 files are available within the particular directory.These 4 files are being transferred remotely from some machine.

4)If ALL 4 files are already in the directory within the specified timeout period, then it's copied to another directory where it's tar-ed and gzip-ed.

5)If one(1) of the 4 files are not the directory within the specified timeout period, these files are copied elsewhere and their names written into a log file.

6) The timeout period in waiting for ALL 4 files to arrive is 5 mins.

However I've encountered some problems within the script, which is captured below:
prod-cingtuna\ :/mkl/users/lows >./Nokia_RAN-tarNew.sh
1003_oxnn2_BSC48379.1.200603130500.768741: dskjfhkdjs: not found
1003_oxnn2_BSC48379.1.200603130500.768741: djkdahlk: not found
1003_oxnn2_BSC48379.1.200603130500.768741: anjhsahdklsad: not found
1003_oxnn2_BSC48379.1.200603130500.768741: askjdhasas^Jdaskj^Jakjchdajk^J: not found
OSS_INSTANCE
1003_oxnn2_BSC48379.1.200603130500.768741: dskjfhkdjs: not found
1003_oxnn2_BSC48379.1.200603130500.768741: djkdahlk: not found
1003_oxnn2_BSC48379.1.200603130500.768741: anjhsahdklsad: not found
1003_oxnn2_BSC48379.1.200603130500.768741: askjdhasas^Jdaskj^Jakjchdajk^J: not found
OSS_NAME
1003_oxnn2_BSC48379.1.200603130500.768741: dskjfhkdjs: not found
1003_oxnn2_BSC48379.1.200603130500.768741: djkdahlk: not found
1003_oxnn2_BSC48379.1.200603130500.768741: anjhsahdklsad: not found
1003_oxnn2_BSC48379.1.200603130500.768741: askjdhasas^Jdaskj^Jakjchdajk^J: not found
TMP_VAR
BSC_VAL
NOKIA_BSC_ID
TIMESTAMP
GRP_DAY 20
hrmin 17:50
GRP_HOUR 17
GRP_MIN 50
NUM_FILES 1
1003_oxnn2_BSC48379.2.200603130500.768750: dskjfhkdjs: not found
1003_oxnn2_BSC48379.2.200603130500.768750: djkdahlk: not found
1003_oxnn2_BSC48379.2.200603130500.768750: anjhsahdklsad: not found
1003_oxnn2_BSC48379.2.200603130500.768750: askjdhasas^Jdaskj^Jakjchdajk^J: not found
OSS_INSTANCE
1003_oxnn2_BSC48379.2.200603130500.768750: dskjfhkdjs: not found
1003_oxnn2_BSC48379.2.200603130500.768750: djkdahlk: not found
1003_oxnn2_BSC48379.2.200603130500.768750: anjhsahdklsad: not found
1003_oxnn2_BSC48379.2.200603130500.768750: askjdhasas^Jdaskj^Jakjchdajk^J: not found
OSS_NAME
1003_oxnn2_BSC48379.2.200603130500.768750: dskjfhkdjs: not found
1003_oxnn2_BSC48379.2.200603130500.768750: djkdahlk: not found
1003_oxnn2_BSC48379.2.200603130500.768750: anjhsahdklsad: not found
1003_oxnn2_BSC48379.2.200603130500.768750: askjdhasas^Jdaskj^Jakjchdajk^J: not found
TMP_VAR
BSC_VAL
NOKIA_BSC_ID
TIMESTAMP
GRP_DAY 20
hrmin 17:50
GRP_HOUR 17
GRP_MIN 50
NUM_FILES 2
1003_oxnn2_BSC48379.3.200603140500.768751: dskjfhkdjs: not found
1003_oxnn2_BSC48379.3.200603140500.768751: djkdahlk: not found
1003_oxnn2_BSC48379.3.200603140500.768751: anjhsahdklsad: not found
1003_oxnn2_BSC48379.3.200603140500.768751: askjdhasas^Jdaskj^Jakjchdajk^J: not found
OSS_INSTANCE
1003_oxnn2_BSC48379.3.200603140500.768751: dskjfhkdjs: not found
1003_oxnn2_BSC48379.3.200603140500.768751: djkdahlk: not found
1003_oxnn2_BSC48379.3.200603140500.768751: anjhsahdklsad: not found
1003_oxnn2_BSC48379.3.200603140500.768751: askjdhasas^Jdaskj^Jakjchdajk^J: not found
OSS_NAME
1003_oxnn2_BSC48379.3.200603140500.768751: dskjfhkdjs: not found
1003_oxnn2_BSC48379.3.200603140500.768751: djkdahlk: not found
1003_oxnn2_BSC48379.3.200603140500.768751: anjhsahdklsad: not found
1003_oxnn2_BSC48379.3.200603140500.768751: askjdhasas^Jdaskj^Jakjchdajk^J: not found
TMP_VAR
BSC_VAL
NOKIA_BSC_ID
TIMESTAMP
GRP_DAY 20
hrmin 17:50
GRP_HOUR 17
GRP_MIN 50
NUM_FILES 3
1003_oxnn2_BSC48379.72.200603130500.768751: dskjfhkdjs: not found
1003_oxnn2_BSC48379.72.200603130500.768751: djkdahlk: not found
1003_oxnn2_BSC48379.72.200603130500.768751: anjhsahdklsad: not found
1003_oxnn2_BSC48379.72.200603130500.768751: askjdhasas^Jdaskj^Jakjchdajk^J: not found
OSS_INSTANCE
1003_oxnn2_BSC48379.72.200603130500.768751: dskjfhkdjs: not found
1003_oxnn2_BSC48379.72.200603130500.768751: djkdahlk: not found
1003_oxnn2_BSC48379.72.200603130500.768751: anjhsahdklsad: not found
1003_oxnn2_BSC48379.72.200603130500.768751: askjdhasas^Jdaskj^Jakjchdajk^J: not found
OSS_NAME
1003_oxnn2_BSC48379.72.200603130500.768751: dskjfhkdjs: not found
1003_oxnn2_BSC48379.72.200603130500.768751: djkdahlk: not found
1003_oxnn2_BSC48379.72.200603130500.768751: anjhsahdklsad: not found
1003_oxnn2_BSC48379.72.200603130500.768751: askjdhasas^Jdaskj^Jakjchdajk^J: not found
TMP_VAR
BSC_VAL
NOKIA_BSC_ID
TIMESTAMP
GRP_DAY 20
hrmin 17:50
GRP_HOUR 17
GRP_MIN 50
NUM_FILES 4
a /tmp/osspkg2s/uma/umabass/cs_bsspm_aif/client/out/1003_oxnn2_BSC48379.1.200603130500.768741 1K
a /tmp/osspkg2s/uma/umabass/cs_bsspm_aif/client/out/1003_oxnn2_BSC48379.2.200603130500.768750 1K
a /tmp/osspkg2s/uma/umabass/cs_bsspm_aif/client/out/1003_oxnn2_BSC48379.3.200603140500.768751 1K
a /tmp/osspkg2s/uma/umabass/cs_bsspm_aif/client/out/1003_oxnn2_BSC48379.72.200603130500.768751 1K
prod-cingtuna\ :/mkl/users/lows >

From the script output above:
1) I'm not sure why it is producing the error"1003_oxnn2_BSC48379.1.200603130500.768741: dskjfhkdjs: not found"
The file contents are:
prod-cingtuna\ :/mkl/users/lows >cat 1003_oxnn2_BSC48379.1.200603130500.768741
dskjfhkdjs
djkdahlk
anjhsahdklsad
askjdhasas'
daskj
akjchdajk

2) It seems that for every line in the file, the script outputs the "not found" error to the standard output - I'm not able to see where it's coming from witin the script.

3) Some of the values for the variables shown below within the script are not having any values inspite being assigned values from AWK:
TMP_VAR
BSC_VAL
NOKIA_BSC_ID
TIMESTAMP
The AWK statement has been tested on the command line and it's producing correct

4)Once all 4 files are in the specified directory, they all do get copied to another directory (specified as /tmp/osspkg2s/uma/umabass/cs_bsspm_aif/client/out in the script)
However, tar and gzip process failed to tar these 4 files correctly(according to the name specified in the directory) into a single file and zip them.
Instead, the file ._.._.BSC.BSC_VAL..tar.gz
is produced:
prod-cingtuna\ :/mkl/users/lows >ls -la
total 112
-rw-r--r-- 1 lows metrica 322 Mar 20 19:44 ._.._.BSC.BSC_VAL..tar.gz

Could anyone help me out by pointing out where did I go wrong in the script?

Feel free to modify the script.

Thanks in advance.
Danny

shell-scripting

Nokia_RAN-tarNew.txt (6.88 KB)

#!/bin/sh

# Set the path for the Unix executables
AWK="/usr/bin/awk"
TAR="/usr/bin/tar"
GZIP="/usr/bin/gzip"
GREP="/usr/xpg4/bin/grep"

VENDOR_TECH="NOKIA-RAN"                       
FTYPE="RAN"                                   
BASE_DIR="/tmp/osspkg2s/uma/umabass/cs_bsspm_aif/client"
#OSS_DIR="$BASE_DIR/$1"                           
INPUT_DIR="$BASE_DIR/in"                          
OUTPUT_DIR="$BASE_DIR/out"                        
TEMP_DIR="$BASE_DIR/tmp"                          

if [ ! -d "$INPUT_DIR" -o ! -d "$OUTPUT_DIR" -o ! -d "$TEMP_DIR" ] ; then       
   echo "Directories missing. Please check if input, output, temp and log directories are created."
   exit 1                                                                                          
fi                                                                                                 
                                                                                    
# Timeout in minutes to wait the missing file               
TIMEOUT=5                                   
TOTAL_MINUTES=0                             

# Function to check the timeout for generating the TAR, even 
# if a file is still missing                                
check_timeout() {                                           
   # File date/time                                         
   ## FDAY, FHOUR and FMINUTE values are being passed by arguments into check_timeout
   
   FDAY=$1                                                  
   FHOUR=$2                                                 
   FMINUTE=$3                                               
   
   # Current date/time                                      
   CDAY=`date +%d`                                          
   CHOUR=`date +%H`                                         
   CMINUTE=`date +%M`                                       
                                                            
   E_DAYS=`expr $CDAY - $FDAY`                              
   E_HOURS=`expr $CHOUR - $FHOUR`                           
   E_MINS=`expr $CMINUTE - $FMINUTE`                        
                                                            
   if [ $E_DAYS -gt 0 ] ; then                              
	TOTAL_HOURS=`expr $E_DAYS \* 24 + $E_HOURS`                
   elif [ $E_DAYS -lt 0 ] ; then                            
	TOTAL_HOURS=`expr 24 + $E_HOURS`                           
   else                                                     
	TOTAL_HOURS=$E_HOURS                                       
   fi                                                       
                                                            
   TOTAL_MINUTES=`expr $TOTAL_HOURS \* 60 + $E_MINS`        
}                                                           
# Error messages                                                                                                  
MSG_TIMEOUT_EXPIRED="TIMEOUT waiting files for group has expired."                                                
MSG_MISSING_FILES="The number of files is lower than expected. It will wait until TIMEOUT expires."               
MSG_INVALID_FILE_NAME="The input filename is invalid. TAR file cannot be generated for this file."                
MSG_PRG_EXECUTION_FAILED="The execution of this program has failed. Files have been moved to the TEMP directory." 


# Function to create the error log file
create_log() {
   TS=`date +%Y%m%d%H%M`
   #ID=$1
   ERROR_TYPE=$2

   case $ERROR_TYPE in
   	TIMEOUT_EXPIRED) MSG=$MSG_TIMEOUT_EXPIRED ;;
   	MISSING_FILES) MSG=$MSG_MISSING_FILES ;;
   	INVALID_FILE_NAME) MSG=$MSG_INVALID_FILE_NAME ;;
   	PRG_EXECUTION_FAILED) MSG=$MSG_PRG_EXECUTION_FAILED ;;
   esac
   
   #flogname="$VENDOR_TECH.$FTYPE.$ERROR_TYPE.$ID.$TS.err"
   flogname="$VENDOR_TECH.$FTYPE.$ERROR_TYPE.$TS.err"
   #echo "[`date`] ID: $ID -  $MSG" > $TEMP_DIR/$flogname;
}
                                                                                                                  
# Result flag to check if it is ok to TAR the files
RESULT=""                                          
                                                   
NUM_FILES=0
NUM_EXPECTED_FILES=4
                
cd $INPUT_DIR
## file format: <OSSInstance>_<OSSName>_BSC<Nokia BSC ID>.<counter no>.<YYYYMMDDHHMM>.<gid>

NOKIA_BSC_ID=""
OSS_INSTANCE=""
OSS_NAME=""
TIMESTAMP=""

## Loop all files in the directory to check if there's a match 
for file in `ls $INPUT_DIR | $GREP -E '^[a-zA-Z0-9]+_[a-zA-Z0-9]+_[a-zA-Z0-9]+\.(1|2|3|72)\.[0-9]{12}\.[0-9]'`                                
do    
    if [ -z $file ];then
       create_log "$file" "INVALID_FILE_NAME"
   	   rm $file
	   continue   
   	else
  	   if [ $NUM_FILES -eq $NUM_EXPECTED_FILES ] ; then
   	      RESULT="OK"
   	      echo "EQUAL"	    
   	      cp $file $OUTPUT_DIR
       else
       	  OSS_INSTANCE=echo $file | awk 'BEGIN { FS = "_"} {print $1}'  ##1003
       	  echo OSS_INSTANCE $OSS_INSTANCE
       	  
    	  OSS_NAME=echo $file | awk 'BEGIN { FS = "_"} {print $2}'   ## oxnn2
    	  echo OSS_NAME $OSS_NAME
    	  
    	  TMP_VAR=echo $file | awk 'BEGIN { FS = "_"} {print $3}'    
    	  echo TMP_VAR $TMP_VAR
    	  
    	  BSC_VAL=echo $TMP_VAR| awk 'BEGIN { FS = "_"} {print $3}'|awk 'BEGIN { FS = "."}{print $1}' ## BSC34565
    	  echo BSC_VAL $BSC_VAL
    	  
    	  NOKIA_BSC_ID=echo $BSC_VAL|cut -c4-
    	  echo NOKIA_BSC_ID $NOKIA_BSC_ID
    	  
    	  TIMESTAMP=echo $TMP_VAR| awk 'BEGIN { FS = "_"} {print $3}'|awk 'BEGIN { FS = "."}{print $3}' ##200603130500
    	  echo TIMESTAMP $TIMESTAMP 
       
   	      GRP_DAY=`ls -l $file | $AWK 'BEGIN { FS = " "} { print $7 }'`
   	      echo GRP_DAY $GRP_DAY
   	      
	      hrmin=`ls -l $file | $AWK 'BEGIN { FS = " "} { print $8 }'` ##11:18
	      echo hrmin $hrmin
	      
	      GRP_HOUR=`echo $hrmin | $AWK 'BEGIN { FS = ":"} { print $1 }'`
	      echo GRP_HOUR $GRP_HOUR
	      
	      GRP_MIN=`echo $hrmin | $AWK 'BEGIN { FS = ":"} { print $2 }'`
	      echo GRP_MIN $GRP_MIN
	      
   	      NUM_FILES=`expr $NUM_FILES + 1`
   	      echo NUM_FILES $NUM_FILES
   	      
   	      check_timeout $GRP_DAY $GRP_HOUR $GRP_MIN

   	   	  if [ $TOTAL_MINUTES -gt $TIMEOUT ] ; then
   	   	  	 cp $file $OUTPUT_DIR
		     create_log "$file" "TIMEOUT_EXPIRED"	  			   	        
		  else  ## missing file - file did not arrive after timeout. 
		     create_log "$file" "MISSING_FILES"   
   	      fi
   	   fi   	     
   	fi   
done
   	
cd $OUTPUT_DIR
TAR_FILE_NAME=$OSS_INTANCE."_".$OSS_NAME."_".BSC."BSC_VAL".$TIMESTAMP.tar
$TAR -cvf $TAR_FILE_NAME $OUTPUT_DIR/*

$GZIP $TAR_FILE_NAME

1 Contributor
0 Replies
88 Views

Be the first to reply

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.

Tar-ing and zip-ing files after a specific time frame - HELP NEEDED