I am writing a simple shell script to automate a simple backup of a website and it's database. This is the working code:

#!/bin/bash
# creates a backup of the mysql & webdata for a specific website.

TODAY=`date +%A`

# these variables cannot contain any spaces and must be modified based on:
#       the name of the /automation/folder/ of the application being backed up (application)
#       on what server (localhost)
#       where the backed up data should be replicated (remotehost)
#       the location of the www data for the specific application (webdata)
#               /var/www/somefolder
#       the username, password and database required to establish the mysql connection (sqluser, sqlpass, sqldb)
#               SQLPASS must contain the -p prefix
#               example: -pS0m3PassW0rd
APPLICATION=someapp
LOCALHOST=thisserver
REMOTEHOST=otherserver
WEBDATA=location
SQLUSER=usernam
SQLPASS=-ppassword
SQLDB=dbname

# these variables are specific to the application being backed up.
ARCHIVE=/automation/$APPLICATION/$LOCALHOST
ERRORLOG=/automation/$APPLICATION/errorlog.txt

SQLFILE=/automation/$APPLICATION/$APPLICATION.sql
WEBFILE=/automation/$APPLICATION/$APPLICATION.web.tar

SQLGZ=/automation/$APPLICATION/$APPLICATION.sql.gz
WEBGZ=/automation/$APPLICATION/$APPLICATION.web.tar.gz

SQLA=/automation/$APPLICATION/$LOCALHOST/$APPLICATION.sql.gz
WEBA=/automation/$APPLICATION/$LOCALHOST/$APPLICATION.web.tar.gz

rm $SQLFILE
rm $WEBFILE
rm $SQLGZ
rm $WEBGZ
echo "Backup started: " `date` >> $ERRORLOG
mysqldump -u $SQLUSER $SQLPASS $SQLDB > $SQLFILE
tar -cf $WEBFILE $WEBDATA

if [ -s $SQLFILE ];
then
    if [ -s $WEBFILE ];
    then
       gzip $SQLFILE
       gzip $WEBFILE
    else
       echo $SQLFILE was created, $WEBFILE was not, script terminating. >> $ERRORLOG
       exit
    fi
else
    echo $SQLFILE was not created, script terminating. >> $ERRORLOG
    exit
fi

if [ -s $SQLGZ ];
then
    if [ -s $WEBGZ ];
    then
       rm $ARCHIVE/*.$TODAY
       mv $SQLGZ $SQLA.$TODAY
       mv $WEBGZ $WEBA.$TODAY
    else
       echo $SQLGZ was created, $WEBGZ was not, script terminating. >> $ERRORLOG
       exit
    fi
else
    echo $SQLGZ was not created, script terminating. >> $ERRORLOG
    exit
fi

scp $SQLA.$TODAY username@$REMOTEHOST.domain.com:/automation/$APPLICATION/$LOCALHOST
scp $WEBA.$TODAY edcns5al@$REMOTEHOST.domain.com:/automation/$APPLICATION/$LOCALHOST
chown -R root:automation /automation/
chmod -R 770 /automation/
echo Backup completed: `date`  >> $ERRORLOG

What I want to do is be able to change the dynamic variables based on an answer file or parameter using something like this from the command line:

backup.sh filename.ans

The filename.ans would contain the variables that I want to set that way I can specify what backup I want to execute without having to maintain multiple copies of the script, each containing their own changes to the variables.

I've tried to use the following code to read the file, but all I can do is cat the file.

for fname in "$@"; do
        bash "$fname"

# these variables are specific to the application being backed up.
ARCHIVE=/automation/$APPLICATION/$LOCALHOST
ERRORLOG=/automation/$APPLICATION/errorlog.txt

SQLFILE=/automation/$APPLICATION/$APPLICATION.sql
WEBFILE=/automation/$APPLICATION/$APPLICATION.web.tar

SQLGZ=/automation/$APPLICATION/$APPLICATION.sql.gz
WEBGZ=/automation/$APPLICATION/$APPLICATION.web.tar.gz

SQLA=/automation/$APPLICATION/$LOCALHOST/$APPLICATION.sql.gz
WEBA=/automation/$APPLICATION/$LOCALHOST/$APPLICATION.web.tar.gz

echo $SQLA
done
exit

But the echo returns /automation///.sql.gz

What is the easiest way to load the variables from another file?

This is probably very basic, but unfortunately I don't know the term to search on if I wanted to do what I am trying, so I am coming up empty.

I think I'm getting closer, I can read and echo each line of a file when I use the following command:

./script.sh file1.ans

The code inside of script.sh:

for fname in $@; do
        while read line; do
        echo $line # or whaterver you want to do with the $line variable
        done < $fname
done

The lines in the file1.ans

alpha
bravo
charlie

I think I need to build an array somehow using each line read so I can then assign each line to a variable. I'm still looking, but any help is appreciated. :)

There are 2 simple options:
1. If the number of params are not too many just pass them on command line:

#/bin/bash
APPLICATION=$1
LOCALHOST=`hostname -s`
REMOTEHOST=$2
WEBDATA=$3
SQLUSER=$4
SQLPASS=$5
SQLDB=$6

# ---------------------------------------------
#run like this:
backup.sh appName remotehost webdataLocation usr pswd db

You name make them named params using the getopt. Google for more examples.

2. Use the file as you mentioned. Usually I would go with the property file syntax.
key=value
In this case you can parse the file like this. An except from one of my scripts:

#!/bin/bash
#
# Sets internal variables based on <config file> contents.
# $1 - <config file>
# returns nothing, sets all variables.
#
process_TC_config_file() {
	_logecho "Entering _process_TC_config_file(): \$* = $*"
	_logecho "TC_CONFIG_FILE=$TC_CONFIG_FILE"
	local NEW_INPUT_FILE=$TC_DIR/new_config
	cp $1 $NEW_INPUT_FILE

	# remove leading spaces/tabs and comments
	sed -i '/^$/d;s/^[ \t]+//g;/^#/d' $NEW_INPUT_FILE
	# work around for "read". It returns non-zero for last line in file.
	# So if the last line contains valid input it will be lost.
	echo "" >> $NEW_INPUT_FILE
	
	while read one_line
	do
		if [ ! -z "$(echo $one_line | grep ^ADAPTATIONS)" ]; then
			ADAP_LIST=`echo $one_line | sed 's/ADAPTATIONS=//'`
		elif [ ! -z "$(echo $one_line | grep ^NUMBER_OF_HOURS_OF_DATA_LOADING)" ]; then
			NUMBER_OF_HOURS_OF_DATA_LOADING=`echo $one_line | sed 's/NUMBER_OF_HOURS_OF_DATA_LOADING=//'`
		elif [ ! -z "$(echo $one_line | grep ^ORACLE_MEM_SET_SIZE_PERCENT)" ]; then
			ORACLE_MEM_SET_SIZE_PERCENT=`echo $one_line | sed 's/ORACLE_MEM_SET_SIZE_PERCENT=//'`
		elif [ ! -z "$(echo $one_line | grep ^DB_ARCHIVE_LOGGING_DIR)" ]; then
			DB_ARCHIVE_LOGGING_DIR=`echo $one_line | sed 's/DB_ARCHIVE_LOGGING_DIR=//'`
		elif [ ! -z "$(echo $one_line | grep ^WEBSPHERE_MIN_HEAP_SIZE)" ]; then
			WEBSPHERE_MIN_HEAP_SIZE=`echo $one_line | sed 's/WEBSPHERE_MIN_HEAP_SIZE=//'`
		elif [ ! -z "$(echo $one_line | grep ^WEBSPHERE_MAX_HEAP_SIZE)" ]; then
			WEBSPHERE_MAX_HEAP_SIZE=`echo $one_line | sed 's/WEBSPHERE_MAX_HEAP_SIZE=//'`
		elif [ ! -z "$(echo $one_line | grep ^OES_LOG_FILE_SIZE_LIMIT_BYTES)" ]; then
			OES_LOG_FILE_SIZE_LIMIT_BYTES=`echo $one_line | sed 's/OES_LOG_FILE_SIZE_LIMIT_BYTES=//'`
		elif [ ! -z "$(echo $one_line | grep ^OES_LOG_FILE_NUMBER_LIMIT)" ]; then
			OES_LOG_FILE_NUMBER_LIMIT=`echo $one_line | sed 's/OES_LOG_FILE_NUMBER_LIMIT=//'`
		elif [ ! -z "$(echo $one_line | grep ^WAS_LOG_FILE_SIZE_LIMIT_MB)" ]; then
			WAS_LOG_FILE_SIZE_LIMIT_MB=`echo $one_line | sed 's/WAS_LOG_FILE_SIZE_LIMIT_MB=//'`
		elif [ ! -z "$(echo $one_line | grep ^WAS_LOG_FILE_NUMBER_LIMIT)" ]; then
			WAS_LOG_FILE_NUMBER_LIMIT=`echo $one_line | sed 's/WAS_LOG_FILE_NUMBER_LIMIT=//'`
		elif [ ! -z "$(echo $one_line | grep ^XXX)" ]; then
			# this is just a template to introduce new params. Wont ever be executed.
			XXX=`echo $one_line | sed 's/XXX=//'`
		else
			_logecho "Line ignored: $one_line"
		fi
	done < $NEW_INPUT_FILE
	rm $NEW_INPUT_FILE
}

# --------main------------
if [ !-r $1 ]; then
   // err
   usage_and_exit(1);
fi
process_TC_config_file $1

#... rest of the script

Example contents of the config file.

#######################################################
# MANDATORY
#######################################################
# NAC trail adaptations
#ADAPTATIONS=com.nsn.brm-1.0 com.nsn.nthlrfe-4.5EP4 
# NAC full load adaptations
ADAPTATIONS=NOKIPA-A8 NOKMGW-U4.2EP NOKMSS-M16.0IP
NUMBER_OF_HOURS_OF_DATA_LOADING=36

#######################################################
# Oracle Configurations -- OPTIONAL
#######################################################
# Oracle's memory limit in % of total available RAM.
# Default 45.
ORACLE_MEM_SET_SIZE_PERCENT=45
# Directory where archive logs are stored.
# Default /d/db/archivelog.
DB_ARCHIVE_LOGGING_DIR=/d/db/archivelog

#################################################################
# WAS (WebSphere Application Server) configurations -- OPTIONAL
#################################################################
# Min/Max of WAS heapsize. Can't set only one. Either both or none.
# Default is to make no change to default WAS configuration.
#WEBSPHERE_MIN_HEAP_SIZE=768
#WEBSPHERE_MAX_HEAP_SIZE=6096
# Error / Output files: Size / file and total number of history files to be kept.
# Default 2 & 50 respectively.
WAS_LOG_FILE_SIZE_LIMIT_MB=2
WAS_LOG_FILE_NUMBER_LIMIT=50

#######################################################
# OES configurations -- OPTIONAL
#######################################################
# Log / Trace files: Size / file and total number of history files to be kept.
# Default 3145728 (3MB) & 50 respectively.
OES_LOG_FILE_SIZE_LIMIT_BYTES=3145728
OES_LOG_FILE_NUMBER_LIMIT=50

HTH

Thank you for your response. I can actually use both of those examples for a couple other ideas I had, thanks!

I found a post on another website that I am very interested in. I think it was also posted by you. :) HTH = hth ?

http://mandrivausers.org/index.php?/topic/21998-reading-a-text-file-line-by-line-with-bash/

old_IFS=$IFS
IFS=$'\n'
lines=($(cat FILE)) # array
IFS=$old_IFS

And my resulting code that works! (not yet fully tested, but appears very promising)

for fname in $@; do
   old_IFS=$IFS
   IFS=$'\n'
   lines=($(cat $fname)) # array
   IFS=$old_IFS
APPLICATION=${lines[0]}
REMOTEHOST=${lines[1]}
WEBDATA=${lines[2]}
done
exit

I have to use the answer file as a flat file with the answers in specific positions, but I can work with that.

I am really interested in your process_TC_config_file() script you posted but I don't yet fully understand it, but am working on it. Is that a function? sub-routine? something else?

Thank you very much for your response!

I am really interested in your process_TC_config_file() script you posted but I don't yet fully understand it, but am working on it. Is that a function? sub-routine? something else?

Yes, it's a function.
Just to make code readable. Hope you noticed that the function is called from line number 56.

I think it was also posted by you.

Nope, I would never show anyone how to use arrays in shell scripting. I hate them for their god-awful-cryptic-unreadable-confusing-crappy syntax.

APPLICATION=${lines[0]}
REMOTEHOST=${lines[1]}
WEBDATA=${lines[2]}

I'm not the most user-friendly guy, but this is stretching limits of user-unfriendliness. Expecting user to specify specific parameter WITHOUT a corresponding key name on a specific line number. Not good.


Technically though all are an option pick whatever sails your boat. I, like any self-respecting poster, would of course root for my way. :)

HTH = hth?

HTH = hth

For my own knowledge, may I ask why the function is at the top instead of the bottom?

Hmm, I ran the code you gave me to see if I can get away from my crazy array of death, and I get this error:

./echotest.sh: line 51: syntax error near unexpected token `1'
./echotest.sh: line 51: ` usage_and_exit(1);'

#!/bin/bash
# creates a backup of mysql & web data
# Usage: ./backup.sh answerfile

TODAY=`date +%A`
LOCALHOST=`hostname -s`

# This script requires an answer file to populate each dynamic variable

# Sets internal variables based on <config file> contents.
# $1 - <config file>
# returns nothing, sets all variables.

process_config_file() {
    _logecho "Entering _process_config_file(): \$* = $*"
    _logecho "CONFIG_FILE=$CONFIG_FILE"
    local NEW_INPUT_FILE=$CFG_DIR/new_config
    cp $1 $NEW_INPUT_FILE

    # remove leading spaces/tabs and comments
    sed -i '/^$/d;s/^[ \t]+//g;/^#/d' $NEW_INPUT_FILE

    # work around for "read". It returns non-zero for last line in file.
    # So if the last line contains valid input it will be lost.
    echo "" >> $NEW_INPUT_FILE

    while read one_line; do
        if [ ! -z "$(echo $one_line | grep ^APPLICATION)" ]; then
            APPLICATION=`echo $one_line | sed 's/APPLICATION=//'`
        elif [ ! -z "$(echo $one_line | grep ^REMOTEHOST)" ]; then
            REMOTEHOST=`echo $one_line | sed 's/REMOTEHOST=//'`
        elif [ ! -z "$(echo $one_line | grep ^WEBDATA)" ]; then
            WEBDATA=`echo $one_line | sed 's/WEBDATA=//'`
        elif [ ! -z "$(echo $one_line | grep ^SQLUSER)" ]; then
           SQLUSER=`echo $one_line | sed 's/SQLUSER=//'`
        elif [ ! -z "$(echo $one_line | grep ^SQLPASS)" ]; then
           SQLPASS=`echo $one_line | sed 's/SQLPASS=//'`
        elif [ ! -z "$(echo $one_line | grep ^SQLDB)" ]; then
           SQLDB=`echo $one_line | sed 's/SQLDB=//'`
        else
            _logecho "Line ignored: $one_line"
        fi
    done < $NEW_INPUT_FILE

    rm $NEW_INPUT_FILE
}

# --------main------------
if [ !-r $1 ]; then
    // err
    usage_and_exit(1);
fi
process_config_file $1
echo $APPLICATION
exit

#... rest of the script

# these variables are built by using the dynamic variables from the for loop (above)
ARCHIVE=/automation/$APPLICATION/$LOCALHOST
ERRORLOG=/automation/$APPLICATION/errorlog.txt

SQLFILE=/automation/$APPLICATION/$APPLICATION.sql
WEBFILE=/automation/$APPLICATION/$APPLICATION.web.tar

SQLGZ=/automation/$APPLICATION/$APPLICATION.sql.gz
WEBGZ=/automation/$APPLICATION/$APPLICATION.web.tar.gz

SQLA=/automation/$APPLICATION/$LOCALHOST/$APPLICATION.sql.gz
WEBA=/automation/$APPLICATION/$LOCALHOST/$APPLICATION.web.tar.gz

# clean the slate - remove any files left over from previous attempts to backup the files
rm $SQLFILE
rm $WEBFILE
rm $SQLGZ
rm $WEBGZ

# stamp the error log, collect the mysql and web data
echo "Backup started: " `date` >> $ERRORLOG
mysqldump -u $SQLUSER $SQLPASS $SQLDB > $SQLFILE
tar -cf $WEBFILE $WEBDATA

# verify the files were created then use gzip to compress them
if [ -s $SQLFILE ];
then
    if [ -s $WEBFILE ];
    then
       gzip $SQLFILE
       gzip $WEBFILE
    else
       echo $SQLFILE was created, $WEBFILE was not, script terminating. >> $ERRORLOG
       exit
    fi
else
    echo $SQLFILE was not created, script terminating. >> $ERRORLOG
    exit
fi

# verify the files were compressed
# remove the files from last week and move the ones from this week to the archive
if [ -s $SQLGZ ];
then
    if [ -s $WEBGZ ];
    then
       rm $ARCHIVE/*.$TODAY
       mv $SQLGZ $SQLA.$TODAY
       mv $WEBGZ $WEBA.$TODAY
    else
       echo $SQLGZ was created, $WEBGZ was not, script terminating. >> $ERRORLOG
       exit
    fi
else
    echo $SQLGZ was not created, script terminating. >> $ERRORLOG
    exit
fi

# copy the files to the remote host
# verify the permissions on the local automation folder are correct
# stamp the error log to complete the job
scp $SQLA.$TODAY edcns5al@$REMOTEHOST.domain.com:/automation/$APPLICATION/$LOCALHOST
scp $WEBA.$TODAY edcns5al@$REMOTEHOST.domain.com:/automation/$APPLICATION/$LOCALHOST

chown -R root:automation /automation/
chmod -R 770 /automation/
echo Backup completed: `date`  >> $ERRORLOG
exit

Why not just source the answer file? You could allow for any bash construct at that point. For instance
File: script.sh

#!/bin/bash

VAR="Original"

if [[ $# -gt 0 && -e ${1} ]]; then
    . ${1}
fi

echo "VAR: ${VAR}"

Which outputs just VAR: Original when run with no parameters. BUt provided an answer file such as
File: answer.file

VAR="Modified"

you can run with script.sh answer.file and get the following output: VAR: Modified

Why not just source the answer file? You could allow for any bash construct at that point. For instance
File: script.sh

#!/bin/bash

VAR="Original"

if [[ $# -gt 0 && -e ${1} ]]; then
    . ${1}
fi

echo "VAR: ${VAR}"

Which outputs just VAR: Original when run with no parameters. BUt provided an answer file such as
File: answer.file

VAR="Modified"

you can run with script.sh answer.file and get the following output: VAR: Modified

Didn't get how this would work? The "answer" file only contains answers, not the variable names (it's NOT property file syntax). It's just a list of values.
See my post with "I'm not the most user-friendly guy" above..

Sourcing a file with the appropriate format is the correct answer. Conforming to horrific requirements such as position dictated variable entries shows a lack of responsibility on the developer. The OP should push back to the designer or redesign him/herself.

In either case, given that the answer file is 'just a list' of entries the best approach is probably something along the lines of:

echo "M1" > .col1
echo "M2" >> .col1
echo "M3" >> .col1
echo "M4" >> .col1

paste -d= .col1 config.ans > .proper.format

. .proper.format

echo "M1: ${M1}"
echo "M2: ${M2}"
echo "M3: ${M3}"
echo "M4: ${M4}"

rm -f .col1 .proper.format

Which will create the proper format on the fly and source that.

However, this is a maintenance nightmare. I'd push hard before having to go this route.

The answer file does not have to be just a list of values, but when working with an array, I could not think of an easier way. I liked thekashyap's approach to use sed to trim the lead and assign the variables, but I am getting an error I don't know how to fix (see above), any help there is appreciated. :)

As to sourcing the answer file, I'm also looking into that right now.

You don't define a usage_and_exit function that I see so trying to call it is meaningless. Also, in the original example you provided (file1.ans) there are no leading space or comment characters, why are you trying to strip them? If there are other formats it would be helpful to know that before trying to answer questions.

The script you are having problems with will not work if your format is not like that shown by thekashyap (i.e. VAR=VAL format). If it is in that format then just source the file; comments and whitespace will be ignored for you by the interpreter.

I don't think you have a grasp on what you want. How about you succinctly describe the problem with relevant examples of your input. From that, we can start to point you in the right direction.

I did know what I wanted. I wanted to set variables using another file instead of in the script being run so that I can apply the same script to several environments and change the values without maintaining multiple copies of the script. I was searching the web for various answers and it looks like I found a few that work thanks to the both of you and some web searching:

I can
1. pass them on the command line
2. set them in an array (ugly)
3. process them in a function (haven't got that to work yet)
4. source the file

I was actually trying to figure out what sourcing a file was L7Sqr, so I missed your comments above. Like you said, sourcing the file is the right answer. If the answer file contains this:

APPLICATION=someapp
REMOTEHOST=someserver
WEBDATA=/var/www/folder
SQLUSER=someuser
SQLPASS=-ppassword
SQLDB=dbname

I can use the variables as if I had set them in the script.sh file directly. The answer file does not appear to require the #!/bin/bash, should I include it anyway?

Here is the resulting working code:

#!/bin/bash
# creates a backup of mysql & web data
# Usage: ./backup.sh answerfile

TODAY=`date +%A`
LOCALHOST=`hostname -s`

# This script requires an answer file to populate each dynamic variable
# Thus eliminating the need to maintain multiple copies of the code
# The answer file contains all of the variables specific to the application being backed up
# sourcing the answerfile executes each line as if it were called in the script directly
source $1

# these variables are built by using the dynamic variables assigned in the answerfile
ARCHIVE=/automation/$APPLICATION/$LOCALHOST
ERRORLOG=/automation/$APPLICATION/errorlog.txt

SQLFILE=/automation/$APPLICATION/$APPLICATION.sql
WEBFILE=/automation/$APPLICATION/$APPLICATION.web.tar

SQLGZ=/automation/$APPLICATION/$APPLICATION.sql.gz
WEBGZ=/automation/$APPLICATION/$APPLICATION.web.tar.gz

SQLA=/automation/$APPLICATION/$LOCALHOST/$APPLICATION.sql.gz
WEBA=/automation/$APPLICATION/$LOCALHOST/$APPLICATION.web.tar.gz

# clean the slate - remove any files left over from previous attempts to backup the files
rm $SQLFILE
rm $WEBFILE
rm $SQLGZ
rm $WEBGZ

# stamp the error log, collect the mysql and web data
echo "Backup started: " `date` >> $ERRORLOG
mysqldump -u $SQLUSER $SQLPASS $SQLDB > $SQLFILE
tar -cf $WEBFILE $WEBDATA

# verify the files were created then use gzip to compress them
if [ -s $SQLFILE ];
then
    if [ -s $WEBFILE ];
    then
       gzip $SQLFILE
       gzip $WEBFILE
    else
       echo $SQLFILE was created, $WEBFILE was not, script terminating. >> $ERRORLOG
       exit
    fi
else
    echo $SQLFILE was not created, script terminating. >> $ERRORLOG
    exit
fi

# verify the files were compressed
# remove the files from last week and move the ones from this week to the archive
if [ -s $SQLGZ ];
then
    if [ -s $WEBGZ ];
    then
       rm $ARCHIVE/*.$TODAY
       mv $SQLGZ $SQLA.$TODAY
       mv $WEBGZ $WEBA.$TODAY
    else
       echo $SQLGZ was created, $WEBGZ was not, script terminating. >> $ERRORLOG
       exit
    fi
else
    echo $SQLGZ was not created, script terminating. >> $ERRORLOG
    exit
fi

# copy the files to the remote host
# verify the permissions on the local automation folder are correct
# stamp the error log to complete the job
scp $SQLA.$TODAY edcns5al@$REMOTEHOST.domain.com:/automation/$APPLICATION/$LOCALHOST
scp $WEBA.$TODAY edcns5al@$REMOTEHOST.domain.com:/automation/$APPLICATION/$LOCALHOST

chown -R root:automation /automation/
chmod -R 770 /automation/
echo Backup completed: `date`  >> $ERRORLOG
exit

I am still interested in using sed in a function as you suggested thekashyap, I just need help finding out why I am getting that error. I don't know where I might use it yet, but I'd like to see it work for my own knowledge.

Thank you both very much for your help!

I can use the variables as if I had set them in the script.sh file directly. The answer file does not appear to require the #!/bin/bash, should I include it anyway?

sourcing is much like including in C/C++ if you are familiar with them or import/require in python or ruby. In this case you get an exact copy of the contents of the file you source as if you had typed it in your file originally. So, if the sourced file contains a #!/bin/bash it will only appear as a comment when sourced in your file (thus causing no damage while providing no benefit).

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.