RMAN “archival backup” – part 2

This will be a quite small follow-up to “part 1” and will almost entirely cover bash scripting details (no rocket science here!)

TOC is:

1. backups will be scheduled in an external software suite: we must provide shell script to run level 0 backups, incremental backups, archivelog backups
2. we want to use the same shell script on different servers which will include Oracle Linux 5,6,7, RHEL 5,6,7, AIX 6.1, AIX 7.1, Sun Solaris 11
3. shell scripts must return with a proper exit code to notify the caller in case of errors
4. shell scripts must both show their output to the caller and save it in a log file (each execution must write its own log to avoid the same log being overwritten)
5. shell scripts must show their output, too
6. shell scripts should work even if their path contains symlinks
7. backup logs must contain any information needed to reinstall Oracle Software and restore the database
8. it would be nice to known which are the shell’s process ancestors0

1) backups will be scheduled in an external software suite: we must provide shell script to run level 0 backups, incremental backups, archivelog backups

This should be an obvious task.

2) we want to use the same shell script on different servers which will include Oracle Linux 5,6,7, RHEL 5,6,7, AIX 6.1, AIX 7.1, Sun Solaris 11

Some standard unix command behave differently on different *nix system (awk, hostname, bash are some of these commands). I used the output of $( uname ) to use different commands and I put all this stuff in a single script to keed code neat.

3) shell scripts must return with a proper exit code to notify the caller in case of errors

This can be achieved running something like one of the followings, depending whether you need to inspect logfile content or not

rman target / [...] cmdfile=[...]
RMAN_RETCODE=$?
[...]
exit ${RMAN_RETCODE}

rman target / [...] cmdfile=[...]
RETCODE=$?

# check whether the log file contains RMAN-xxxxx messages, even if rman's retcode looks fine

if [[ ${RETCODE} -eq 0 ]] ; then
echo
  echo "looking for ^RMAN-xxxxx messages in log file..."
  echo
  egrep -q "^RMAN-([0-9]){5}" ${LOG_FILE}
  if [[ $? -eq 0 ]] ; then
    RETCODE=2
    echo "ERROR(s) found:"
    egrep "^RMAN-([0-9]){5}" ${LOG_FILE}
    echo
  else
    echo "no RMAN-xxxxx messages in log file"
  fi
fi


# check whether my own log shows EXPIRED backup; if this is the case set
# RETCODE to a non-zero value

egrep -q "^validation failed for archived log|^crosschecked backup piece: found to be 'EXPIRED'" ${LOG_FILE}

if [[ $? -eq 0 ]] ; then
  echo ERROR found EXPIRED backup or MISSING archivelog
  RETCODE=3
fi

[...]

exit ${RETCODE}

4-5) shell scripts must both show their output to the caller and save it in a log file (each execution must write its own log to avoid the same log being overwritten) and they must show their output, too

Showing stdout and saving it in a file can be done with tee , but I found using tee with both stdout and stderr a bit more tricky. On Linux you can done this from bash leveraging process substitution with & redirection and exec like this:

exec &> >(tee -a "${LOG_FILE}")

while on different *nix system I used to a simpler approach witch does not show anything real time to the caller process:

if [[ $( uname ) == "Linux" ]] ; then
  [...]
else
  exec 6>&1 ## save stdout linking it to file desriptor 6
  exec 7>&2 ## save stderr linking it to file desriptor 7
  exec >> ${LOG_FILE} ## redirects all stdout to ${LOG_FILE}
  exec 2>&1 ## redirects all stderr to stdout
fi

[...]

#end of the script:
# restore stdout and stderr file descriptors

if [[ $( uname ) != "Linux" ]]; then
  exec 1>&6 6>&- ## restore stdout and close file descriptor 6
  exec 2>&7 7>&- ## restore stderr and close file descriptor 7
  cat ${LOG_FILE}
fi

exit

To have separate log files for different executions (this is really fundamental to debug overlapping executions!) I build the log file name for script.sh as script.DATE_HOUR.PID.log

6) scripts should work even if their path contains symlinks

This should be working out-of-the-box and will allow for more flexibility

7) backup logs must contain any information needed to reinstall Oracle Software and restore the database

In a better world, any installed O.S. and software component should be documented and backed up, including OS version, ORACLE_HOME contents and ORACLE_HOME’s inventory. However it could be useful to know exactly which OS, Oracle version, Oracle database component(s) and patch(es) have been installed.

The print* scripts you can find in https://github.com/yawod/rman-archival-backup/tree/master/lib will show such informations.

uname and uname -a will tell witch OS and kernel we are using.

To know which Oracle Database Software we are using I choosed a multiple approach:

1. ${ORACLE_HOME}/bin/sqlplus will tell you which release we are on
2. ${ORACLE_HOME}/OPatch/opatch lspatches willl tell us which patches we installed
3. On some *nix systems, ar -t ${ORACLE_HOME}/rdbms/lib/libknlopt.a can be used to see which components have been installed or linked (i.e. ASM, ConText, Database Vault, OLAP, RAC, Unified Auditing, …)
4. dba_registry can always show which database components are configured in this database (i.e. OJVM, ConText, OLAP, ApEx, …)
5. dba_registry_sqlpatch and dba_registry_history can show us which patches have been installed, too

the print_oracle_home_info.sh script will use all of these to give us any information needed to reinstall a fully compatible database server.

8) it would be nice to known which are the shell’s process ancestors

Knowing a shell’s process ancestors can be useful to check whether it has been started by an interactive shell, a crond job, a scheduler agent… moreover, we could match such informations with audit data to know “how did what”.

I got this with a bash recursive function which use ps to show current process parents up to PID 1:

echo
echo current process tree follows:

print_ps_tree()
{
if [[ ${1} -eq 1 ]]; then
  ps -fp 1
else
  print_ps_tree $( ps -fp $1 | tail -n 1 | tr -s [:space:]| cut -d " " -f 3 )
  ps -fp $1 | tail -n 1
fi;
}

print_ps_tree $$

That’s all, folks!