Categories
Blog

Database Patching issues and resolutions that I have experienced

These are some of the issues I have experienced when patching Oracle databases.
Issue A
You have stopped the databases,ASM and listener but when you run the opatch auto you may see the following in the log:

patch /source/kiran_oracle_patch_oct2015/Grid/20996923/20760982 apply failed for home /u01/app/oracle/product/11.2.0/dbhome_1

If you look in the log, it shows :

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.

The details are:

Following executables are active :
/u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Check what process is locking the file :

$./fuser /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1
/u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1: 6041m
$ ps -ef | grep 6041
oracle 6041 1 0 2015 ? 00:03:57 /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr LISTENER -inherit
oracle 13806 31703 0 11:53 pts/1 00:00:00 grep 6041

a) Solution
It’s the listener, stop it and rerun the patch apply, In some cases it could be different file locking the patching, you can follow the same process to identify the lock and stop/shutdown the process that is locking the patching.
Issue B
Once all the patching has been completed, you try to start the ASM you get the ORA-29701: unable to connect to Cluster Synchronization Service:

SQL> startup
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service
SQL> exit

If you do a crsctl stat res -t, you can see the cssd is offline.

$ ./crsctl stat res -t
----------------------------------------------------------------------------
NAME TARGET STATE SERVER
--------------------------------------------------------------------------------
ora.DATA.dg ONLINE OFFLINE kvt25678
ora.FRA.dg ONLINE OFFLINE kvt25678
ora.LISTENER.lsnr ONLINE ONLINE kvt25678
ora.asm OFFLINE OFFLINE kvt25678 <<<<<<<<<<<
ora.ons OFFLINE OFFLINE kvt25678
ora.cssd OFFLINE OFFLINE kvt25678
ora.diskmon OFFLINE OFFLINE

b) Solution
Start the ASM using ‘srvctl start asm’