Applying PSU Patch on Two Node RAC GRID Oracle Home.

Step-1
Take the backup of Grid Home
cd $GRID_HOME
tar -cvpf /u01/app/11.2.0/grid/gridhome_bkp.tar ./
Perform this Backup on Both the RAC Nodes
Step-2
Download  & Copy Latest OPatch
OPatch is the utility used to apply the patch. Download the latest OPatch utility from My Oracle Support patch 6880880. As software home owner, unzip the downloaded OPatch to the $GRID_HOME and $ORACLE_HOME

Take Backup of Original OPatch
Mv OPatch OPatch_bk
Now test for new opatch version

Step-3
OCM Configuration
During patch apply, OPatch will prompt for the location of the OCM (Oracle Configuration Manager) response file. If you do not have one, run the following command to create it.
cd /u01/app/11.2.0/grid/OPatch/ocm/bin
 
Make a note of the response file location as we will need it. Or copy this ocm.rsp /tmp
Step-4
Validation of Oracle Inventory
 Before patch application, let check the consistency of the oracle inventory. If the below command succeeds, we are good.
 > $GRID_HOME/OPatch/opatch lsinventory -detail -oh $GRID_HOME
   








One-off Patch Conflict Detection and Resolution
 Check for conflicting patches by doing the following for each GI home you are planning to apply the patch on.
 [grid@rac1 db_grid_psu]$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./16902043
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
PREREQ session
Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2013-12-09_17-43-37PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
[grid@rac1 db_grid_psu]$ 


Patch Installation  Note: -- Stop EM agent if it is running on your server before you apply the patch.  

emctl stop agent  

[root@rac1 db_grid_psu]# /u01/app/11.2.0/grid/OPatch/opatch auto /tmp/db_grid_psu/ -oh /u01/app/11.2.0/grid [root@rac1 ~]# /u01/app/11.2.0/grid/OPatch/opatch auto /tmp/db_grid_psu/ -oh /u01/app/11.2.0/grid/    [root@rac1 ~]# /u01/app/11.2.0/grid/OPatch/opatch auto /tmp/db_grid_psu/ -oh /u01/app/11.2.0/grid/ Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /tmp -patchn db_grid_psu -oh /u01/app/11.2.0/grid/ -paramfile /u01/app/11.2.0/grid/crs/install/crsconfig_params /u01/app/11.2.0/grid/crs/install/crsconfig_params /u01/app/11.2.0/grid/crs/install/s_crsconfig_defs This is the main log file: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto2013-12-09_19-13-07.log This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto2013-12-09_19-13-07.report.log 2013-12-09 19:13:07: Starting Clusterware Patch Setup Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Enter 'yes' if you have unzipped this patch to an empty directory to proceed  (yes/no):yes OPatch  is bundled with OCM, Enter the absolute OCM response file path: /tmp/ocm.rsp CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1' CRS-2673: Attempting to stop 'ora.crsd' on 'rac1' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1' CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac1' CRS-2673: Attempting to stop 'ora.prod.db' on 'rac1' CRS-2673: Attempting to stop 'ora.cvu' on 'rac1' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1' CRS-2673: Attempting to stop 'ora.oc4j' on 'rac1' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1' CRS-2677: Stop of 'ora.cvu' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.cvu' on 'rac1' CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1' CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1' CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac1' CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac1' CRS-2676: Start of 'ora.scan1.vip' on 'rac1' succeeded CRS-2677: Stop of 'ora.prod.db' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1' CRS-2676: Start of 'ora.rac1.vip' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac1' CRS-2676: Start of 'ora.cvu' on 'rac1' succeeded CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded CRS-2677: Stop of 'ora.oc4j' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.oc4j' on 'rac1' CRS-2677: Stop of 'ora.FRA.dg' on 'rac1' succeeded CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rac1' CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded CRS-2676: Start of 'ora.oc4j' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.ons' on 'rac1' CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1' CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1' CRS-2673: Attempting to stop 'ora.evmd' on 'rac1' CRS-2673: Attempting to stop 'ora.asm' on 'rac1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1' CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rac1' CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.crf' on 'rac1' CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1' CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1' CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed CRS-4133: Oracle High Availability Services has been stopped. Successfully unlock /u01/app/11.2.0/grid patch /tmp/db_grid_psu/16902043  apply successful for home  /u01/app/11.2.0/grid CRS-4123: Oracle High Availability Services has been started.  Do the same on rac2 node as well. Before applying PSU patch copy new opatch zip file to GRID Oracle Home  and take the backup of existing OPatch Directory and unzip the p6880880_11200*.zip file under Grid Oracle Home once it is done.  [root@rac2 ~]# /u01/app/11.2.0/grid/OPatch/opatch auto /tmp/db_grid_psu/ -oh /u01/app/11.2.0/grid/ Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/a