How to Apply PSU Patch in Oracle and Patch Apply Steps

Hi,

I will explain How to Apply PSU Patch in Oracle and Patch Apply Steps in this post. I will show you How to Apply PSU Patch in Oracle and Patch Apply Steps on Oracle 11g R2 RAC Database on IBM AIX.

 

 

Oracle releases Patches (Patch) due to errors and bugs in every version of database it releases. By applying these Patches, errors or bugs are fixed. On the other hand, Oracle routinely releases Patches which we call PSU (Patch Set Update). These Patches contain many sub-Patches, and when you apply these PSUs, all related bugs and bugs will be resolved.

 

As a DBA, I recommend applying the PSUs that are released every 3- 6 months or once a year. Thus, Proactive solution will be provided in case of errors and bugs that may encounter later.

 

I will explain the steps of 19955028 Patch Apply with the Rolling Patch (no Downtime) method to the 11.2.0.4 RAC database.

 

Steps of the Patch apply process are as follows.

  • Download the related patch from Oracle support.
  • Download the latest OPatch Utility from Oracle support otherwise you may get an error for OPatch utility.
  • Stop and Shutdown EM Agent,Listener and DB Instance on Node 1.
  • Apply the Patch on Node 1 and startup Node 1.
  • Stop and Shutdown EM Agent,Listener and DB Instance on Node 2.
  • Apply the Patch on Node 2 and startup Node 2.

 

Step -1: Check Cluster and RAC Services before patch as follows.

 

oracle:deveci01:/home/users/oracle:> crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    dev...eci01 
ora....ER.lsnr ora....er.type ONLINE    ONLINE    dev...eci01 
ora....N1.lsnr ora....er.type ONLINE    ONLINE    dev...eci01 
.......................
ora....ry.acfs ora....fs.type ONLINE    ONLINE    dev...eci01 
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    dev...eci01 
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    dev...eci02 
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    dev...eci02 


Check the Database Registry history as follows.

oracle:deveci01:/home/users/oracle:>. profile.11g.db
oracle:deveci01:/home/users/oracle:>sqlplus / as sysdba

SQL> set lines 1000
SQL> select * from dba_registry_history;

ACTION_TIME ACTION NAMESPACE VERSION ID BUNDLE_SERIES COMMENTS
--------------------------------------------------------------------------- ------------------------------ ------------------------------ ------------------------------ ---------- ------------------------------ ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
04-JUL-14 01.58.39.987161 AM APPLY SERVER 11.2.0.4 0 PSU Patchset 11.2.0.2.0

SQL> exit

 

 

Check the Patch history that have been applied before.

 

oracle:deveci01:/home/users/oracle:>. profile.11g.asm
oracle:deveci01:/home/users/oracle:>opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.

Oracle Home : /oracle/appgrid/product/11.2.0.4/grid
Central Inventory : /oracle/app/oraInventory
from : /oracle/appgrid/product/11.2.0.4/grid/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.4.0
Log file location : /oracle/appgrid/product/11.2.0.4/grid/cfgtoollogs/opatch/opatch2015-02-24_09-55-37AM_1.log

Lsinventory Output file location : /oracle/appgrid/product/11.2.0.4/grid/cfgtoollogs/opatch/lsinv/lsinventory2015-02-24_09-55-37AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Grid Infrastructure 11g 11.2.0.4.0
There are 1 product(s) installed in this Oracle Home.

Interim patches (1) :

Patch 18031668 : applied on Fri Jul 04 02:17:50 EEST 2014
Unique Patch ID: 17308007
Patch description: "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Created on 24 Feb 2014, 21:39:00 hrs PST8PDT
Sub-patch 17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
Bugs fixed:
16929165, 17239687, 17602269, 16220077, 17205719, 17465741, 17235750
16721594, 17071721, 17265217, 17441661, 16450169, 17389192, 17468141
17761775, 17501491, 14133975, 17622427, 16399083, 16837842, 17716305
18139690, 16956380, 17288409, 14852021, 18203837, 17313525, 12905058
17446237, 18203838, 18203835, 16618694, 17375354, 17546761, 17721717
17344412, 17050888, 17341326, 17546973, 17551709, 17393683, 17614227
17726838, 16285691, 16472716, 17346671, 17545847, 16228604, 13364795
17040764, 16912439, 17811438, 17323222, 16180763, 17016369, 17325413
17088068, 16785708, 17865671, 17612828, 10136473, 18180390, 16850630
16613964, 14010183, 17811447, 17296856, 17478514, 16875449, 16069901
17390431, 17610798, 17385178, 17443671, 17082359, 17783588, 17811429
16043574, 18094246, 13866822, 16863422, 17027426, 16731148, 17080436
17186905, 16943711, 17437634, 17394950, 18031668, 14338435, 13944971
17752121, 17284817, 17811456, 17332800

Rac system comprising of multiple nodes
Local node = deveci01
Remote node = deveci02

--------------------------------------------------------------------------------

OPatch succeeded.

 

 

 

 

Step -2: Create ocm.rsp using the emocmrsp for the patch as follows.

oracle:deveci01:/home/users/oracle:>cd /oracle/Patch/
oracle:deveci01:/oracle/Patch:>
oracle:deveci01:/oracle/Patch:>/oracle/appgrid/product/11.2.0.4/OPatch/ocm/bin/emocmrsp 
OCM Installation Response Generator 10.3.4.0.0 - Production
Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.

Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
The OCM configuration response file (ocm.rsp) was successfully created.
oracle:deveci01:/oracle/Patch:>ls -all
total 1028536
drwxr-xr-x 2 oracle oinstall 256 Feb 24 09:59 .
drwxr-xr-x 12 oracle oinstall 4096 Dec 01 08:45 ..
-rw-r--r-- 1 oracle oinstall 623 Feb 24 09:59 ocm.rsp
-rw-r--r-- 1 oracle oinstall 526596210 Feb 24 09:59 p19955028_112040_AIX64-5L.zip

 

 

Step -3: Stop and Shutdown EM Agent,Listener and DB Instance on Node 1.

oracle:deveci01:/home/users/oracle:>. profile.11g.db 
oracle:deveci01:/home/users/oracle:>
oracle:deveci01:/home/users/oracle:>ps -ef |grep smon
root 11731186 1 18 10:05:48 - 0:00 /oracle/appgrid/product/11.2.0.4/grid/bin/osysmond.bin
oracle 12058826 1 0 10:06:56 - 0:00 ora_smon_MSDB1
oracle 12976246 1 0 10:06:24 - 0:00 asm_smon_+ASM1
oracle 38338572 28049424 1 10:07:02 pts/0 0:00 grep smon
oracle:deveci01:/home/users/oracle:>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Tue Feb 24 10:07:12 2015

Copyright (c) 1982, 2013, Oracle. All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> shu immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
oracle:deveci01:/home/users/oracle:>

oracle:deveci01:/home/users/oracle:>. profile.11g.asm 
oracle:deveci01:/home/users/oracle:>
oracle:deveci01:/home/users/oracle:>lsnrctl stop

LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.4.0 - Production on 24-FEB-2015 10:07:49

Copyright (c) 1991, 2013, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
The command completed successfully
oracle:deveci01:/home/users/oracle:>
oracle:deveci01:/home/users/oracle:>
oracle:deveci01:/home/users/oracle:>lsnrctl status

LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.4.0 - Production on 24-FEB-2015 10:07:53

Copyright (c) 1991, 2013, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
IBM/AIX RISC System/6000 Error: 2: No such file or directory
oracle:deveci01:/home/users/oracle:>



Especially for IBM AIX environments, TFA (Trace File Analyzer) is also stopped as follows.

root:deveci01:/root:> /etc/init.tfa stop

 

 

Step -4:  Apply the related PSU patch as follows. Firstly run the following commands for IBM AIX

root:deveci01:/root:> /usr/sbin/slibclean 

root:deveci01:/root:>/usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle



Then set the Oracle Home and Path, start the apply patch using opatch auto as follows.

 

root:deveci01:/root:>export ORACLE_HOME=/oracle/appgrid/product/11.2.0.4
root:deveci01:/root:>export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH

root:deveci01:/root:> opatch auto /oracle/Patch/22646198 -ocmrf /oracle/Patch/ocm.rsp
Executing /oracle/appgrid/product/11.2.0.4/grid/perl/bin/perl /oracle/appgrid/product/11.2.0.4/grid/OPatch/crs/patch11203.pl -patchdir /oracle/Patch -patchn 19955028 -ocmrf /oracle/Patch/ocm.rsp -paramfile /oracle/appgrid/product/11.2.0.4/grid/crs/install/crsconfig_params

This is the main log file: /oracle/appgrid/product/11.2.0.4/grid/cfgtoollogs/opatchauto2015-02-24_10-08-12.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/oracle/appgrid/product/11.2.0.4/grid/cfgtoollogs/opatchauto2015-02-24_10-08-12.report.log

2015-02-24 10:08:12: Starting Clusterware Patch Setup
Using configuration parameter file: /oracle/appgrid/product/11.2.0.4/grid/crs/install/crsconfig_params

Stopping RAC /oracle/appora/product/11.2.0.4/db ...
Stopped RAC /oracle/appora/product/11.2.0.4/db successfully

patch /oracle/Patch/19955028/19769489 apply successful for home /oracle/appora/product/11.2.0.4/db
patch /oracle/Patch/19955028/19769476/custom/server/19769476 apply successful for home /oracle/appora/product/11.2.0.4/db

Stopping CRS...
Stopped CRS successfully

patch /oracle/Patch/19955028/19769489 apply successful for home /oracle/appgrid/product/11.2.0.4/grid
patch /oracle/Patch/19955028/19769476 apply failed for home /oracle/appgrid/product/11.2.0.4/grid

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /oracle/appora/product/11.2.0.4/db ...
Started RAC /oracle/appora/product/11.2.0.4/db successfully

opatch auto succeeded.
root:deveci01:/root:>

 

Repeat the Step-3 and Step-4 for the Second or other nodes.

Once you completed the Step-3 and Step-4 on Second node, you should perform the following Post Patch steps.

 

 

 

Run the catbundle.sql on node 1 as follows.

 

oracle:deveci01:/home/users/oracle:>cd $ORACLE_HOME/rdbms/admin
oracle:deveci01:/oracle/appora/product/11.2.0.4/db/rdbms/admin:>sqlplus /nolog

SQL*Plus: Release 11.2.0.4.0 Production on Tue Feb 24 10:28:58 2015

Copyright (c) 1982, 2013, Oracle. All rights reserved.

SQL> CONNECT / AS SYSDBA
Connected to an idle instance.
SQL> STARTUP
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 8551575552 bytes
Fixed Size 2263488 bytes
Variable Size 1744832064 bytes
Database Buffers 6777995264 bytes
Redo Buffers 26484736 bytes
Database mounted.
Database opened.
SQL> @catbundle.sql psu apply

PL/SQL procedure successfully completed.


Function created.

 

Now Check the Patch if it is applied or not with the opatch lsinventory command.

 

oracle:deveci01:/home/users/oracle:> opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.


Oracle Home : /oracle/appgrid/product/11.2.0.4/grid
Central Inventory : /oracle/app/oraInventory
from : /oracle/appgrid/product/11.2.0.4/grid/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.4.0
Log file location : /oracle/appgrid/product/11.2.0.4/grid/cfgtoollogs/opatch/opatch2015-02-24_10-25-12AM_1.log

Lsinventory Output file location : /oracle/appgrid/product/11.2.0.4/grid/cfgtoollogs/opatch/lsinv/lsinventory2015-02-24_10-25-12AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Grid Infrastructure 11g 11.2.0.4.0
There are 1 product(s) installed in this Oracle Home.


Interim patches (1) :

Patch 19769489 : applied on Tue Feb 24 10:16:16 EET 2015
Unique Patch ID: 18280822
Patch description: "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Created on 30 Dec 2014, 03:35:13 hrs PST8PDT
Sub-patch 19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch 18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch 18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch 17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
Bugs fixed:
16929165, 16220077, 17235750, 18436647, 18191164, 17468141, 17006183
17571039, 16315398, 17501491, 16934803, 13955826, 17274537, 17288409
12905058, 17446237, 17375354, 16992075, 16855292, 18273830, 18282562
18828868, 17922254, 15990359, 17050888, 19680952, 17546973, 18554871
17726838, 17614134, 19827973, 9756271, 16422541, 18673325, 17232014
17227277, 13853126, 17545847, 17390160, 19769489, 18096714, 12747740
18334586, 18315328, 17016369, 17786518, 19271443, 10136473, 16785708
17215560, 18135678, 17311728, 17302277, 18018515, 17610798, 16268425
18293054, 18280813, 17082359, 14245531, 17783588, 16538760, 13866822
18094246, 17477958, 16943711, 18673304, 18031668, 19463897, 19463893
19211724, 13498382, 12364061, 16450169, 17397545, 17786278, 17767676
14458214, 19289642, 17622427, 17824637, 18482502, 17716305, 16399083
18674047, 18744139, 14852021, 19727057, 17242746, 16777840, 17174582
17258090, 18277454, 13645875, 19006849, 14084247, 17551709, 17393683
17614227, 17705023, 17835627, 17883081, 19854503, 16042673, 18996843
16285691, 17393915, 19972570, 20074391, 16228604, 17655634, 17596908
17600719, 17323222, 17754782, 16180763, 18264060, 17325413, 19730508
17343514, 18471685, 17865671, 17308789, 18098207, 16613964, 17811447
16069901, 19197175, 17390431, 19615136, 16494615, 19972564, 16043574
17282229, 19972566, 17006570, 19972568, 19972569, 17027426, 14602788
18673342, 17080436, 17186905, 17011832, 17394950, 13944971, 17752121
17811456, 17284817, 17238511, 14354737, 17239687, 17042658, 14764829
17602269, 17672719, 15913355, 17891946, 18262334, 17205719, 16721594
17071721, 14565184, 17265217, 17389192, 17761775, 16360112, 17982555
12816846, 17842825, 19121551, 14106803, 16837842, 17891943, 18139690
17313525, 18203837, 18964939, 11883252, 18203838, 18203835, 18436307
17546761, 17721717, 17344412, 18228645, 18641451, 18614015, 16344544
17787259, 16472716, 19309466, 18306996, 18235390, 17588480, 17346671
18674024, 13364795, 14657740, 11733603, 18641461, 17811438, 19466309
17040527, 17037130, 17088068, 17612828, 18180390, 17449815, 19458377
19554106, 18973907, 18230522, 19544839, 18244962, 17811429, 17237521
17951233, 16863422, 17752995, 16392068, 18339044, 17437634, 14338435
17332800, 13609098, 18199537, 17465741, 14034426, 17441661, 18522509
18061914, 14133975, 17570240, 16692232, 18247991, 18328509, 16956380
17587063, 19049453, 16618694, 17586955, 17851160, 18009564, 14285317
16542886, 17341326, 17571306, 17036973, 18641419, 18747196, 16833527
16524926, 17478145, 17297939, 17299889, 17040764, 18202441, 16912439
15861775, 14054676, 16595641, 17912217, 13951456, 17267114, 17025461
16850630, 17648596, 14010183, 17296856, 17478514, 16875449, 18681862
18356166, 17936109, 17443671, 14829250, 17385178, 18189036, 18554763
16901385, 13558557, 18508861, 17835048, 18441944, 16314254, 16731148
17165204, 15979965, 19584068, 17381384, 18092127, 20142975, 17892268
17184721, 16198143

Rac system comprising of multiple nodes
Local node = deveci01
Remote node = deveci02

--------------------------------------------------------------------------------

OPatch succeeded.




Check the Patch if it is applied on DBA Registry history as follows.

 

oracle:deveci01:/oracle/appora/product/11.2.0.4/db/rdbms/admin:>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Tue Feb 24 10:30:30 2015

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> set lines 1000
select * from dba_registry_history;

ACTION_TIME ACTION NAMESPACE VERSION ID BUNDLE_SERIES COMMENTS
--------------------------------------------------------------------------- ------------------------------ ------------------------------ ------------------------------ ---------- ------------------------------ ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
04-JUL-14 01.58.39.987161 AM APPLY SERVER 11.2.0.4 0 PSU Patchset 11.2.0.2.0
24-FEB-15 10.30.11.962660 AM APPLY SERVER 11.2.0.4 5 PSU PSU 11.2.0.4.5

SQL>

Patch Apply steps are changed on Oracle 12c and above ( Oracle 18c, 19c and 20c ), I have explained the new version patch apply steps at the following post.

You can read the following post to learn how to Patch Apply in Oracle 18c on Exadata.

Oracle Grid Infrastructure & RAC Upgrade from Oracle 12c to Oracle 18c in Full Rack Exadata X7 Step by Step -1 Patch 27006180 Apply and 24600431 Patch Rollback 

 

 

Do you want to learn Oracle Database for Beginners, then read the following articles.

https://ittutorial.org/oracle-database-19c-tutorials-for-beginners/

 707 views last month,  2 views today

About Mehmet Salih Deveci

I am Founder of SysDBASoft IT and IT Tutorial and Certified Expert about Oracle & SQL Server database, Goldengate, Exadata Machine, Oracle Database Appliance administrator with 10+years experience.I have OCA, OCP, OCE RAC Expert Certificates I have worked 100+ Banking, Insurance, Finance, Telco and etc. clients as a Consultant, Insource or Outsource.I have done 200+ Operations in this clients such as Exadata Installation & PoC & Migration & Upgrade, Oracle & SQL Server Database Upgrade, Oracle RAC Installation, SQL Server AlwaysOn Installation, Database Migration, Disaster Recovery, Backup Restore, Performance Tuning, Periodic Healthchecks.I have done 2000+ Table replication with Goldengate or SQL Server Replication tool for DWH Databases in many clients.If you need Oracle DBA, SQL Server DBA, APPS DBA,  Exadata, Goldengate, EBS Consultancy and Training you can send my email adress mehmetsalih.deveci@outlook.com.-                                                                                                                                                                                                                                                 -Oracle DBA, SQL Server DBA, APPS DBA,  Exadata, Goldengate, EBS ve linux Danışmanlık ve Eğitim için  mehmetsalih.deveci@outlook.com a mail atabilirsiniz.

Leave a Reply