PRCR-1013 and PRCR-1064 and ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr]

Main culprit error here is “ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr],”

Solution:

 

oracle@rac2 ~]$ . oraenv
ORACLE_SID = [oracle] ? dev2
ORACLE_HOME = [/home/oracle] ? /raczone/db_home1
The Oracle base for ORACLE_HOME=/raczone/db_home1 is /raczone/11.2.0
[oracle@rac2 ~]$ cd /raczone/db_home1/bin/ 
[oracle@rac2 bin]$ ./sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.1.0 Production on Thu Jan 17 13:38:31 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount
ORACLE instance started.

Total System Global Area  790941696 bytes
Fixed Size                  1339428 bytes
Variable Size             583012316 bytes
Database Buffers          192937984 bytes
Redo Buffers               13651968 bytes
Database mounted.
SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
[oracle@rac2 bin]$

 

root@rac1 bin]# ./srvctl status database -d dev

Instance dev1 is not running on node rac1
Instance dev2 is not running on node rac2
[root@rac1 bin]# ./srvctl start instance -n rac2 -d dev
[root@rac1 bin]# ./srvctl status database -d dev
Instance dev1 is not running on node rac1
Instance dev2 is running on node rac2
[root@rac1 bin]# ./srvctl start instance -n rac1 -d dev
[root@rac1 bin]# ./srvctl status database -d dev
Instance dev1 is running on node rac1
Instance dev2 is running on node rac2
[root@rac1 bin]#

 

PROC-26: Error while accessing the physical storage ASM error AMDU-00204: AMDU-00201:

Hi,

Today, I was facing problem in 11gR2 RAC, after server got restarted, one of the node was not able to startup the clusterware services.When I check OCRCHECK, i got a message:

[root@rac1 bin]# ./ocrcheck

PROT-602: Failed to retrieve data from the cluster registry

PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep=204, loc=kgfokge

AMDU-00204: Disk N0003 is in currently mounted diskgroup CRS

AMDU-00201: Disk N0003: ‘ORCL:VOTE1’

[root@rac1 bin]# ./crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

[root@rac1 bin]# ./crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

But on the another node all the services were successfully running

So ,I tried to find out any error in OCSSD.log,CRSD.log :

 

Getting an error in ocssd.log

2014-01-29 21:10:13.288: [    CSSD][2997357456]clssgmEvtInformation: reqtype (11) cmProc (0x9d02110) client ((nil))
2014-01-29 21:10:13.288: [    CSSD][2997357456]clssgmEvtInformation: reqtype (11) req (0x9c1aa70)
2014-01-29 21:10:13.288: [    CSSD][2997357456]clssnmQueueNotification: type (11) 0x9c1aa70
2014-01-29 21:10:13.999: [    GPnP][3046368960]clsgpnpm_newWiredMsg: [at clsgpnpm.c:633] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri “http://www.grid-pnp.org/2005/12/gpnp-errors#”%5D
2014-01-29 21:10:13.999: [    GPnP][3046368960]clsgpnpm_newWiredMsg: [at clsgpnpm.c:663] Result: (31) CLSGPNP_CALL_AGAIN. Received gpnp meg-reply 0x9cebdb0 with soap fault, reason ‘Operation returned Retry (error CLSGPNP_CALL_AGAIN)'(10) 10
2014-01-29 21:10:13.999: [    GPnP][3046368960]clsgpnpm_receiveMsg: [at clsgpnpm.c:2954] Result: (31) CLSGPNP_CALL_AGAIN. error in gpnp-msg from ‘ipc’ buf=0x9ce9af8
2014-01-29 21:10:13.999: [    GPnP][3046368960]clsgpnpm_exchange: [at clsgpnpm.c:1198] Result: (31) CLSGPNP_CALL_AGAIN. Error soap response from url “ipc://GPNPD_rac1”,msg=0x9cebdb0 rdom=0x9d58c40 try=1/500
@

crsd.log file

{code}
2014-02-03 11:22:40.021: [    GPnP][3046098640]clsgpnpwu_walletfopen: [at clsgpnpwu.c:494] Opened SSO wallet: ‘/raczone/grid_home/gpnp/rac1/wallets/peer/cwallet.sso’
2014-02-03 11:22:40.021: [    GPnP][3046098640]clsgpnp_getCK: [at clsgpnp0.c:1965] Result: (0) CLSGPNP_OK. Get gpnp wallet – provider 1 of 2 (LSKP-FSW(1))
2014-02-03 11:22:40.021: [    GPnP][3046098640]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2014-02-03 11:22:40.021: [    GPnP][3046098640]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=5545, tl=3, f=0

2014-02-03 11:22:40.058: [GIPCXCPT][3046098640] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)

2014-02-03 11:22:40.060: [GIPCXCPT][3046098640] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)

2014-02-03 11:22:40.069: [  OCRASM][3046098640]proprasmo: Error in open/create file in dg [CRS]

[  OCRASM][3046098640]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

ORA-15077: could not locate ASM instance serving a required diskgroup

2014-02-03 11:22:40.071: [  OCRASM][3046098640]proprasmo: kgfoCheckMount returned [7]

2014-02-03 11:22:40.071: [  OCRASM][3046098640]proprasmo: The ASM instance is down

2014-02-03 11:22:40.071: [  OCRRAW][3046098640]proprioo: Failed to open [+CRS]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.

2014-02-03 11:22:40.071: [  OCRRAW][3046098640]proprioo: No OCR/OLR devices are usable

2014-02-03 11:22:40.071: [  OCRASM][3046098640]proprasmcl: asmhandle is NULL

2014-02-03 11:22:40.071: [  OCRRAW][3046098640]proprinit: Could not open raw device

2014-02-03 11:22:40.071: [  OCRASM][3046098640]proprasmcl: asmhandle is NULL

2014-02-03 11:22:40.072: [  OCRAPI][3046098640]a_init:16!: Backend init unsuccessful : [26]

2014-02-03 11:22:40.072: [  CRSOCR][3046098640] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

ORA-15077: could not locate ASM instance serving a required diskgroup

] [7]

2014-02-03 11:22:40.072: [    CRSD][3046098640][PANIC] CRSD exiting: Could not init OCR, code: 26

2014-02-03 11:22:40.072: [    CRSD][3046098640] Done.
I have also checked using amdu utility:

 

RAC1 were the error is coming up:

[grid@rac1 bin]$ ./asmcmd lsdsk -G DATA

Connected to an idle instance.

[grid@rac1 bin]$ ./amdu -diskstring=”ORCL:*” -dump DATA  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0001 is in currently mounted diskgroup DATA

AMDU-00201: Disk N0001: ‘ORCL:DATA1’

** HEARTBEAT DETECTED **

[grid@rac1 bin]$ ./amdu -diskstring=”ORCL:*” -dump VOTE  -nomap -nodir   | grep HEARTBEAT

AMDU-00210: No disks found in diskgroup VOTE

[grid@rac1 bin]$ ./amdu -diskstring=”ORCL:*” -dump FRA  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0002 is in currently mounted diskgroup FRA

AMDU-00201: Disk N0002: ‘ORCL:FRA1’

** HEARTBEAT DETECTED **

[grid@rac1 bin]$ ./amdu -diskstring=”ORCL:*” -dump CRS  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0003 is in currently mounted diskgroup CRS

AMDU-00201: Disk N0003: ‘ORCL:VOTE1’

** HEARTBEAT DETECTED **

[grid@rac1 bin]$

RAC2 (Here, clusterware is running properly)

ORACLE_SID = [grid] ? +ASM2

The Oracle base for ORACLE_HOME=/raczone/grid_home is /raczone/11.2.0

[grid@rac2 bin]$ ./asmcmd lsdsk -G DATA

Path

ORCL:DATA1

[grid@rac2 bin]$ ./amdu -diskstring=”ORCL:*” -dump DATA  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0001 is in currently mounted diskgroup DATA

AMDU-00201: Disk N0001: ‘ORCL:DATA1’

** HEARTBEAT DETECTED **

[grid@rac2 bin]$

[grid@rac2 bin]$ ./amdu -diskstring=”ORCL:*” -dump VOTE  -nomap -nodir   | grep HEARTBEAT

AMDU-00210: No disks found in diskgroup VOTE

[grid@rac2 bin]$ ./amdu -diskstring=”ORCL:*” -dump CRS  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0003 is in currently mounted diskgroup CRS

AMDU-00201: Disk N0003: ‘ORCL:VOTE1’

** HEARTBEAT DETECTED **

[grid@rac2 bin]$ ./amdu -diskstring=”ORCL:*” -dump FRA  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0002 is in currently mounted diskgroup FRA

AMDU-00201: Disk N0002: ‘ORCL:FRA1’

** HEARTBEAT DETECTED **

So, heartbeat also was not a problem,

When I executed Asmcmd, I got to know that ASM instance was not up,so I tried to startup the instance:

[grid@rac1 bin]$ asmcmd
Connected to an idle instance.

but in rac2, asm instance was up

[grid@rac2 bin]$ asmcmd
ASMCMD> ls
CRS/
DATA/
FRA/

So I have started the instance ,

ASMCMD> startup
ASM instance started

Total System Global Area  284565504 bytes
Fixed Size                  1336036 bytes
Variable Size             258063644 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
ASMCMD> ls
CRS/
DATA/
FRA/
ASMCMD> cd CRS
ASMCMD> ls
scan-cluster/
ASMCMD> cd sc*
ASMCMD> ls
ASMPARAMETERFILE/
OCRFILE/
ASMCMD> cd OCR*
ASMCMD> ls -l
Type     Redund  Striped  Time             Sys  Name
OCRFILE  UNPROT  COARSE   FEB 03 15:00:00  Y    REGISTRY.255.837088345
ASMCMD>
ASMCMD> exit
[grid@rac1 bin]$ ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          3
Total space (kbytes)     :     262120
Used space (kbytes)      :       2800
Available space (kbytes) :     259320
ID                       : 1486492518
Device/File Name         :       +CRS
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

[grid@rac1 bin]$ exit
logout

[root@rac1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac1 bin]# ./crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    rac1
ora.DATA.dg    ora….up.type ONLINE    ONLINE    rac1
ora.FRA.dg     ora….up.type ONLINE    ONLINE    rac1
ora….ER.lsnr ora….er.type ONLINE    ONLINE    rac1
ora….N1.lsnr ora….er.type ONLINE    ONLINE    rac1
ora….N2.lsnr ora….er.type ONLINE    ONLINE    rac2
ora….N3.lsnr ora….er.type ONLINE    ONLINE    rac2
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1
ora.dev.db     ora….se.type ONLINE    ONLINE    rac2
ora.eons       ora.eons.type  ONLINE    ONLINE    rac1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    rac1
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1
ora….SM1.asm application    ONLINE    ONLINE    rac1
ora….C1.lsnr application    ONLINE    ONLINE    rac1
ora.rac1.gsd   application    OFFLINE   OFFLINE
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   ora….t1.type ONLINE    ONLINE    rac1
ora….SM2.asm application    ONLINE    ONLINE    rac2
ora….C2.lsnr application    ONLINE    ONLINE    rac2
ora.rac2.gsd   application    OFFLINE   OFFLINE
ora.rac2.ons   application    ONLINE    ONLINE    rac2
ora.rac2.vip   ora….t1.type ONLINE    ONLINE    rac2
ora….ry.acfs ora….fs.type ONLINE    ONLINE    rac1
ora.scan1.vip  ora….ip.type ONLINE    ONLINE    rac1
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    rac2
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    rac2
[root@rac1 bin]# ./crs_stat -t|grep OFFLINE
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE
ora.rac1.gsd   application    OFFLINE   OFFLINE
ora.rac2.gsd   application    OFFLINE   OFFLINE
[root@rac1 bin]#
[root@rac1 bin]#

And now all the services are now successfully runinng.

 

Relocating SCAN VIP from one node to another node

Here, I am giving you a demonstration of relocation SCAN from one node to another.

In the below section, we can see that 3 SCAN VIP are running .e 1 VIP is running on rac2 and 2 VIP are running on rac1.

[grid@rac2 bin]$ ./srvctl status scan

SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node rac2
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node rac1
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node rac1

 

[grid@rac2 bin]$ ./srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node rac2
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node rac1
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node rac1

So, now lets see the syntax for relocating the scan.

[grid@rac2 bin]$ ./srvctl relocate scan -h

Temporarily relocate SCAN VIP from one node of the cluster to another.

Usage: srvctl relocate scan -i <ordinal_number> [-n <node_name>]
-i <scan_ordinal_number> Ordinal number of IP address
-n <node_name> SCAN VIP node name
-h Print usages

Now, lets try to relocate SCAN VIP 1 to rac1.

[grid@rac2 bin]$ ./srvctl relocate scan -i 1 -n rac1

So, as per the below section we can see that now all the scan listener are runing on rac1.Hence, we can also conclude that it is not necessare that SCAN listener has to be run on every node.It can run on all nodes or on partially some nodes.

[grid@rac2 bin]$ ./srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node rac1
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node rac1
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node rac1
[grid@rac2 bin]$

 
[grid@rac2 bin]$ ./srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node rac1
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node rac1
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node rac1
[grid@rac2 bin]$

Node addition in 11gR2 RAC

Here, we will see a demonstration of adding a node in 11gR2.

[root@rac2 ~]# export DISPLAY=:0.0

[root@rac2 ~]# xhost +

access control disabled, clients can connect from any host

[root@rac2 ~]# su – oracle

[oracle@rac2 ~]$ cd /raczone/db_home1/oui/bin/

[oracle@rac2 bin]$ ./addNode.sh “CLUSTER_NEW_NODES={rac3}”

Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB.   Actual 1599 MB    Passed

Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed

Oracle Universal Installer, Version 11.2.0.1.0 Production

Copyright (C) 1999, 2009, Oracle. All rights reserved.

Performing tests to see whether nodes rac1,rac3 are available

……………………………………………………… 100% Done.

.

—————————————————————————–

Cluster Node Addition Summary

Global Settings

Source: /raczone/db_home1

New Nodes

Space Requirements

New Nodes

rac3

/: Required 3.60GB : Available 28.41GB

Installed Products

Product Names

Oracle Database 11g 11.2.0.1.0

Sun JDK 1.5.0.17.0

Installer SDK Component 11.2.0.1.0

Oracle One-Off Patch Installer 11.2.0.0.2

Oracle Universal Installer 11.2.0.1.0

Oracle Configuration Manager Deconfiguration 10.3.1.0.0

Oracle DBCA Deconfiguration 11.2.0.1.0

Oracle RAC Deconfiguration 11.2.0.1.0

Oracle Database Deconfiguration 11.2.0.1.0

Oracle Configuration Manager 10.3.1.1.0

Oracle ODBC Driverfor Instant Client 11.2.0.1.0

LDAP Required Support Files 11.2.0.1.0

SSL Required Support Files for InstantClient 11.2.0.1.0

Bali Share 1.1.18.0.0

Oracle Extended Windowing Toolkit 3.4.47.0.0

Oracle JFC Extended Windowing Toolkit 4.2.36.0.0

Oracle Real Application Testing 11.2.0.1.0

Oracle Database Vault J2EE Application 11.2.0.1.0

Oracle Label Security 11.2.0.1.0

Oracle Data Mining RDBMS Files 11.2.0.1.0

Oracle OLAP RDBMS Files 11.2.0.1.0

Oracle OLAP API 11.2.0.1.0

Platform Required Support Files 11.2.0.1.0

Oracle Database Vault option 11.2.0.1.0

Oracle RAC Required Support Files-HAS 11.2.0.1.0

SQL*Plus Required Support Files 11.2.0.1.0

Oracle Display Fonts 9.0.2.0.0

Oracle Ice Browser 5.2.3.6.0

Oracle JDBC Server Support Package 11.2.0.1.0

Oracle SQL Developer 11.2.0.1.0

Oracle Application Express 11.2.0.1.0

XDK Required Support Files 11.2.0.1.0

RDBMS Required Support Files for Instant Client 11.2.0.1.0

SQLJ Runtime 11.2.0.1.0

Database Workspace Manager 11.2.0.1.0

RDBMS Required Support Files 11.2.0.1.0

Oracle Globalization Support 11.2.0.1.0

Exadata Storage Server 11.2.0.1.0

Provisioning Advisor Framework 10.2.0.4.2

Enterprise Manager Database Plugin — Repository Support 11.2.0.1.0

Enterprise Manager Repository Core Files 10.2.0.4.2

Enterprise Manager Database Plugin — Agent Support 11.2.0.1.0

Enterprise Manager Grid Control Core Files 10.2.0.4.2

Enterprise Manager Common Core Files 10.2.0.4.2

Enterprise Manager Agent Core Files 10.2.0.4.2

Agent Required Support Files 10.2.0.4.2

regexp 2.1.9.0.0

Parser Generator Required Support Files 11.2.0.1.0

Oracle 11g Warehouse Builder Required Files 11.2.0.1.0

Oracle Notification Service (eONS) 11.2.0.1.0

Oracle Text Required Support Files 11.2.0.1.0

Precompiler Required Support Files 11.2.0.1.0

Oracle Database 11g Multimedia Files 11.2.0.1.0

Oracle Multimedia Java Advanced Imaging 11.2.0.1.0

Oracle Multimedia Annotator 11.2.0.1.0

Oracle JDBC/OCI Instant Client 11.2.0.1.0

Oracle Multimedia Locator RDBMS Files 11.2.0.1.0

Oracle Core Required Support Files 11.2.0.1.0

Oracle Help For Java 4.2.9.0.0

Sample Schema Data 11.2.0.1.0

Oracle Starter Database 11.2.0.1.0

Oracle Message Gateway Common Files 11.2.0.1.0

Oracle XML Query 11.2.0.1.0

XML Parser for Oracle JVM 11.2.0.1.0

Expat libraries 2.0.1.0.1

Installation Plugin Files 11.2.0.1.0

Enterprise Manager Common Files 10.2.0.4.2

Perl Modules 5.10.0.0.1

Deinstallation Tool 11.2.0.1.0

Oracle Quality of Service Management (Client) 11.2.0.1.0

Perl Interpreter 5.10.0.0.1

JAccelerator (COMPANION) 11.2.0.1.0

Oracle Containers for Java 11.2.0.1.0

Oracle Code Editor 1.2.1.0.0I

Oracle Net Required Support Files 11.2.0.1.0

Secure Socket Layer 11.2.0.1.0

Oracle Universal Connection Pool 11.2.0.1.0

Oracle JDBC/THIN Interfaces 11.2.0.1.0

Oracle Multimedia Client Option 11.2.0.1.0

Oracle Java Client 11.2.0.1.0

Character Set Migration Utility 11.2.0.1.0

Oracle Locale Builder 11.2.0.1.0

PL/SQL Embedded Gateway 11.2.0.1.0

OLAP SQL Scripts 11.2.0.1.0

Database SQL Scripts 11.2.0.1.0

Oracle Globalization Support 11.2.0.1.0

Required Support Files 11.2.0.1.0

SQL*Plus Files for Instant Client 11.2.0.1.0

Oracle ODBC Driver 11.2.0.1.0

Oracle Database User Interface 2.2.13.0.0

Oracle Notification Service 11.2.0.0.0

Enterprise Manager Minimal Integration 11.2.0.1.0

XML Parser for Java 11.2.0.1.0

Oracle Security Developer Tools 11.2.0.1.0

Oracle Wallet Manager 11.2.0.1.0

Cluster Verification Utility Common Files 11.2.0.1.0

Oracle Clusterware RDBMS Files 11.2.0.1.0

Oracle UIX 2.2.24.5.0

Enterprise Manager plugin Common Files 11.2.0.1.0

HAS Common Files 11.2.0.1.0

Precompiler Common Files 11.2.0.1.0

Installation Common Files 11.2.0.1.0

Oracle Help for the  Web 2.0.14.0.0

Oracle LDAP administration 11.2.0.1.0

Buildtools Common Files 11.2.0.1.0

Assistant Common Files 11.2.0.1.0

Oracle Recovery Manager 11.2.0.1.0

PL/SQL 11.2.0.1.0

Generic Connectivity Common Files 11.2.0.1.0

Oracle Database Gateway for ODBC 11.2.0.1.0

Oracle Programmer 11.2.0.1.0

Oracle Database Utilities 11.2.0.1.0

Enterprise Manager Agent 10.2.0.4.2

Oracle Netca Client 11.2.0.1.0

SQL*Plus 11.2.0.1.0

Oracle Call Interface (OCI) 11.2.0.1.0

Oracle Multimedia Locator 11.2.0.1.0

Oracle Multimedia 11.2.0.1.0

Oracle Net 11.2.0.1.0

Database Configuration and Upgrade Assistants 11.2.0.1.0

Oracle XML Development Kit 11.2.0.1.0

Oracle JVM 11.2.0.1.0

Oracle Advanced Security 11.2.0.1.0

Oracle Internet Directory Client 11.2.0.1.0

HAS Files for DB 11.2.0.1.0

Oracle Enterprise Manager Console DB 11.2.0.1.0

Oracle Net Listener 11.2.0.1.0

Oracle Text 11.2.0.1.0

Oracle Net Services 11.2.0.1.0

Oracle Database 11g 11.2.0.1.0

Oracle OLAP 11.2.0.1.0

Oracle Spatial 11.2.0.1.0

Oracle Partitioning 11.2.0.1.0

Enterprise Edition Options 11.2.0.1.0

—————————————————————————–

Instantiating scripts for add node (Thursday, June 28, 2012 1:58:45 PM IST)

.                                                                 1% Done.

Instantiation of add node scripts complete

Copying to remote nodes (Thursday, June 28, 2012 1:58:55 PM IST)

…………………………………………………………………………………..                                 96% Done.

Home copied to new nodes

Saving inventory on nodes (Thursday, June 28, 2012 2:24:28 PM IST)

.                                                               100% Done.

Save inventory complete

WARNING:

The following configuration scripts need to be executed as the “root” user in each cluster node.

/raczone/db_home1/root.sh #On nodes rac3

To execute the configuration scripts:

1. Open a terminal window

2. Log in as “root”

3. Run the scripts in each cluster node

The Cluster Node Addition of /raczone/db_home1 was successful.

Please check ‘/tmp/silentInstall.log’ for more details.

[oracle@rac2 bin]$

Now, add an instance in a rac enviornment

srvctl add instance -d dev -i dev3 -n rac3

So, after successfully adding node, you can check the status of your database

[grid@rac3 ~]$ srvctl status database -d dev

Instance dev1 is running on node rac1

Instance dev2 is running on node rac2

Instance dev3 is running on node rac3

oifcfg

allocating and deallocating network interfaces, get values from OCR

To display a list of networks
oifcfg getif
eth1  10.0.2.0  global  cluster_interconnect
eth0  10.0.1.0.  global  public

display a list of current subnets
oifcfg iflist
etho  10.0.1.0
ethl  10.0.2.0

To include a description of the subnet, specify the -p option:
oifcfg iflist -p
ethO  10.0.1.0  PRIVATE
ethl  10.0.2.0 PRIVATE

In 10.2 public interfaces are UNKNOWN.
To include the subnet mask, append the -n option to the -p option:
oifcfg if list -p -n
etho  10.0.1.0  PRIVATE  255.255.255.0
ethl  10.0.2.0 PRIVATE   255.255.255.0

How to create services in oracle rac 10g

Create service by using (srvctl)

$srvctl add service -d racdb -s oltp -r rac1 -a rac2

$lsnrctl services |grep oltp

Not Found

$srvctl start service -d racdb -s oltp

$./crs_stat

NAME=ora.racdb.oltp.cs

TYPE=application

TARGET=ONLINE

STATE=ONLINE on rac1

NAME=ora-racdb.racdb1.srv

TYPE=application

TARGET=ONLINE

STATE=ONLINE on rac1

Manual Cleanup of CRS installation – Oracle CRS 10g R2 (10.2.0.1)

Sometimes we have to do a manual cleanup for the failed installation :

1) Run rootdelete.sh script present in $ORA_CRS_HOME/install directory

2) Run rootdeinstall.sh script present in $ORA_CRS_HOME/install directory

If you have unknowingly deleted the CRS HOME before running the above scripts then you can follow below procedure to manually clean up the CRS.

Manual Cleanup of CRS on Linux

1. Remove following files as root user on ALL NODES IN CLUSTER

rm /etc/oracle/*
rm -f /etc/init.d/init.cssd
rm -f /etc/init.d/init.crs
rm -f /etc/init.d/init.crsd
rm -f /etc/init.d/init.evmd
rm -f /etc/rc2.d/K96init.crs
rm -f /etc/rc2.d/S96init.crs
rm -f /etc/rc3.d/K96init.crs
rm -f /etc/rc3.d/S96init.crs
rm -f /etc/rc5.d/K96init.crs
rm -f /etc/rc5.d/S96init.crs
rm -Rf /etc/oracle/scls_scr
rm -f /etc/inittab.crs
cp /etc/inittab.orig /etc/inittab

2. If they are not already down then kill off EVM, CRS and CSS process as root user on ALL NODES

ps -ef | grep crs    
kill <crs pid>
ps -ef | grep evm
kill <evm pid>
ps -ef | grep css    
kill <css pid>

3. Remove the files in /var/tmp/.oracle or /tmp/.oracle on ALL NODES

rm -f /var/tmp/.oracle/*
or
rm -f /tmp/.oracle/*

4. Remove the ocr.loc at /etc/oracle on ALL NODES

5. De-install the CRS home in the Oracle Universal Installer
This is to remove the CRS home from inventory. Installer may not be able to remove the files in the slot as CRS home is already deleted, but it will clean up the inventory.

6. Remove the CRS install location on ALL NODES
In case you have not removed the CRS install directory, please do so.

7. Remove OCR and Voting disk from shared location from any 1 node

Once above manual steps are done, the hosts are cleaned up and ready for next CRS installation.

Hope this helps !!

References:

Metalink Note ID : 239998.1