Lokeshaggarwal's DBA

Live the life you love. Love the life you live

PROC-26: Error while accessing the physical storage ASM error AMDU-00204: AMDU-00201:


Hi,

Today, I was facing problem in 11gR2 RAC, after server got restarted, one of the node was not able to startup the clusterware services.When I check OCRCHECK, i got a message:

[root@rac1 bin]# ./ocrcheck

PROT-602: Failed to retrieve data from the cluster registry

PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep=204, loc=kgfokge

AMDU-00204: Disk N0003 is in currently mounted diskgroup CRS

AMDU-00201: Disk N0003: ‘ORCL:VOTE1’

[root@rac1 bin]# ./crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

[root@rac1 bin]# ./crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

But on the another node all the services were successfully running

So ,I tried to find out any error in OCSSD.log,CRSD.log :

 

Getting an error in ocssd.log

2014-01-29 21:10:13.288: [    CSSD][2997357456]clssgmEvtInformation: reqtype (11) cmProc (0x9d02110) client ((nil))
2014-01-29 21:10:13.288: [    CSSD][2997357456]clssgmEvtInformation: reqtype (11) req (0x9c1aa70)
2014-01-29 21:10:13.288: [    CSSD][2997357456]clssnmQueueNotification: type (11) 0x9c1aa70
2014-01-29 21:10:13.999: [    GPnP][3046368960]clsgpnpm_newWiredMsg: [at clsgpnpm.c:633] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri “http://www.grid-pnp.org/2005/12/gpnp-errors#”%5D
2014-01-29 21:10:13.999: [    GPnP][3046368960]clsgpnpm_newWiredMsg: [at clsgpnpm.c:663] Result: (31) CLSGPNP_CALL_AGAIN. Received gpnp meg-reply 0x9cebdb0 with soap fault, reason ‘Operation returned Retry (error CLSGPNP_CALL_AGAIN)'(10) 10
2014-01-29 21:10:13.999: [    GPnP][3046368960]clsgpnpm_receiveMsg: [at clsgpnpm.c:2954] Result: (31) CLSGPNP_CALL_AGAIN. error in gpnp-msg from ‘ipc’ buf=0x9ce9af8
2014-01-29 21:10:13.999: [    GPnP][3046368960]clsgpnpm_exchange: [at clsgpnpm.c:1198] Result: (31) CLSGPNP_CALL_AGAIN. Error soap response from url “ipc://GPNPD_rac1”,msg=0x9cebdb0 rdom=0x9d58c40 try=1/500
@

crsd.log file

{code}
2014-02-03 11:22:40.021: [    GPnP][3046098640]clsgpnpwu_walletfopen: [at clsgpnpwu.c:494] Opened SSO wallet: ‘/raczone/grid_home/gpnp/rac1/wallets/peer/cwallet.sso’
2014-02-03 11:22:40.021: [    GPnP][3046098640]clsgpnp_getCK: [at clsgpnp0.c:1965] Result: (0) CLSGPNP_OK. Get gpnp wallet – provider 1 of 2 (LSKP-FSW(1))
2014-02-03 11:22:40.021: [    GPnP][3046098640]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2014-02-03 11:22:40.021: [    GPnP][3046098640]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=5545, tl=3, f=0

2014-02-03 11:22:40.058: [GIPCXCPT][3046098640] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)

2014-02-03 11:22:40.060: [GIPCXCPT][3046098640] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)

2014-02-03 11:22:40.069: [  OCRASM][3046098640]proprasmo: Error in open/create file in dg [CRS]

[  OCRASM][3046098640]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

ORA-15077: could not locate ASM instance serving a required diskgroup

2014-02-03 11:22:40.071: [  OCRASM][3046098640]proprasmo: kgfoCheckMount returned [7]

2014-02-03 11:22:40.071: [  OCRASM][3046098640]proprasmo: The ASM instance is down

2014-02-03 11:22:40.071: [  OCRRAW][3046098640]proprioo: Failed to open [+CRS]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.

2014-02-03 11:22:40.071: [  OCRRAW][3046098640]proprioo: No OCR/OLR devices are usable

2014-02-03 11:22:40.071: [  OCRASM][3046098640]proprasmcl: asmhandle is NULL

2014-02-03 11:22:40.071: [  OCRRAW][3046098640]proprinit: Could not open raw device

2014-02-03 11:22:40.071: [  OCRASM][3046098640]proprasmcl: asmhandle is NULL

2014-02-03 11:22:40.072: [  OCRAPI][3046098640]a_init:16!: Backend init unsuccessful : [26]

2014-02-03 11:22:40.072: [  CRSOCR][3046098640] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

ORA-15077: could not locate ASM instance serving a required diskgroup

] [7]

2014-02-03 11:22:40.072: [    CRSD][3046098640][PANIC] CRSD exiting: Could not init OCR, code: 26

2014-02-03 11:22:40.072: [    CRSD][3046098640] Done.
I have also checked using amdu utility:

 

RAC1 were the error is coming up:

[grid@rac1 bin]$ ./asmcmd lsdsk -G DATA

Connected to an idle instance.

[grid@rac1 bin]$ ./amdu -diskstring=”ORCL:*” -dump DATA  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0001 is in currently mounted diskgroup DATA

AMDU-00201: Disk N0001: ‘ORCL:DATA1’

** HEARTBEAT DETECTED **

[grid@rac1 bin]$ ./amdu -diskstring=”ORCL:*” -dump VOTE  -nomap -nodir   | grep HEARTBEAT

AMDU-00210: No disks found in diskgroup VOTE

[grid@rac1 bin]$ ./amdu -diskstring=”ORCL:*” -dump FRA  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0002 is in currently mounted diskgroup FRA

AMDU-00201: Disk N0002: ‘ORCL:FRA1’

** HEARTBEAT DETECTED **

[grid@rac1 bin]$ ./amdu -diskstring=”ORCL:*” -dump CRS  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0003 is in currently mounted diskgroup CRS

AMDU-00201: Disk N0003: ‘ORCL:VOTE1’

** HEARTBEAT DETECTED **

[grid@rac1 bin]$

RAC2 (Here, clusterware is running properly)

ORACLE_SID = [grid] ? +ASM2

The Oracle base for ORACLE_HOME=/raczone/grid_home is /raczone/11.2.0

[grid@rac2 bin]$ ./asmcmd lsdsk -G DATA

Path

ORCL:DATA1

[grid@rac2 bin]$ ./amdu -diskstring=”ORCL:*” -dump DATA  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0001 is in currently mounted diskgroup DATA

AMDU-00201: Disk N0001: ‘ORCL:DATA1’

** HEARTBEAT DETECTED **

[grid@rac2 bin]$

[grid@rac2 bin]$ ./amdu -diskstring=”ORCL:*” -dump VOTE  -nomap -nodir   | grep HEARTBEAT

AMDU-00210: No disks found in diskgroup VOTE

[grid@rac2 bin]$ ./amdu -diskstring=”ORCL:*” -dump CRS  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0003 is in currently mounted diskgroup CRS

AMDU-00201: Disk N0003: ‘ORCL:VOTE1’

** HEARTBEAT DETECTED **

[grid@rac2 bin]$ ./amdu -diskstring=”ORCL:*” -dump FRA  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0002 is in currently mounted diskgroup FRA

AMDU-00201: Disk N0002: ‘ORCL:FRA1’

** HEARTBEAT DETECTED **

So, heartbeat also was not a problem,

When I executed Asmcmd, I got to know that ASM instance was not up,so I tried to startup the instance:

[grid@rac1 bin]$ asmcmd
Connected to an idle instance.

but in rac2, asm instance was up

[grid@rac2 bin]$ asmcmd
ASMCMD> ls
CRS/
DATA/
FRA/

So I have started the instance ,

ASMCMD> startup
ASM instance started

Total System Global Area  284565504 bytes
Fixed Size                  1336036 bytes
Variable Size             258063644 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
ASMCMD> ls
CRS/
DATA/
FRA/
ASMCMD> cd CRS
ASMCMD> ls
scan-cluster/
ASMCMD> cd sc*
ASMCMD> ls
ASMPARAMETERFILE/
OCRFILE/
ASMCMD> cd OCR*
ASMCMD> ls -l
Type     Redund  Striped  Time             Sys  Name
OCRFILE  UNPROT  COARSE   FEB 03 15:00:00  Y    REGISTRY.255.837088345
ASMCMD>
ASMCMD> exit
[grid@rac1 bin]$ ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          3
Total space (kbytes)     :     262120
Used space (kbytes)      :       2800
Available space (kbytes) :     259320
ID                       : 1486492518
Device/File Name         :       +CRS
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

[grid@rac1 bin]$ exit
logout

[root@rac1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac1 bin]# ./crs_stat -t
Name           Type           Target    State     Host
————————————————————
ora.CRS.dg     ora….up.type ONLINE    ONLINE    rac1
ora.DATA.dg    ora….up.type ONLINE    ONLINE    rac1
ora.FRA.dg     ora….up.type ONLINE    ONLINE    rac1
ora….ER.lsnr ora….er.type ONLINE    ONLINE    rac1
ora….N1.lsnr ora….er.type ONLINE    ONLINE    rac1
ora….N2.lsnr ora….er.type ONLINE    ONLINE    rac2
ora….N3.lsnr ora….er.type ONLINE    ONLINE    rac2
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1
ora.dev.db     ora….se.type ONLINE    ONLINE    rac2
ora.eons       ora.eons.type  ONLINE    ONLINE    rac1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora….network ora….rk.type ONLINE    ONLINE    rac1
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1
ora….SM1.asm application    ONLINE    ONLINE    rac1
ora….C1.lsnr application    ONLINE    ONLINE    rac1
ora.rac1.gsd   application    OFFLINE   OFFLINE
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   ora….t1.type ONLINE    ONLINE    rac1
ora….SM2.asm application    ONLINE    ONLINE    rac2
ora….C2.lsnr application    ONLINE    ONLINE    rac2
ora.rac2.gsd   application    OFFLINE   OFFLINE
ora.rac2.ons   application    ONLINE    ONLINE    rac2
ora.rac2.vip   ora….t1.type ONLINE    ONLINE    rac2
ora….ry.acfs ora….fs.type ONLINE    ONLINE    rac1
ora.scan1.vip  ora….ip.type ONLINE    ONLINE    rac1
ora.scan2.vip  ora….ip.type ONLINE    ONLINE    rac2
ora.scan3.vip  ora….ip.type ONLINE    ONLINE    rac2
[root@rac1 bin]# ./crs_stat -t|grep OFFLINE
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE
ora.rac1.gsd   application    OFFLINE   OFFLINE
ora.rac2.gsd   application    OFFLINE   OFFLINE
[root@rac1 bin]#
[root@rac1 bin]#

And now all the services are now successfully runinng.

 

Advertisements

February 3, 2014 - Posted by | RAC

No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: