Monday, January 5, 2009

Failure at final check of Oracle CRS stack.

# SYMPTOM :- when installing CRS you get Failure while running root.sh on second RAC node.

Failure at final check of Oracle CRS stack.
10 # Status = 10

[oracle@prod-oranode-221 bin]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

# ocssd.log

[ CSSD]2008-12-31 16:26:19.165 [1158695232] >TRACE: clssnmReadDskHeartbeat: node 1, prod-oranode-220, has a disk HB, but no network HB, DHB has rcfg 126203009, wrtcnt, 374, LATS 2000274, lastSeqNo 374, timestamp 1230740777/2052924

# DEBUG :- Metalink Note :- 369699.1

[oracle@prod-oranode-220 ~]$ traceroute prod-oranode-221.cmc.local
traceroute to prod-oranode-221.cmc.local (10.220.1.104), 30 hops max, 40 byte packets
1 prod-oranode-221.cmc.local (10.220.1.104) 1.120 ms !X 1.094 ms !X 1.057 ms !X

man traceroute

After the time some additional annotation can be printed: !H, !N, or !P (host, network
or protocol unreachable), !S (source route failed), !F (fragmentation needed), !X (com-
munication administratively prohibited), !V (host precedence violation), !C (precedence
cutoff in effect), or ! (ICMP unreachable code ). If almost all the probes
result in some kind of unreachable, traceroute will give up and exit.

# FIX :- check firewall / routing / iptables issues

# 1. reboot and check that CRS processes are now running OK

[oracle@prod-oranode-221 bin]$ ./crs_stat -t
CRS-0202: No resources are registered.

# 2. Look at /opt/oracle/product/11.1.0/crs/install/rootconfig and run the missing steps (VIPCA)

[root@prod-oranode-221 bin]# CRS_NODELIST=prod-oranode-220,prod-oranode-221
[root@prod-oranode-221 bin]# CRS_NODEVIPS='prod-oranode-220/prod-vip-220/255.255.248.0/bond0,prod-oranode-221/prod-vip-221/255.255.248.0/bond0'
[root@prod-oranode-221 bin]# ./vipca -silent -nodelist $CRS_NODELIST -nodevips $CRS_NODEVIPS

Creating VIP application resource on (2) nodes...
Creating GSD application resource on (2) nodes...
Creating ONS application resource on (2) nodes...
Starting VIP application resource on (2) nodes...
Starting GSD application resource on (2) nodes...
Starting ONS application resource on (2) nodes...


Done.
[root@prod-oranode-221 bin]# ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....220.gsd application ONLINE ONLINE prod...-220
ora....220.ons application ONLINE ONLINE prod...-220
ora....220.vip application ONLINE ONLINE prod...-220
ora....221.gsd application ONLINE ONLINE prod...-221
ora....221.ons application ONLINE ONLINE prod...-221
ora....221.vip application ONLINE ONLINE prod...-221

# 3. Go back to OUI and click OK for post root.sh steps to continue.