今天接到一个朋友的电话,说他有个客户rac安装的时候总是有问题。cluster软件已经装上,但是没法装数据库。由于网络环境比较差,无法远程,只能通过QQ来了解情况和诊断了。
一开始,先让对方运行crs_stat -t看看各个资源的状况:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[oracle@rac01 bin]$ ./crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....SM1.asm application ONLINE ONLINE rac01 ora....01.lsnr application ONLINE ONLINE rac01 ora.rac01.gsd application ONLINE ONLINE rac01 ora.rac01.ons application ONLINE ONLINE rac01 ora.rac01.vip application ONLINE ONLINE rac01 ora....SM2.asm application ONLINE OFFLINE ora....02.lsnr application ONLINE ONLINE rac02 ora.rac02.gsd application ONLINE ONLINE rac02 ora.rac02.ons application ONLINE ONLINE rac02 ora.rac02.vip application ONLINE ONLINE rac02 |
发现在rac02上asm没起来,并且通过ps -ef 看asm的进程也不存在:
1 2 3 |
[root@rac02 etc]# ps -ef|grep asm oracle 21691 15798 0 11:11 pts/1 00:00:00 more /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_pmon_12899.trc root 21892 19317 0 11:11 pts/3 00:00:00 grep asm |
由于很多时候,特别是在虚拟机中,crs_start启动总是会有点问题,一般只要重启,都会解决该问题,于是尝试重启crs,用crs_stop -all和crs_start -all重启。
在启动的时候,报错了:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
[oracle@rac01 bin]$ ./crs_start -all Attempting to start `ora.rac01.vip` on member `rac01` Attempting to start `ora.rac02.vip` on member `rac02` Attempting to start `ora.rac02.ASM2.asm` on member `rac02` Attempting to start `ora.rac01.ASM1.asm` on member `rac01` Start of `ora.rac01.vip` on member `rac01` succeeded. Start of `ora.rac02.vip` on member `rac02` succeeded. Attempting to start `ora.rac01.LISTENER_RAC01.lsnr` on member `rac01` Attempting to start `ora.rac02.LISTENER_RAC02.lsnr` on member `rac02` Start of `ora.rac01.LISTENER_RAC01.lsnr` on member `rac01` succeeded. Start of `ora.rac02.LISTENER_RAC02.lsnr` on member `rac02` succeeded. Start of `ora.rac01.ASM1.asm` on member `rac01` failed. rac02 : CRS-1019: Resource ora.rac01.ASM1.asm (application) cannot run on rac02 Start of `ora.rac02.ASM2.asm` on member `rac02` succeeded. Attempting to start `ora.rac01.gsd` on member `rac01` Attempting to start `ora.rac01.ons` on member `rac01` CRS-1002: Resource 'ora.rac02.ons' is already running on member 'rac02' Attempting to start `ora.rac02.gsd` on member `rac02` Start of `ora.rac01.gsd` on member `rac01` succeeded. Start of `ora.rac02.gsd` on member `rac02` succeeded. Start of `ora.rac01.ons` on member `rac01` succeeded. CRS-0215: Could not start resource 'ora.rac01.ASM1.asm'. CRS-0223: Resource 'ora.rac02.ons' has placement error. |
上面的报错中,关键的一句还是:rac02 : CRS-1019: Resource ora.rac01.ASM1.asm (application) cannot run on rac02。检查crs_stat -t,发现:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[oracle@rac01 bin]$ ./crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....SM1.asm application ONLINE OFFLINE ora....01.lsnr application ONLINE ONLINE rac01 ora.rac01.gsd application ONLINE ONLINE rac01 ora.rac01.ons application ONLINE ONLINE rac01 ora.rac01.vip application ONLINE ONLINE rac01 ora....SM2.asm application ONLINE ONLINE rac02 ora....02.lsnr application ONLINE ONLINE rac02 ora.rac02.gsd application ONLINE ONLINE rac02 ora.rac02.ons application ONLINE ONLINE rac02 ora.rac02.vip application ONLINE ONLINE rac02 |
问题似乎是asm实例只能在一个节点上启动,要去看看asm的log了。
1 2 3 4 5 6 7 8 9 |
到asm的bdump下发现: [oracle@rac01 admin]$ cd +ASM [oracle@rac01 +ASM]$ ls hdump pfile [oracle@rac01 +ASM]$ ll total 8 drwxr-x--- 2 oracle oinstall 4096 Oct 9 10:30 hdump drwxr-x--- 2 oracle oinstall 4096 Oct 9 10:30 pfile [oracle@rac01 +ASM]$ |
没有asm的bdump的log?!那就是似乎还没到crs去拉起asm实例的那一步了。于是继续往上追溯,去看看crs的log:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 |
Oracle Database 11g CRS Release 11.1.0.6.0 - Production Copyright 1996, 2007 Oracle. All rights reserved. 2010-10-09 10:03:29.059: [ default][4277080288] CRS Daemon Starting 2010-10-09 10:03:29.060: [ CRSMAIN][4277080288] Checking the OCR device 2010-10-09 10:03:29.079: [ CRSMAIN][4277080288] Connecting to the CSS Daemon 2010-10-09 10:03:29.080: [ COMMCRS][1107462464]clsc_connect: (0x1c7d86e0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac01_)) 2010-10-09 10:03:29.081: [ CSSCLNT][4277080288]clsssInitNative: failed to connect to (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac01_)), rc 9 2010-10-09 10:03:29.081: [ CRSRTI][4277080288] CSS is not ready. Received status 3 from CSS. Waiting for good status .. 2010-10-09 10:03:30.082: [ COMMCRS][1107462464]clsc_connect: (0x1c7d86c0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac01_)) 2010-10-09 10:03:30.083: [ CSSCLNT][4277080288]clsssInitNative: failed to connect to (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac01_)), rc 9 2010-10-09 10:03:30.083: [ CRSRTI][4277080288] CSS is not ready. Received status 3 from CSS. Waiting for good status .. 2010-10-09 10:03:31.084: [ COMMCRS][1107462464]clsc_connect: (0x1c7d86c0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac01_)) 2010-10-09 10:03:31.084: [ CSSCLNT][4277080288]clsssInitNative: failed to connect to (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac01_)), rc 9 2010-10-09 10:03:31.084: [ CRSRTI][4277080288] CSS is not ready. Received status 3 from CSS. Waiting for good status .. 2010-10-09 10:06:52.786: [ CRSMAIN][4277080288] CRSD running as the Privileged user 2010-10-09 10:06:52.823: [ CLSVER][4277080288] Active Version from OCR:10.1.0.2.0 2010-10-09 10:06:52.823: [ CLSVER][4277080288] Active Version is less than Software Version 2010-10-09 10:06:52.823: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:06:53.824: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:06:54.825: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:06:55.826: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:06:56.827: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:06:57.828: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:06:58.829: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:06:59.831: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:00.831: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:01.832: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:02.833: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retr 2010-10-09 10:07:02.833: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:03.834: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:04.835: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:05.836: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:06.837: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:07.838: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:08.839: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:09.840: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:10.841: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:11.842: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:12.843: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:13.844: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:14.845: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:15.846: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:16.847: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:17.848: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:18.849: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:19.850: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:20.851: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:21.852: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:22.853: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:23.854: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:24.855: [ CSSCLNT][4277080288]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying 2010-10-09 10:07:40.874: [ CLSVER][4277080288] Registered in CSS group crs_version 2010-10-09 10:07:40.874: [ CRSMAIN][4277080288] Initializing OCR 2010-10-09 10:07:40.875: [ CLSVER][1117952320] Monitoring the crs_version group for AV change notification 2010-10-09 10:07:40.875: [ CLSVER][1117952320] Doing grpstat on crs_version group 2010-10-09 10:07:40.875: [ CLSVER][1117952320] Returned from grpstat with event 1 2010-10-09 10:07:40.875: [ CLSVER][1117952320] Doing grpstat on crs_version group 2010-10-09 10:07:40.911: [ OCRRAW][4277080288]proprioo: for disk 0 (/dev/sdd1), id match (1), my id set (1084942139,1028247821) total id sets (1), 1st set (1084942139,1028247821), 2nd set (0,0) my votes (2), total votes (2) 2010-10-09 10:07:41.074: [ CRSD][4277080288] ENV Logging level for Module: allcomp 0 2010-10-09 10:07:41.084: [ CRSD][4277080288] ENV Logging level for Module: default 0 2010-10-09 10:07:41.093: [ CRSD][4277080288] ENV Logging level for Module: OCRRAW 0 2010-10-09 10:07:41.102: [ CRSD][4277080288] ENV Logging level for Module: OCROSD 0 2010-10-09 10:07:41.111: [ CRSD][4277080288] ENV Logging level for Module: OCRCAC 0 2010-10-09 10:07:41.121: [ CRSD][4277080288] ENV Logging level for Module: COMMCRS 0 2010-10-09 10:07:41.130: [ CRSD][4277080288] ENV Logging level for Module: COMMNS 0 |
从log上看,应该是css的错误了,CSS,即Cluster Synchronization Services,根据文档的意思是说:Manages the cluster configuration by controlling which nodes are members of the cluster and by notifying members when a node joins or leaves the cluster. If you are using third-party clusterware, then the css process interfaces with your clusterware to manage node membership information.主要是负责节点间的控制和通信问题了。
尝试ping各个节点:ping rac01没问题,ping rac02没问题,ping rac01-priv没问题,ping rac02-priv没问题;尝试验证互信机制,尝试ssh rac01 date没问题,ssh rac02 date没问题,ssh rac01-priv date没问题,ssh rac02-priv date也没问题。
再次尝试用srvctl重启rac01上的asm,出现了很重要的报错信息:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
[oracle@rac01 dbs]$ srvctl start asm -n rac01 PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac01", [PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac01", [rac01:ora.rac01.ASM1.asm: rac01:ora.rac01.ASM1.asm:SQL*Plus: Release 11.1.0.6.0 - Production on Sat Oct 9 11:58:36 2010 rac01:ora.rac01.ASM1.asm: rac01:ora.rac01.ASM1.asm:Copyright (c) 1982, 2007, Oracle. All rights reserved. rac01:ora.rac01.ASM1.asm: rac01:ora.rac01.ASM1.asm:Enter user-name: Connected to an idle instance. rac01:ora.rac01.ASM1.asm: rac01:ora.rac01.ASM1.asm:SQL> ORA-03113: end-of-file on communication channel rac01:ora.rac01.ASM1.asm:SQL> Disconnected rac01:ora.rac01.ASM1.asm: CRS-0215: Could not start resource 'ora.rac01.ASM1.asm'.]] [PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac01", [rac01:ora.rac01.ASM1.asm: rac01:ora.rac01.ASM1.asm:SQL*Plus: Release 11.1.0.6.0 - Production on Sat Oct 9 11:58:36 2010 rac01:ora.rac01.ASM1.asm: rac01:ora.rac01.ASM1.asm:Copyright (c) 1982, 2007, Oracle. All rights reserved. rac01:ora.rac01.ASM1.asm: rac01:ora.rac01.ASM1.asm:Enter user-name: Connected to an idle instance. rac01:ora.rac01.ASM1.asm: rac01:ora.rac01.ASM1.asm:SQL> ORA-03113: end-of-file on communication channel rac01:ora.rac01.ASM1.asm:SQL> Disconnected rac01:ora.rac01.ASM1.asm: CRS-0215: Could not start resource 'ora.rac01.ASM1.asm'.]] |
根据PRKS-1009和CRS-0215,基本可以断定是网卡设置的问题了。用oifcfg检查:
1 2 3 4 5 6 |
[oracle@rac01 bin]$ ./oifcfg getif eth0 10.0.253.0 global public eth1 192.168.253.0 global cluster_interconnect eth2 192.168.130.0 global cluster_interconnect eth3 192.168.131.0 global cluster_interconnect [oracle@rac01 bin]$ |
问了一下,130和131网段是连存储的,和rac间的priv通信没关系。rac0x-priv是在253网段,因此不应该有eth2和eth3的配置。
1 2 3 4 5 6 7 8 9 10 11 12 |
# Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost #Public IP 10.0.253.151 rac01 10.0.253.152 rac02 #Private IP 192.168.253.151 rac01-priv 192.168.253.152 rac02-priv #Virtual IP 10.0.253.156 rac01-vip 10.0.253.157 rac02-vip |
用oifcfg del删除:
1 2 3 4 5 |
[oracle@rac01 bin]$ ./oifcfg delif -global eth2/192.168.130.0 [oracle@rac01 bin]$ ./oifcfg delif -global eth3/192.168.131.0 [oracle@rac01 bin]$ ./oifcfg getif eth0 10.0.253.0 global public eth1 192.168.253.0 global cluster_interconnect |
再次重启crs:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
--在一个窗口运行crs_stop: [oracle@rac01 bin]$ ./crs_stop -all --在另一窗口看: [oracle@rac01 bin]$ ./crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....SM1.asm application ONLINE ONLINE rac01 ora....01.lsnr application OFFLINE OFFLINE ora.rac01.gsd application OFFLINE OFFLINE ora.rac01.ons application ONLINE ONLINE rac01 ora.rac01.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE OFFLINE ora....02.lsnr application OFFLINE OFFLINE ora.rac02.gsd application OFFLINE OFFLINE ora.rac02.ons application OFFLINE OFFLINE ora.rac02.vip application OFFLINE OFFLINE |
发现还有2个asm和nodeapp没停下来,用srvctl停:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
srvctl stop asm -n rac01 srvctl stop nodeapps -n rac01 [oracle@rac01 bin]$ ./crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....SM1.asm application OFFLINE OFFLINE ora....01.lsnr application OFFLINE OFFLINE ora.rac01.gsd application OFFLINE OFFLINE ora.rac01.ons application OFFLINE OFFLINE ora.rac01.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE OFFLINE ora....02.lsnr application OFFLINE OFFLINE ora.rac02.gsd application OFFLINE OFFLINE ora.rac02.ons application OFFLINE OFFLINE ora.rac02.vip application OFFLINE OFFLINE |
再次启动:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
[oracle@rac01 bin]$ ./crs_start -all Attempting to start `ora.rac01.vip` on member `rac01` Attempting to start `ora.rac02.vip` on member `rac02` Attempting to start `ora.rac02.ASM2.asm` on member `rac02` Attempting to start `ora.rac01.ASM1.asm` on member `rac01` Start of `ora.rac01.vip` on member `rac01` succeeded. Start of `ora.rac02.vip` on member `rac02` succeeded. Attempting to start `ora.rac01.LISTENER_RAC01.lsnr` on member `rac01` Attempting to start `ora.rac02.LISTENER_RAC02.lsnr` on member `rac02` Start of `ora.rac02.ASM2.asm` on member `rac02` succeeded. Start of `ora.rac01.ASM1.asm` on member `rac01` succeeded. Start of `ora.rac01.LISTENER_RAC01.lsnr` on member `rac01` succeeded. Start of `ora.rac02.LISTENER_RAC02.lsnr` on member `rac02` succeeded. CRS-1002: Resource 'ora.rac01.ons' is already running on member 'rac01' CRS-1002: Resource 'ora.rac02.ons' is already running on member 'rac02' Attempting to start `ora.rac01.gsd` on member `rac01` Attempting to start `ora.rac02.gsd` on member `rac02` Start of `ora.rac01.gsd` on member `rac01` succeeded. Start of `ora.rac02.gsd` on member `rac02` succeeded. CRS-0223: Resource 'ora.rac01.ons' has placement error. CRS-0223: Resource 'ora.rac02.ons' has placement error. [oracle@rac01 bin]$ ./crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....SM1.asm application ONLINE ONLINE rac01 ora....01.lsnr application ONLINE ONLINE rac01 ora.rac01.gsd application ONLINE ONLINE rac01 ora.rac01.ons application ONLINE ONLINE rac01 ora.rac01.vip application ONLINE ONLINE rac01 ora....SM2.asm application ONLINE ONLINE rac02 ora....02.lsnr application ONLINE ONLINE rac02 ora.rac02.gsd application ONLINE ONLINE rac02 ora.rac02.ons application ONLINE ONLINE rac02 ora.rac02.vip application ONLINE ONLINE rac02 |
搞定,可以继续安装rac数据库了!
3条评论
根据PRKS-1009和CRS-0215,基本可以断定是网卡设置的问题了。用oifcfg检查:
根据这个 怎么就能断定一定是网卡的问题呢?给个充分的理由先~~~
高手.
碰到一样的问题,谢谢前辈。