..experience – 第 19 页

rac修改异步IO导致挂死

2009-12-04 3条评论

今天遇到个rac的异步IO的问题，环境是oracle 10201的双节点rac，存储使用ASM，机器是HP 的PA的机器，操作系统是11.23. 在udump下的日志中有大量的报错：

oracle@rp8440a:/oracle/oracle/admin/gzxnh/bdump> cat gzxnh1_ora_19552.trc
/oracle/oracle/admin/gzxnh/bdump/gzxnh1_ora_19552.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /oracle/oracle/product/10.2.0/db_2
System name:    HP-UX
Node name:      rp8440a
Release:        B.11.23
Version:        U
Machine:        9000/800
Instance name: gzxnh1
Redo thread mounted by this instance: 1
Oracle process number: 0
Unix process pid: 19552, image: oracle@rp8440a

Ioctl ASYNC_CONFIG error, errno = 1

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

oracle@rp8440a:/oracle/oracle/admin/gzxnh/bdump> cat gzxnh1_ora_19552.trc

/oracle/oracle/admin/gzxnh/bdump/gzxnh1_ora_19552.trc

Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining options

ORACLE_HOME = /oracle/oracle/product/10.2.0/db_2

System name: HP-UX

Node name: rp8440a

Release: B.11.23

Version: U

Machine: 9000/800

Instance name: gzxnh1

Redo thread mounted by this instance: 1

Oracle process number: 0

Unix process pid: 19552, image: oracle@rp8440a

Ioctl ASYNC_CONFIG error, errno = 1

看到这个报错，觉得很轻松，之前也遇到过，是orac […]

..experience, Working case

sqlnet文件格式导致侦听无法启动

2009-09-04 4条评论

今天凌晨接到某省的一个电话，数据库进行MC service guard的切换，但是切换过去后，发现侦听无法启动，启动时侦听报错信息为：

oracle@ha_dc02:/oracle/app/oracle/product/9.2.0/network/admin > lsnrctl start

LSNRCTL for HPUX: Version 9.2.0.6.0 - Production on 04-SEP-2009 01:46:07

Copyright (c) 1991, 2002, Oracle Corporation.  All rights reserved.

Starting /oracle/app/oracle/product/9.2.0/bin/tnslsnr: please wait...

TNSLSNR for HPUX: Version 9.2.0.6.0 - Production
System parameter file is /oracle/app/oracle/product/9.2.0/network/admin/listener.ora
Log messages written to /oracle/app/oracle/product/9.2.0/network/log/listener.log
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.203.41.11)(PORT=1521)))
TNS-12532: TNS:invalid argument
 TNS-12560: TNS:protocol adapter error
  TNS-00502: Invalid argument


Listener failed to start. See the error message(s) above...

oracle@ha_dc02:/oracle/app/oracle/product/9.2.0/network/admin >

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

oracle@ha_dc02:/oracle/app/oracle/product/9.2.0/network/admin > lsnrctl start

LSNRCTL for HPUX: Version 9.2.0.6.0 - Production on 04-SEP-2009 01:46:07

Starting /oracle/app/oracle/product/9.2.0/bin/tnslsnr: please wait...

TNSLSNR for HPUX: Version 9.2.0.6.0 - Production

System parameter file is /oracle/app/oracle/product/9.2.0/network/admin/listener.ora

Log messages written to /oracle/app/oracle/product/9.2.0/network/log/listener.log

Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.203.41.11)(PORT=1521)))

TNS-12532: TNS:invalid argument

TNS-12560: TNS:protocol adapter error

TNS-00502: Invalid argument

Listener failed to start. See the error message(s) above...

oracle@ha_dc02:/oracle/app/oracle/product/9.2.0/network/admin >

检查配置，网络，都没有问题，浮动IP也是启动在10.203.41.11上，但是侦听就是起不来，检查 […]

..experience, Working case

用crs实现single instance的HA

2009-08-06 6条评论

数据库的高可用方案我们通常采用的是双机热备模式，一个机器跑主机，另一个机器进行standby。当发生故障的时候，故障机上的数据库被切换到备机，在备机上起浮动IP和数据库，对应用来讲是透明的，只是发生了短暂的切换。目前行业中常用的高可用软件如HP的Service Guard、IBM的HACMP，价格都 […]

..experience, Working case

启动lnsr需要写tmp权限

2009-07-29 6条评论

今天凌晨被叫醒起来，某省的数据库db01坏了一个cpu，由于有HA架构的保护，切换到了db02，但是在db02上却无法启动侦听，以下是报错信息：

oracle@gz_dc02:/oracle/app/oracle/admin/gzmisc/bdump > lsnrctl start

LSNRCTL for HPUX: Version 9.2.0.6.0 - Production on 29-JUL-2009 03:53:16

Copyright (c) 1991, 2002, Oracle Corporation.  All rights reserved.

Starting /oracle/app/oracle/product/9.2.0/bin/tnslsnr: please wait...

TNSLSNR for HPUX: Version 9.2.0.6.0 - Production
System parameter file is /oracle/app/oracle/product/9.2.0/network/admin/listener.ora
Log messages written to /oracle/app/oracle/product/9.2.0/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.203.104.11)(PORT=1521)))
Error listening on: (ADDRESS=(PROTOCOL=ipc)(PARTIAL=yes)(QUEUESIZE=1))
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.203.104.11)(PORT=1521)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   HPUX Error: 2: No such file or directory

Listener failed to start. See the error message(s) above...

oracle@gz_dc02:/oracle/app/oracle/admin/gzmisc/bdump >

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

oracle@gz_dc02:/oracle/app/oracle/admin/gzmisc/bdump > lsnrctl start

LSNRCTL for HPUX: Version 9.2.0.6.0 - Production on 29-JUL-2009 03:53:16

Starting /oracle/app/oracle/product/9.2.0/bin/tnslsnr: please wait...

TNSLSNR for HPUX: Version 9.2.0.6.0 - Production

System parameter file is /oracle/app/oracle/product/9.2.0/network/admin/listener.ora

Log messages written to /oracle/app/oracle/product/9.2.0/network/log/listener.log

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.203.104.11)(PORT=1521)))

Error listening on: (ADDRESS=(PROTOCOL=ipc)(PARTIAL=yes)(QUEUESIZE=1))

No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.203.104.11)(PORT=1521)))

TNS-12541: TNS:no listener

TNS-12560: TNS:protocol adapter error

TNS-00511: No listener

HPUX Error: 2: No such file or directory

Listener failed to start. See the error message(s) above...

oracle@gz_dc02:/oracle/app/oracle/admin/gzmisc/bdump >

看报错很奇怪：Error listening on: (ADDRESS=(PROTOCO […]

..experience, Working case

设置侦听IP鉴权一段时间后侦听挂死

2009-07-24 一条评论

某省的数据库时不时的发生侦听挂死的现象。我们来看看具体的情况：当出现侦听挂死的时候，sqlplus登录会出现如下报错：

$ sqlplus user/pwd@tnsnames

SQL*Plus: Release 9.2.0.6.0 - Production on Sun Jun 28 11:43:32 2009

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.


ERROR:
ORA-12535: TNS:operation timed out
$

1

2

3

4

5

6

7

8

9

10

$ sqlplus user/pwd@tnsnames

SQL*Plus: Release 9.2.0.6.0 - Production on Sun Jun 28 11:43:32 2009

ERROR:

ORA-12535: TNS:operation timed out

$

而此时，可以看到listener.log中的内容为： [crayon-69070d5b466e071487710 […]

..experience, Working case

ntp服务的安装

2009-07-04 3条评论

在rac环境中，会要求几个节点之间的时间差不能超时。一般如果超过30秒，节点很可能会重启。我们需要配置一个ntp时钟服务器，来给rac的各个节点进行时间同步。在这里，我们以windows机器（192.168.1.189）做ntp server，以2台虚拟机上的rac节点（192.168.1.131 […]

..experience, Working case

安腾平台的legato的lib link

2009-06-29 2条评论

某省新上安腾平台的oracle，备份软件是legato，首先在安腾的机器上安装了networker.pkg和nmo.pkg，配置好client后，发现备份失败。查/nsr/applogs/msglog.log发现：

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of allocate command on t1 channel at 06/29/2009 11:55:45
ORA-19554: error allocating device, device type: SBT_TAPE, device name: 
ORA-27211: Failed to load Media Management Library
Additional information: 25

1

2

3

4

5

6

7

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of allocate command on t1 channel at 06/29/2009 11:55:45

ORA-19554: error allocating device, device type: SBT_TAPE, device name:

ORA-27211: Failed to load Media Management Library

Additional information: 25

对于这个报错，我已 […]

..experience, Working case

rac安装报错和修复小结

2009-06-27 2条评论

一、rac安装步骤主要分以下几步： 1、安装操作系统。 2、配置时间同步，如ntp 3、创建oracle用户和配置rsa和dsa 4、配置hosts文件 5、配置hangcheck-timer 6、配置共享存储（ocr和voting disk） 7、安装clusterware（crs） 8、配置as […]

..experience, Working case

swap空间不够导致连接侦听失败

2009-06-16 3条评论

早上4点多就被叫起来，说某现网的数据库侦听挂了。数据库连不上去，报以下的错误：

$ sqlplus user/pwd@sid

SQL*Plus: Release 9.2.0.6.0 - Production on Fri Jun 12 04:58:05 2009

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

ERROR:
ORA-12500: TNS:listener failed to start a dedicated server process

1

2

3

4

5

6

7

8

$ sqlplus user/pwd@sid

SQL*Plus: Release 9.2.0.6.0 - Production on Fri Jun 12 04:58:05 2009

ERROR:

ORA-12500: TNS:listener failed to start a dedicated server process

登录后检查数据库的侦听进程还在，检查lsnrctl status的状态也是正常。检查侦听的log发现，有大量连接拒连： [crayon-69070d5b […]

..experience, Working case

修改hosts后，rac通讯失败

2009-06-15 没有评论

今天收到个告警某省的一个数据库的一个节点down了，重启后，只能到started状态，数据库无法open，登录上去后，看到alertlog中：

Mon Jun 15 15:38:28 2009
Errors in file /oracle/app/oracle/admin/fjmisc/udump/fjmisc2_ora_26950.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:gethostbyname failed with status: 3
ORA-27301: OS failure message: No such process
ORA-27302: failure occurred at: sskgxpmyip2
ORA-27303: additional information: nodename FJ_DB02
 Mon Jun 15 15:41:19 2009
Errors in file /oracle/app/oracle/admin/fjmisc/udump/fjmisc2_ora_27062.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:gethostbyname failed with status: 3
ORA-27301: OS failure message: No such process
ORA-27302: failure occurred at: sskgxpmyip2
ORA-27303: additional information: nodename FJ_DB02
Mon Jun 15 15:42:37 2009
Errors in file /oracle/app/oracle/admin/fjmisc/udump/fjmisc2_ora_27147.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:gethostbyname failed with status: 3
ORA-27301: OS failure message: No such process
ORA-27302: failure occurred at: sskgxpmyip2
ORA-27303: additional information: nodename FJ_DB02
 Mon Jun 15 15:46:19 2009
Errors in file /oracle/app/oracle/admin/fjmisc/udump/fjmisc2_ora_27362.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

Mon Jun 15 15:38:28 2009

Errors in file /oracle/app/oracle/admin/fjmisc/udump/fjmisc2_ora_26950.trc:

ORA-00603: ORACLE server session terminated by fatal error

ORA-27504: IPC error creating OSD context

ORA-27300: OS system dependent operation:gethostbyname failed with status: 3

ORA-27301: OS failure message: No such process

ORA-27302: failure occurred at: sskgxpmyip2

ORA-27303: additional information: nodename FJ_DB02

Mon Jun 15 15:41:19 2009

Errors in file /oracle/app/oracle/admin/fjmisc/udump/fjmisc2_ora_27062.trc:

ORA-00603: ORACLE server session terminated by fatal error

ORA-27504: IPC error creating OSD context

ORA-27300: OS system dependent operation:gethostbyname failed with status: 3

ORA-27301: OS failure message: No such process

ORA-27302: failure occurred at: sskgxpmyip2

ORA-27303: additional information: nodename FJ_DB02

Mon Jun 15 15:42:37 2009

Errors in file /oracle/app/oracle/admin/fjmisc/udump/fjmisc2_ora_27147.trc:

ORA-00603: ORACLE server session terminated by fatal error

ORA-27504: IPC error creating OSD context

ORA-27300: OS system dependent operation:gethostbyname failed with status: 3

ORA-27301: OS failure message: No such process

ORA-27302: failure occurred at: sskgxpmyip2

ORA-27303: additional information: nodename FJ_DB02

Mon Jun 15 15:46:19 2009

Errors in file /oracle/app/oracle/admin/fjmisc/udump/fjmisc2_ora_27362.trc:

ORA-00603: ORACLE server session terminated by fatal error

ORA-27504: IPC error creating OSD context

当时第一个反映是gethostbyname failed 应该是主机名解析有问题了。但是p […]

..experience, Working case

主机安全加固导致dp分发客户端失败

2009-06-14 没有评论

今年的315晚会，曝光了某省移动的一些违规操作后，移动集团公司对各地分公司都进行了安全检查。在进行一系列的安全加固操作后，我们发现，还是有部分之前很顺利的操作会受到影响。问题的起因是这样的，某省的备份软件是HP的DP。DP能对数据库进行备份，也能对文件系统进行备份（这样的功能，我想一般的备份软件都 […]

..experience, Working case

用hanganalyze解决row cache lock

2009-05-20 3条评论

今天某省的一位同事来说，在执行一个split分区的脚本时长时间没有响应。登录上去查看，手工执行了split脚本，发现确实会hang住：

SQL> l
  1  ALTER TABLE A_PDA_SP_STAT SPLIT PARTITION P_MAX AT (20090609)
  2   INTO (PARTITION P_20090608 TABLESPACE TS_DATA_A
  3*  , PARTITION P_MAX TABLESPACE TS_DATA_A)
SQL>

1

2

3

4

5

SQL> l

1 ALTER TABLE A_PDA_SP_STAT SPLIT PARTITION P_MAX AT (20090609)

2 INTO (PARTITION P_20090608 TABLESPACE TS_DATA_A

3* , PARTITION P_MAX TABLESPACE TS_DATA_A)

SQL>

检查该session的等待事件：

EVENT                                  P1         P2         P3
------------------------------ ---------- ---------- ----------
row cache lock                          8          0          5

1

2

3

EVENT P1 P2 P3

------------------------------ ---------- ---------- ----------

row cache lock 8 0 5

[…]