今天收到监控的一个告警:
1 |
ORA-16146: standby destination control file enqueue unavailable |
登录主库和备库检查其报错信息:
备库上的alertlog:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
Media Recovery Waiting for thread 1 seq# 2506 Media Recovery Log /oracle_log/JPJDE/arch_logs/stby_1_2506.arc Wed Oct 27 20:30:24 2010 Media Recovery Waiting for thread 1 seq# 2507 Media Recovery Log /oracle_log/JPJDE/arch_logs/stby_1_2507.arc Thu Oct 28 00:13:35 2010 Media Recovery Waiting for thread 1 seq# 2508 Media Recovery Log /oracle_log/JPJDE/arch_logs/stby_1_2508.arc Thu Oct 28 01:24:57 2010 RFS: controlfile enqueue unavailable Possible invalid cross-instance archival configuration Thu Oct 28 01:25:00 2010 Errors in file /oracle_log/JPJDE/admin/udump/jpjde_rfs_1331404.trc: ORA-16146: standby destination control file enqueue unavailable Thu Oct 28 01:44:42 2010 RFS: Error State mode '2' Thu Oct 28 01:45:31 2010 Media Recovery Log /oracle_log/JPJDE/arch_logs/stby_1_2509.arc Thu Oct 28 01:46:53 2010 RFS: Forced Shutdown due to RFS_ERROR state Thu Oct 28 01:50:33 2010 Media Recovery Waiting for thread 1 seq# 2510 (in transit) Media Recovery Log /oracle_log/JPJDE/arch_logs/stby_1_2510.arc Media Recovery Log /oracle_log/JPJDE/arch_logs/stby_1_2511.arc Media Recovery Log /oracle_log/JPJDE/arch_logs/stby_1_2512.arc Thu Oct 28 02:11:34 2010 Media Recovery Waiting for thread 1 seq# 2513 Media Recovery Log /oracle_log/JPJDE/arch_logs/stby_1_2513.arc |
对应同一时间,主库上的信息:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
Thread 1 advanced to log sequence 2511 Current log# 4 seq# 2511 mem# 0: /oracle_log/JPJDE/redo_logs/redo_g04.log Thu Oct 28 01:23:58 2010 ARC1: Evaluating archive log 5 thread 1 sequence 2510 ARC1: Beginning to archive log 5 thread 1 sequence 2510 Creating archive destination LOG_ARCHIVE_DEST_2: 'stby' Thu Oct 28 01:25:24 2010 Creating archive destination LOG_ARCHIVE_DEST_1: '/oracle_db_rss/oracle_jpjde/arch_logs/arch_1_2510.arc' Thu Oct 28 01:34:06 2010 Thread 1 advanced to log sequence 2512 Current log# 1 seq# 2512 mem# 0: /oracle_log/JPJDE/redo_logs/redo_g01.log Thu Oct 28 01:34:06 2010 ARC0: Evaluating archive log 5 thread 1 sequence 2510 ARC0: Unable to archive log 5 thread 1 sequence 2510 Log actively being archived by another process ARC0: Evaluating archive log 4 thread 1 sequence 2511 ARC0: Beginning to archive log 4 thread 1 sequence 2511 Creating archive destination LOG_ARCHIVE_DEST_2: 'stby' Thu Oct 28 01:34:24 2010 Creating archive destination LOG_ARCHIVE_DEST_1: '/oracle_db_rss/oracle_jpjde/arch_logs/arch_1_2511.arc' Thu Oct 28 01:44:43 2010 Thread 1 advanced to log sequence 2513 Current log# 2 seq# 2513 mem# 0: /oracle_log/JPJDE/redo_logs/redo_g02.log Thu Oct 28 01:44:43 2010 ARC2: Evaluating archive log 5 thread 1 sequence 2510 ARC2: Unable to archive log 5 thread 1 sequence 2510 Log actively being archived by another process ARC2: Evaluating archive log 4 thread 1 sequence 2511 ARC2: Unable to archive log 4 thread 1 sequence 2511 Log actively being archived by another process ARC2: Evaluating archive log 1 thread 1 sequence 2512 ARC2: Beginning to archive log 1 thread 1 sequence 2512 Creating archive destination LOG_ARCHIVE_DEST_2: 'stby' ARC2: Error 16011 Creating archive log file to 'stby' Thu Oct 28 01:44:43 2010 Errors in file /oracle_log/JPJDE/admin/bdump/jpjde_arc2_6234236.trc: ORA-16011: Archivelog Remote File Server process in Error state Creating archive destination LOG_ARCHIVE_DEST_1: '/oracle_db_rss/oracle_jpjde/arch_logs/arch_1_2512.arc' Thu Oct 28 01:47:05 2010 ARC2: Completed archiving log 1 thread 1 sequence 2512 ARC2: Evaluating archive log 5 thread 1 sequence 2510 ARC2: Unable to archive log 5 thread 1 sequence 2510 Log actively being archived by another process ARC2: Evaluating archive log 4 thread 1 sequence 2511 ARC2: Unable to archive log 4 thread 1 sequence 2511 Log actively being archived by another process ARC2: Evaluating archive log 5 thread 1 sequence 2510 ARC2: Unable to archive log 5 thread 1 sequence 2510 Log actively being archived by another process ARC2: Evaluating archive log 4 thread 1 sequence 2511 ARC2: Unable to archive log 4 thread 1 sequence 2511 Log actively being archived by another process Thu Oct 28 01:47:45 2010 ARC0: Completed archiving log 4 thread 1 sequence 2511 Thu Oct 28 01:47:45 2010 |
因此,结合上面的主库和备库,我们来整理一下2个机器报错的步骤:
(1点24分之前,状态正常)
主库:
1点03分的时候,redo的current切到2509号,之前的2508号切到arch日志中。
Thread 1 advanced to log sequence 2509
Current log# 3 seq# 2509 mem# 0: /oracle_log/JPJDE/redo_logs/redo_g03.log
Thu Oct 28 01:03:20 2010
ARC1: Evaluating archive log 2 thread 1 sequence 2508
ARC1: Beginning to archive log 2 thread 1 sequence 2508
Creating archive destination LOG_ARCHIVE_DEST_2: ‘stby’
Creating archive destination LOG_ARCHIVE_DEST_1: ‘/oracle_db_rss/oracle_jpjde/arch_logs/arch_1_2508.arc’
备库:
从0点13分开始等待2508号日志,并且在1点24之前,完成2508的恢复。
Thu Oct 28 00:13:35 2010
Media Recovery Waiting for thread 1 seq# 2508
Media Recovery Log /oracle_log/JPJDE/arch_logs/stby_1_2508.arc
(1)Thu Oct 28 01:24:57 2010
备库报错:
RFS: controlfile enqueue unavailable
Possible invalid cross-instance archival configuration
(2)Thu Oct 28 01:25:00 2010
备库上继续抛出报错信息:
Errors in file /oracle_log/JPJDE/admin/udump/jpjde_rfs_1331404.trc:
ORA-16146: standby destination control file enqueue unavailable
(3)主库和备库同时写报错(经检查主库时间比备库时间快1秒左右,所以从log中看二者的报错是同时的):
备库:
Thu Oct 28 01:44:42 2010
RFS: Error State mode ‘2’
主库:
Thu Oct 28 01:44:43 2010
Errors in file /oracle_log/JPJDE/admin/bdump/jpjde_arc2_6234236.trc:
ORA-16011: Archivelog Remote File Server process in Error state
Creating archive destination LOG_ARCHIVE_DEST_1: ‘/oracle_db_rss/oracle_jpjde/arch_logs/arch_1_2512.arc’
同时在主库的trace文件中(jpjde_arc2_6234236.trc),显示RFS进程是在一个error的状态,即在备库中写的State mode 2:
Error 16011 creating standby archive log file at host ‘stby’
*** 2010-10-28 01:44:43.977
ARC2: Error 16011 Creating archive log file to ‘stby’
*** 2010-10-28 01:44:43.977
kcrrfail: dest:2 err:16011 force:0
ORA-16011: Archivelog Remote File Server process in Error state
什么都没做,备库自动恢复正常,开始继续的managed recovery.
(4)备库开始恢复2509号日志
备库:
Thu Oct 28 01:45:31 2010
Media Recovery Log /oracle_log/JPJDE/arch_logs/stby_1_2509.arc
查了网上的一些资料,网上的情况和我们不同,别人的是rac的结构,我们只是单实例的dg,再继续上metalink查,发现该报错可能是属于是Bug 3253153。对应的数据库版本是9204,正是我们的数据库版本。该bug会造成在进行managed recover 之前,报错:RFS: CONTROLFILE ENQUEUE UNAVAILABLE,详情见下面的pdf的bug信息。
Bug 3253153 RFS may error starting managed recovery.pdf
而且从alertlog来看只出现了一次:
> cat alert_JPJDE.log |grep ORA-16146
ORA-16146: standby destination control file enqueue unavailable
因此建议可以忽略该报错。没有必要打patch,如果后续持续出现该报错,可以为该报错打p3253153。
一条评论
收藏学习啦