Thursday, August 6, 2009

ORA-00328: archived log ends at change 35826448060, need later change 35829206909

# OAO dataguard was behind. Recovery was running, but it was not doing anything. When recovery was stopped and started it reported :-

Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT
Thu Aug 6 13:36:57 2009
Recovery interrupted!
Thu Aug 6 13:36:57 2009
Errors in file /opt/oracle/product/admin/pl01oao1/bdump/pl01oao1_mrp0_14017.trc:
ORA-00328: archived log ends at change 35826448060, need later change 35829206909
ORA-00334: archived log: '+ASM_ORADATA226/pl01oao/archivelog/2009_08_06/thread_2_seq_5570.2263.694186543'
Thu Aug 6 13:36:57 2009

# Seems like this was caused by a network glitch while sending the log, or a restart on the standby.

Thu Aug 6 11:34:32 2009
Errors in file /opt/oracle/product/admin/pd01oao2/bdump/pd01oao2_lns1_3510.trc:
ORA-03113: end-of-file on communication channel
Thu Aug 6 11:34:32 2009
LGWR: I/O error 3113 archiving log 22 to 'pl01oao'
Thu Aug 6 11:36:43 2009
Beginning log switch checkpoint up to RBA [0x15c3.2.10], SCN: 35826448061
Thu Aug 6 11:36:43 2009
Thread 2 advanced to log sequence 5571 (LGWR switch)
Current log# 23 seq# 5571 mem# 0: +ASM_ORADATA216/pd01oao/onlinelog/group_23.264.685470871
Current log# 23 seq# 5571 mem# 1: +ASM_ORADATA211/pd01oao/onlinelog/group_23.270.685470873
Thu Aug 6 11:36:43 2009
LNS: Failed to archive log 22 thread 2 sequence 5570 (3113)

# Seems like there was data missing from an archivelog, and recovery was waiting for that data. (not a gap, so FAL server did not help)

I tried deleting the archivelog in ASM, running crosscheck, starting recovery, but the bad log was still online in a standby redo log, and recovery of the standby redo log failed.

# Fixed by clearing the standby redo log group and restarting recovery.

SYS@pl01oao1>alter database clear logfile group 24;

Database altered.