背景：
昨天中午，我们公司楼层由于电路原因，突然断电。
我们开发小组的数据库开发环境（Linux）也遭遇宕机。以下是alert_sid.log
ORA-00474: SMON process terminated with errorThu Jul 5 23:01:54 2012
PMON: terminating instance due to error 474Instance terminated by PMON, pid = 10718
Thu Jul 5 23:02:21 2012
Starting ORACLE instance (normal)
Cannot determine all dependent dynamic libraries for /proc/self/exe
Unable to find dynamic library libocr10.so in search paths
RPATH = /ade/aime1_build2101/oracle/has/lib/:/ade/aime1_build2101/oracle/lib/:/ade/aime1_build2101/oracle/has/lib/:
LD_LIBRARY_PATH is not set!
The default library directories are /lib and /usr/lib
Unable to find dynamic library libocrb10.so in search paths
Unable to find dynamic library libocrutl10.so in search paths
Unable to find dynamic library libocrutl10.so in search paths
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /usr/local/oracle/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
processes = 150
__shared_pool_size = 79691776
__large_pool_size = 4194304
__java_pool_size = 4194304
__streams_pool_size = 4194304
sga_target = 285212672
control_files = /usr/local/oracle/oradata/techdb/control01.ctl, /usr/local/oracle/oradata/techdb/control02.ctl, /usr/local/oracle/oradata/techdb/control03.ctl
db_block_size = 8192
__db_cache_size = 184549376
compatible = 10.2.0.1.0
db_file_multiblock_read_count= 16
undo_management = AUTO
undo_tablespace = UNDOTBS1
_kgl_large_heap_warning_threshold= 52428800
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=techdbXDB)
job_queue_processes = 10
background_dump_dest = /usr/local/oracle/admin/techdb/bdump
user_dump_dest = /usr/local/oracle/admin/techdb/udump
core_dump_dest = /usr/local/oracle/admin/techdb/cdump
audit_file_dest = /usr/local/oracle/admin/techdb/adump
db_name = techdb
open_cursors = 300
pga_aggregate_target = 94371840
PMON started with pid=2, OS id=10967
PSP0 started with pid=3, OS id=10969
MMAN started with pid=4, OS id=10971
DBW0 started with pid=5, OS id=10973
LGWR started with pid=6, OS id=10975
CKPT started with pid=7, OS id=10977
SMON started with pid=8, OS id=10979
RECO started with pid=9, OS id=10981
CJQ0 started with pid=10, OS id=10983
MMON started with pid=11, OS id=10985
Thu Jul 5 23:02:21 2012
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=10987
Thu Jul 5 23:02:21 2012
starting up 1 shared server(s) ...
Thu Jul 5 23:02:21 2012
ALTER DATABASE MOUNT
Thu Jul 5 23:02:25 2012
Setting recovery target incarnation to 2
Thu Jul 5 23:02:25 2012
Successful mount of redo thread 1, with mount id 3665578877
Thu Jul 5 23:02:25 2012
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Thu Jul 5 23:02:25 2012
ALTER DATABASE OPEN
Thu Jul 5 23:02:25 2012
Beginning crash recovery of 1 threads
parallel recovery started with 2 processes
Thu Jul 5 23:02:25 2012
Started redo scan
Thu Jul 5 23:02:25 2012
Completed redo scan
1972 redo blocks read, 336 data blocks need recovery
Thu Jul 5 23:02:26 2012
Started redo application at
Thread 1: logseq 209575, block 553
Thu Jul 5 23:02:26 2012
Recovery of Online Redo Log: Thread 1 Group 3 Seq 209575 Reading mem 0
Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo03.log
Thu Jul 5 23:02:26 2012
Completed redo application
Thu Jul 5 23:02:26 2012
Completed crash recovery at
Thread 1: logseq 209575, block 2525, scn 7788557050
336 data blocks read, 335 data blocks written, 1972 redo blocks read
Thu Jul 5 23:02:26 2012
Thread 1 advanced to log sequence 209576
Thread 1 opened at log sequence 209576
Current log# 1 seq# 209576 mem# 0: /usr/local/oracle/oradata/techdb/redo01.log
Successful open of redo thread 1
Thu Jul 5 23:02:26 2012
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Jul 5 23:02:26 2012
SMON: enabling cache recovery
Thu Jul 5 23:02:27 2012
Successfully onlined Undo Tablespace 1.
Thu Jul 5 23:02:27 2012
SMON: enabling tx recovery
Thu Jul 5 23:02:27 2012
Database Characterset is ZHS16GBK
Thu Jul 5 23:02:27 2012
SMON: Restarting fast_start parallel rollbackThu Jul 5 23:02:27 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_p000_10995.trc:
ORA-00600: internal error code, arguments: [4198], [9], [], [], [], [], [], []
Thu Jul 5 23:02:27 2012
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=20, OS id=11003
Thu Jul 5 23:02:28 2012
Completed: ALTER DATABASE OPEN
Thu Jul 5 23:02:28 2012
Doing block recovery for file 2 block 41
Block recovery from logseq 209576, block 3 to scn 7788557080
Thu Jul 5 23:02:28 2012
Recovery of Online Redo Log: Thread 1 Group 1 Seq 209576 Reading mem 0
Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo01.log
Block recovery completed at rba 209576.29.16, scn 1.3493589785
Thu Jul 5 23:02:28 2012
SMON: ignoring slave err,downgrading to serial rollback
Thu Jul 5 23:02:28 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_10979.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
ORACLE Instance techdb (pid = 8) - Error 600 encountered while recovering transaction (3, 17).Thu Jul 5 23:02:28 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_10979.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
Thu Jul 5 23:08:08 2012
SMON: Restarting fast_start parallel rollback
Thu Jul 5 23:08:08 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_p000_11052.trc:
ORA-00600: internal error code, arguments: [4198], [9], [], [], [], [], [], []
Thu Jul 5 23:08:09 2012
Doing block recovery for file 2 block 41
Block recovery from logseq 209576, block 373 to scn 7788557661
Thu Jul 5 23:08:09 2012
Recovery of Online Redo Log: Thread 1 Group 1 Seq 209576 Reading mem 0
Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo01.log
Block recovery stopped at EOT rba 209576.374.16
Block recovery completed at rba 209576.374.16, scn 1.3493590364
Thu Jul 5 23:08:09 2012
SMON: ignoring slave err,downgrading to serial rollback
Thu Jul 5 23:08:10 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_10979.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
ORACLE Instance techdb (pid = 8) - Error 600 encountered while recovering transaction (3, 17).
Thu Jul 5 23:08:10 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_10979.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []techdb_smon_10979.trc日志如下
/usr/local/oracle/admin/techdb/bdump/techdb_smon_10979.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
ORACLE_HOME = /usr/local/oracle
System name: Linux
Node name: db.com
Release: 2.6.18-92.el5
Version: #1 SMP Tue Apr 29 13:16:12 EDT 2008
Machine: i686
Instance name: techdb
Redo thread mounted by this instance: 1
Oracle process number: 8
Unix process pid: 10979, image: [email protected] (SMON)*** SERVICE NAME:() 2012-07-05 23:02:27.160
*** SESSION ID:(164.1) 2012-07-05 23:02:27.160
Parallel Transaction recovery caught exception 12801
Parallel Transaction recovery caught error 30317
*** 2012-07-05 23:02:27.162
SMON: Restarting fast_start parallel rollback
Parallel Transaction recovery caught exception 12801
Parallel Transaction recovery caught error 607
*** 2012-07-05 23:02:28.306
SMON: ignoring slave err,downgrading to serial rollback
XID passed in =xid: 0x0003.011.00003c2b
XID from Undo block =xid: 0x0004.020.00002b35
Dump of buffer cache at level 4 for tsn=1, rdba=8388649
BH (0x2b7e986c) file#: 2 rdba: 0x00800029 (2/41) class: 21 ba: 0x2b4a4000
set: 3 blksize: 8192 bsi: 0 set-flg: 2 pwbcnt: 0
dbwrid: 0 obj: -1 objn: 0 tsn: 1 afn: 2
hash: [2b7fb19c,300f6bcc] lru: [2b7e9970,2b7e9810]
ckptq: [2b7e93c4,2b7e9f74] fileq: [2b7e93cc,301929f0] objq: [2e726648,2b7e9444]
st: XCURRENT md: NULL tch: 1
flags: buffer_dirty gotten_in_current_mode redo_since_read
LRBA: [0x332a8.3.0] HSCN: [0x1.d03bf318] HSUB: [1]
buffer tsn: 1 rdba: 0x00800029 (2/41)
scn: 0x0001.d03bf318 seq: 0x01 flg: 0x00 tail: 0xf3182601
frmt: 0x02 chkval: 0x0000 type: 0x26=KTU SMU HEADER BLOCK
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x2B4A4000 to 0x2B4A6000
********中间省略**************
internal error ORA-600 seen on the error stack
process appears to be having problems repeatedly, committing suicide
Background_Core_Dump = partial
昨天中午，我们公司楼层由于电路原因，突然断电。
我们开发小组的数据库开发环境（Linux）也遭遇宕机。以下是alert_sid.log
ORA-00474: SMON process terminated with errorThu Jul 5 23:01:54 2012
PMON: terminating instance due to error 474Instance terminated by PMON, pid = 10718
Thu Jul 5 23:02:21 2012
Starting ORACLE instance (normal)
Cannot determine all dependent dynamic libraries for /proc/self/exe
Unable to find dynamic library libocr10.so in search paths
RPATH = /ade/aime1_build2101/oracle/has/lib/:/ade/aime1_build2101/oracle/lib/:/ade/aime1_build2101/oracle/has/lib/:
LD_LIBRARY_PATH is not set!
The default library directories are /lib and /usr/lib
Unable to find dynamic library libocrb10.so in search paths
Unable to find dynamic library libocrutl10.so in search paths
Unable to find dynamic library libocrutl10.so in search paths
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /usr/local/oracle/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
processes = 150
__shared_pool_size = 79691776
__large_pool_size = 4194304
__java_pool_size = 4194304
__streams_pool_size = 4194304
sga_target = 285212672
control_files = /usr/local/oracle/oradata/techdb/control01.ctl, /usr/local/oracle/oradata/techdb/control02.ctl, /usr/local/oracle/oradata/techdb/control03.ctl
db_block_size = 8192
__db_cache_size = 184549376
compatible = 10.2.0.1.0
db_file_multiblock_read_count= 16
undo_management = AUTO
undo_tablespace = UNDOTBS1
_kgl_large_heap_warning_threshold= 52428800
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=techdbXDB)
job_queue_processes = 10
background_dump_dest = /usr/local/oracle/admin/techdb/bdump
user_dump_dest = /usr/local/oracle/admin/techdb/udump
core_dump_dest = /usr/local/oracle/admin/techdb/cdump
audit_file_dest = /usr/local/oracle/admin/techdb/adump
db_name = techdb
open_cursors = 300
pga_aggregate_target = 94371840
PMON started with pid=2, OS id=10967
PSP0 started with pid=3, OS id=10969
MMAN started with pid=4, OS id=10971
DBW0 started with pid=5, OS id=10973
LGWR started with pid=6, OS id=10975
CKPT started with pid=7, OS id=10977
SMON started with pid=8, OS id=10979
RECO started with pid=9, OS id=10981
CJQ0 started with pid=10, OS id=10983
MMON started with pid=11, OS id=10985
Thu Jul 5 23:02:21 2012
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=10987
Thu Jul 5 23:02:21 2012
starting up 1 shared server(s) ...
Thu Jul 5 23:02:21 2012
ALTER DATABASE MOUNT
Thu Jul 5 23:02:25 2012
Setting recovery target incarnation to 2
Thu Jul 5 23:02:25 2012
Successful mount of redo thread 1, with mount id 3665578877
Thu Jul 5 23:02:25 2012
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Thu Jul 5 23:02:25 2012
ALTER DATABASE OPEN
Thu Jul 5 23:02:25 2012
Beginning crash recovery of 1 threads
parallel recovery started with 2 processes
Thu Jul 5 23:02:25 2012
Started redo scan
Thu Jul 5 23:02:25 2012
Completed redo scan
1972 redo blocks read, 336 data blocks need recovery
Thu Jul 5 23:02:26 2012
Started redo application at
Thread 1: logseq 209575, block 553
Thu Jul 5 23:02:26 2012
Recovery of Online Redo Log: Thread 1 Group 3 Seq 209575 Reading mem 0
Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo03.log
Thu Jul 5 23:02:26 2012
Completed redo application
Thu Jul 5 23:02:26 2012
Completed crash recovery at
Thread 1: logseq 209575, block 2525, scn 7788557050
336 data blocks read, 335 data blocks written, 1972 redo blocks read
Thu Jul 5 23:02:26 2012
Thread 1 advanced to log sequence 209576
Thread 1 opened at log sequence 209576
Current log# 1 seq# 209576 mem# 0: /usr/local/oracle/oradata/techdb/redo01.log
Successful open of redo thread 1
Thu Jul 5 23:02:26 2012
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Jul 5 23:02:26 2012
SMON: enabling cache recovery
Thu Jul 5 23:02:27 2012
Successfully onlined Undo Tablespace 1.
Thu Jul 5 23:02:27 2012
SMON: enabling tx recovery
Thu Jul 5 23:02:27 2012
Database Characterset is ZHS16GBK
Thu Jul 5 23:02:27 2012
SMON: Restarting fast_start parallel rollbackThu Jul 5 23:02:27 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_p000_10995.trc:
ORA-00600: internal error code, arguments: [4198], [9], [], [], [], [], [], []
Thu Jul 5 23:02:27 2012
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=20, OS id=11003
Thu Jul 5 23:02:28 2012
Completed: ALTER DATABASE OPEN
Thu Jul 5 23:02:28 2012
Doing block recovery for file 2 block 41
Block recovery from logseq 209576, block 3 to scn 7788557080
Thu Jul 5 23:02:28 2012
Recovery of Online Redo Log: Thread 1 Group 1 Seq 209576 Reading mem 0
Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo01.log
Block recovery completed at rba 209576.29.16, scn 1.3493589785
Thu Jul 5 23:02:28 2012
SMON: ignoring slave err,downgrading to serial rollback
Thu Jul 5 23:02:28 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_10979.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
ORACLE Instance techdb (pid = 8) - Error 600 encountered while recovering transaction (3, 17).Thu Jul 5 23:02:28 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_10979.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
Thu Jul 5 23:08:08 2012
SMON: Restarting fast_start parallel rollback
Thu Jul 5 23:08:08 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_p000_11052.trc:
ORA-00600: internal error code, arguments: [4198], [9], [], [], [], [], [], []
Thu Jul 5 23:08:09 2012
Doing block recovery for file 2 block 41
Block recovery from logseq 209576, block 373 to scn 7788557661
Thu Jul 5 23:08:09 2012
Recovery of Online Redo Log: Thread 1 Group 1 Seq 209576 Reading mem 0
Mem# 0 errs 0: /usr/local/oracle/oradata/techdb/redo01.log
Block recovery stopped at EOT rba 209576.374.16
Block recovery completed at rba 209576.374.16, scn 1.3493590364
Thu Jul 5 23:08:09 2012
SMON: ignoring slave err,downgrading to serial rollback
Thu Jul 5 23:08:10 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_10979.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
ORACLE Instance techdb (pid = 8) - Error 600 encountered while recovering transaction (3, 17).
Thu Jul 5 23:08:10 2012
Errors in file /usr/local/oracle/admin/techdb/bdump/techdb_smon_10979.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []techdb_smon_10979.trc日志如下
/usr/local/oracle/admin/techdb/bdump/techdb_smon_10979.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
ORACLE_HOME = /usr/local/oracle
System name: Linux
Node name: db.com
Release: 2.6.18-92.el5
Version: #1 SMP Tue Apr 29 13:16:12 EDT 2008
Machine: i686
Instance name: techdb
Redo thread mounted by this instance: 1
Oracle process number: 8
Unix process pid: 10979, image: [email protected] (SMON)*** SERVICE NAME:() 2012-07-05 23:02:27.160
*** SESSION ID:(164.1) 2012-07-05 23:02:27.160
Parallel Transaction recovery caught exception 12801
Parallel Transaction recovery caught error 30317
*** 2012-07-05 23:02:27.162
SMON: Restarting fast_start parallel rollback
Parallel Transaction recovery caught exception 12801
Parallel Transaction recovery caught error 607
*** 2012-07-05 23:02:28.306
SMON: ignoring slave err,downgrading to serial rollback
XID passed in =xid: 0x0003.011.00003c2b
XID from Undo block =xid: 0x0004.020.00002b35
Dump of buffer cache at level 4 for tsn=1, rdba=8388649
BH (0x2b7e986c) file#: 2 rdba: 0x00800029 (2/41) class: 21 ba: 0x2b4a4000
set: 3 blksize: 8192 bsi: 0 set-flg: 2 pwbcnt: 0
dbwrid: 0 obj: -1 objn: 0 tsn: 1 afn: 2
hash: [2b7fb19c,300f6bcc] lru: [2b7e9970,2b7e9810]
ckptq: [2b7e93c4,2b7e9f74] fileq: [2b7e93cc,301929f0] objq: [2e726648,2b7e9444]
st: XCURRENT md: NULL tch: 1
flags: buffer_dirty gotten_in_current_mode redo_since_read
LRBA: [0x332a8.3.0] HSCN: [0x1.d03bf318] HSUB: [1]
buffer tsn: 1 rdba: 0x00800029 (2/41)
scn: 0x0001.d03bf318 seq: 0x01 flg: 0x00 tail: 0xf3182601
frmt: 0x02 chkval: 0x0000 type: 0x26=KTU SMU HEADER BLOCK
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x2B4A4000 to 0x2B4A6000
********中间省略**************
internal error ORA-600 seen on the error stack
process appears to be having problems repeatedly, committing suicide
Background_Core_Dump = partial
分享给大家！
1.使用隐含参数屏蔽异常回滚段_offline_rollback_segments= _SYSSMU3$
2.修改undo_tablespace=SYSTEM/undo_management=MANUAL
3.启动数据库,快速删除包含_SYSSMU3$ undo表空间
4.新建undo表空间
5.修改undo_tablespace=new_undo/undo_management=AUTO,除掉隐含参数
6.使用新参数文件重启数据库
7.建议:使用逻辑导出导入重建数据库