紧急求助 oracle建索引失败后 RAC宕机！

Question

系统 AIX 6.1  oracle 10.2.0.4 昨天下午添加一个索引，建了2个小时还没有成功就手动取消了！
之后查看别人smon会做善后工作 就没有处理。一直观察也没有什么问题
突然在刚才23:45开始 出现大量的 ora 600错误。
同事 OS也报错
alert .logORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []
Wed Sep  7 00:26:42 2011
Errors in file /apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:
ORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []
Wed Sep  7 00:26:44 2011
Trace dumping is performing id=[cdmp_20110907002644]
Wed Sep  7 00:27:10 2011
Errors in file /apps/oracle/admin/sfc3db/udump/sfc3db2_ora_2756730.trc:
ORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []
Wed Sep  7 00:28:11 2011
Errors in file /apps/oracle/admin/sfc3db/udump/sfc3db2_ora_2756730.trc:
ORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []erppt 
LABEL:          CORE_DUMP
IDENTIFIER:     A924A5FCDate/Time:       Wed Sep  7 00:13:35 GMT+08:00 2011
Sequence Number: 2190
Machine Id:      00C8B9724C00
Node Id:         sfc3rac2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSPROCDescription
SOFTWARE PROGRAM ABNORMALLY TERMINATEDProbable Causes
SOFTWARE PROGRAMUser Causes
USER GENERATED SIGNAL        Recommended Actions
        CORRECT THEN RETRYFailure Causes
SOFTWARE PROGRAM        Recommended Actions
        RERUN THE APPLICATION PROGRAM
        IF PROBLEM PERSISTS THEN DO THE FOLLOWING
        CONTACT APPROPRIATE SERVICE REPRESENTATIVEDetail Data
SIGNAL NUMBER
           6
USER'S PROCESS ID:
               5452002
FILE SYSTEM SERIAL NUMBER
          14
INODE NUMBER
      931776
CORE FILE NAME
/apps/oracle/admin/sfc3db/cdump/core_5452002/core
PROGRAM NAME
oracle
STACK EXECUTION DISABLED
           0
COME FROM ADDRESS REGISTER
splay 0PROCESSOR ID
  hw_fru_id: 0
  hw_cpu_id: 1ADDITIONAL INFORMATION
slcra 3EC
??
ksedmp 330
ksfdmp 1C
kgeriv 10C
kgesiv 84
ksesic0 64
kxfprigdb 2F0
kxfpssg 3D4
kxfpqsod_ 610
kxfpqsod 31C
kxfrdel 1D0
kxsffir 244
kxsFreeWo 100
kxsFreeEx 2C
kksumc 8D8
opifch2 1C9C
opiefn0 3E4
opiefn E0
opiodr AE4
ttcpip 1024
opitsk 1128
opiino 994
opiodr AE4
opidrv 488
sou2o 94
opimai_re 1C0
main 9C
__start 74Symptom Data
REPORTABLE
1
INTERNAL ERROR
0
SYMPTOM CODE
PCSS/SPI2 FLDS/oracle SIG/6 FLDS/slcra VALU/3ec根本连接不上 随便执行一个查询就报ORA 600错误
没有办法只有关掉这个节点！和我取消掉这个在建的索引有关吗？
紧急求助 最高只能给100分

Answer 1

This is the generic internal error number for Oracle program exceptions. This indicates that a process has encountered an exceptional condition.To recreate a control file,issue:
alter database backup controlfile to trace;
Check your trace file and modify/copy/paste in sqlplus. 
 
 

Answer 2

是生产库 重新启动了 没用 无法连接这个节点  报ORA600 错误

Answer 3

还有没有高手来帮忙看看 要什么log 说  我贴上来

Answer 4

ID 839536.1
An ORA-600 [15764] is highly transient in nature. Most bugs filed for this issue have been closed as not reproducible 
问题是这个bug如何解决呢

Answer 5

ORA-00600: internal error code, arguments: [15764], ORA-00600[15764] 和parallel query 有关。 现在可能定位报错时，有那些SQL 在之行，可能是这些SQL 导致的。
 索引重建失败，一般是报：ORA-08104ORA-600 各个参数含义说明
http://blog.csdn.net/tianlesoftware/article/details/6645809Oracle alter index rebuild 说明
http://blog.csdn.net/tianlesoftware/article/details/6538928 

Answer 6

MOS 上给的解决方案是重启所有节点上的instance.
Solution
Workaround:
   
Bounce all instances in the RAC cluster.

Answer 7

Errors in file /apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:
ORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []这个trace 文件里应该有对应的SQL 语句. 

Answer 8

我用 
grep -E '(select|insert|delete|update|merge|begin|exec|\.trc)' -i  /apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc得到下面的信心  dave帮忙在看一下/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:Dump file /apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:100000000      95505K  read/exec         oracle
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:9fffffff0000000        51K  read/exec         /usr/ccs/bin/usla64
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000ac8898         2K  read/exec         /usr/lib/libC.a[shr3_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000aaaa80       115K  read/exec         /usr/lib/libC.a[shrcore_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000992500       112K  read/exec         /usr/lib/libC.a[ansicore_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:9000000009f2468         0K  read/exec         /usr/lib/libicudata.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:9000000009af6a0         2K  read/exec         /usr/lib/libC.a[shr2_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:90000000080d580      1552K  read/exec         /usr/lib/libC.a[ansi_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000439f80       134K  read/exec         /usr/lib/libC.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000668480      1680K  read/exec         /usr/lib/libicuuc.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:90000000045ca80      2089K  read/exec         /usr/lib/libicui18n.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:9000000003c7280        91K  read/exec         /usr/lib/libsrc.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:9000000009f3c80       727K  read/exec         /usr/lib/liblvm.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000ae4200        98K  read/exec         /usr/lib/libcorcfg.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:9000000003ee700        81K  read/exec         /usr/lib/libcfg.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000403380         2K  read/exec         /usr/lib/libcrypt.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000b00240         5K  read/exec         /usr/lib/libc.a[aio_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:9000000003a8a00       119K  read/exec         /usr/lib/libodm.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000aca580       100K  read/exec         /usr/lib/libperfstat.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000aff000         0K  read/exec         /usr/lib/libdl.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000002303100      8636K  read/exec         /apps/oracle/product/10.2.0/lib/libjox10.a[shr.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:9000000009b1000       257K  read/exec         /usr/lib/libpthreads.a[shr_xpg5_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:900000000000800      3740K  read/exec         /usr/lib/libc.a[shr_64.o]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:      calls aborted: 0, num est exec limit hit: 0
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:      name=update seq$ set increment$=:2,minvalue=:3,maxvalue=:4,cycle#=:5,order$=:6,cache=:7,highwater=:8,audit$=:9,flags=:10 where obj#=:1
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:      name=select decode(failover_method, NULL, 0 , 'BASIC', 1, 'PRECONNECT', 2 , 'PREPARSE', 4 , 0), decode(failover_type, NULL, 1 , 'NONE', 1 , 'SESSION', 2, 'SELECT',  4, 1), failover_retries, failover_delay, flags from service$ where name = :1
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN') from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:      name=select value$ from props$ where name = 'GLOBAL_DB_NAME'
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:      name=select privilege#,level from sysauth$ connect by grantee#=prior privilege# and privilege#>0 start with grantee#=:1 and privilege#>0
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:      name=select privilege# from sysauth$ where (grantee#=:1 or grantee#=1) and privilege#>0
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:  parallel_execution_enabled          = false
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:  bitmap_merge_area_size              = 1048576
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:  _enable_type_dep_selectivity        = true
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:  _optimizer_sortmerge_join_enabled   = true
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:  _optimizer_correct_sq_selectivity   = true
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:  _update_bji_ipdml_enabled           = 0
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:  _optimizer_complex_pred_selectivity = true
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:  _optimizer_sortmerge_join_inequality = true
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:  Chunk  70000007c60e920 sz=       48    freeable  "execution fro i"
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc: sqltxt(70000008e304be8)=select privilege#,level from sysauth$ connect by grantee#=prior privilege# and privilege#>0 start with grantee#=:1 and privilege#>0
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc: sqltxt(70000008e308988)=select privilege# from sysauth$ where (grantee#=:1 or grantee#=1) and privilege#>0
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc: sqltxt(70000008e3739a8)=select value$ from props$ where name = 'GLOBAL_DB_NAME'
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'SERVICE_NAME'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN') from v$instance where INSTANCE_NAME=SYS_CONTEXT('USERENV', 'INSTANCE_NAME')
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:select decode(failover_method, NULL, 0 , 'BASIC', 1, 'PRECONNECT', 2 , 'PREPARSE', 4 , 0), decode(failover_type, NULL, 1 , 'NONE', 1 , 'SESSION', 2, 'SELECT',  4, 1), failover_retries, failover_delay, flags from service$ where name = :1
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc: sqltxt(70000008e39b380)=update seq$ set increment$=:2,minvalue=:3,maxvalue=:4,cycle#=:5,order$=:6,cache=:7,highwater=:8,audit$=:9,flags=:10 where obj#=:1
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:boolean kmgs_spfile_update_ [11022E910, 11022E914) = 00000000
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:word kcf_exec_read_count_ [11022ED98, 11022ED9C) = 0000024D
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:1100004C0 3730362E 74726300 00000000 00000000  [706.trc.........]
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:word xsaggrmergethresh_ [11022D500, 11022D504) = 00000558
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:word xsoqshareexecutors_ [11022D61C, 11022D620) = 00000591
/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:word xsoqsqlenableawqdrmerge_ [11022D618, 11022D61C) = 0000059B