Environment： AIX 6.1 ，TWO node rac，oracle 10.2.0.4
Symptom ： 用户反映无法打开MES程序，连接到2号节点查看，发现sqlplus /as sysdba回车后卡死，cancel不掉？重新开一个连接，到root账户下 crsctl check crs 命令也卡死，lsnrctl status 命令卡死，OS命令正常。1号节点正常。Alterlog有如下报错
Sat Nov 3 16:37:32 2012Thread 2 advanced to log sequence 55778 (LGWR switch) Current log# 8 seq# 55778 mem# 0: +SFC3DB_DATA/sfc3db/onlinelog/group_8.304.722515055 Current log# 8 seq# 55778 mem# 1: +SFC3DB_ARCH/sfc3db/onlinelog/group_8.264.722515055Sat Nov 3 16:40:19 2012Errors in file /apps/oracle/admin/sfc3db/bdump/sfc3db2_arc0_1843352.trc:ORA-00240: control file enqueue held for more than 120 secondsSat Nov 3 16:42:29 2012Errors in file /apps/oracle/admin/sfc3db/bdump/sfc3db2_arc1_565424.trc:ORA-00240: control file enqueue held for more than 120 secondsSat Nov 3 16:47:00 2012kkjcre1p: unable to spawn jobq slave process Sat Nov 3 16:47:00 2012Sat Nov 3 17:04:22 2012ksvsubmit: Process(O001) creation failed系统应该是无法分配新的进程了，所以这些命令都卡死， 但是当时OS free的内存还有10多G，也远没有达到最大进程数。我最后的解决办法是直接reboot OS，重启后OK， 各位大神有遇到过这种情况的吗？需要什么LOG 我发上来。如下是trc文件sfc3rac2:/apps/oracle/admin/sfc3db/bdump$ cat /apps/oracle/admin/sfc3db/bdump/sfc3db2_cjq0_5861420.trcDump file /apps/oracle/admin/sfc3db/bdump/sfc3db2_cjq0_5861420.trcOracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, OLAP, Data Miningand Real Application Testing optionsORACLE_HOME = /apps/oracle/product/10.2.0System name: AIXNode name: sfc3rac2Release: 1Version: 6Machine: 00C8B9724C00Instance name: sfc3db2Redo thread mounted by this instance: 2Oracle process number: 15Unix process pid: 5861420, image: oracle@sfc3rac2 (CJQ0)
*** 2012-11-03 06:00:41.427*** SERVICE NAMESYS$BACKGROUND) 2012-11-03 06:00:41.419*** SESSION ID2192.1) 2012-11-03 06:00:41.419GATHER_STATS_JOB: Stopped by Scheduler.Consider increasing the maintenance window duration if this happens frequently.The following objects/segments were not analyzed due to timeout:*** 2012-11-03 16:31:39.479Waited for process J000 to initialize for 60 seconds*** 2012-11-03 16:31:39.479Dumping diagnostic information for J000:OS pid = 917522loadavg : 3.77 3.49 3.60swap info: free_mem = 9427.14M rsv = 64.00M alloc = 241.15M avail = 16384.00M swap_free = 16142.85M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD1040005 A oracle 917522 1 0 60 20 1ad230590 89196 16:30:38 - 0:00 [oracle]skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 917522 2>&1' timed out after 60 seconds*** 2012-11-03 16:33:40.701*** 2012-11-03 16:33:50.907Waited for process J000 to initialize for 70 seconds*** 2012-11-03 16:33:50.907Dumping diagnostic information for J000:OS pid = 917522loadavg : 2.86 3.25 3.49swap info: free_mem = 9348.35M rsv = 64.00M alloc = 241.11M avail = 16384.00M swap_free = 16142.89M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD1040005 A oracle 917522 1 0 60 20 1ad230590 89196 16:30:38 - 0:00 [oracle]skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 917522 2>&1' timed out after 60 seconds*** 2012-11-03 16:35:52.258*** 2012-11-03 16:36:02.459Waited for process J000 to initialize for 80 seconds*** 2012-11-03 16:36:02.459Dumping diagnostic information for J000:OS pid = 917522loadavg : 2.74 3.00 3.36swap info: free_mem = 9385.39M rsv = 64.00M alloc = 241.10M avail = 16384.00M swap_free = 16142.90M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD1040005 A oracle 917522 1 0 60 20 1ad230590 89196 16:30:38 - 0:00 [oracle]skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 917522 2>&1' timed out after 60 seconds*** 2012-11-03 16:38:03.960*** 2012-11-03 16:38:14.165Waited for process J000 to initialize for 90 seconds*** 2012-11-03 16:38:14.165Dumping diagnostic information for J000:OS pid = 917522loadavg : 2.56 2.96 3.30swap info: free_mem = 9440.27M rsv = 64.00M alloc = 241.00M avail = 16384.00M swap_free = 16143.00M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD1040005 A oracle 917522 1 0 60 20 1ad230590 89196 16:30:38 - 0:00 [oracle]skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 917522 2>&1' timed out after 60 seconds*** 2012-11-03 16:40:16.175*** 2012-11-03 16:40:26.384Waited for process J000 to initialize for 100 seconds*** 2012-11-03 16:40:26.384Dumping diagnostic information for J000:OS pid = 917522loadavg : 2.82 2.90 3.23swap info: free_mem = 9488.88M rsv = 64.00M alloc = 240.96M avail = 16384.00M swap_free = 16143.04M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD1040005 A oracle 917522 1 0 60 20 1ad230590 89196 16:30:38 - 0:00 [oracle]skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 917522 2>&1' timed out after 60 seconds*** 2012-11-03 16:42:29.100*** 2012-11-03 16:42:39.305Waited for process J000 to initialize for 110 seconds*** 2012-11-03 16:42:39.305Dumping diagnostic information for J000:OS pid = 917522loadavg : 4.05 3.26 3.32swap info: free_mem = 9640.50M rsv = 64.00M alloc = 240.90M avail = 16384.00M swap_free = 16143.10M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD1040005 A oracle 917522 1 0 60 20 1ad230590 89196 16:30:38 - 0:00 [oracle]skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 917522 2>&1' timed out after 60 seconds*** 2012-11-03 16:44:43.732*** 2012-11-03 16:44:53.936Waited for process J000 to initialize for 120 seconds*** 2012-11-03 16:44:53.936Dumping diagnostic information for J000:OS pid = 917522loadavg : 3.03 3.16 3.28swap info: free_mem = 9667.09M rsv = 64.00M alloc = 240.84M avail = 16384.00M swap_free = 16143.16M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD1040005 A oracle 917522 1 0 60 20 1ad230590 89196 16:30:38 - 0:00 [oracle]skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 917522 2>&1' timed out after 60 seconds*** 2012-11-03 16:47:00.265Killing process (ospid 917522): (reason=x4 error=0)... and the process is still alive after kill!*** 2012-11-03 17:05:23.892Waited for process J000 to initialize for 60 seconds*** 2012-11-03 17:05:23.892Dumping diagnostic information for J000:OS pid = 3915970loadavg : 2.43 2.42 2.75swap info: free_mem = 10365.04M rsv = 64.00M alloc = 239.53M avail = 16384.00M swap_free = 16144.47M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD1040005 A oracle 3915970 1 0 60 20 105d26590 89196 17:04:22 - 0:00 [oracle]skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 3915970 2>&1' timed out after 60 seconds*** 2012-11-03 17:07:35.954*** 2012-11-03 17:07:46.156Waited for process J000 to initialize for 70 seconds*** 2012-11-03 17:07:46.156Dumping diagnostic information for J000:OS pid = 3915970loadavg : 2.24 2.33 2.67swap info: free_mem = 10400.53M rsv = 64.00M alloc = 239.37M avail = 16384.00M swap_free = 16144.63M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD1040005 A oracle 3915970 1 0 60 20 105d26590 89196 17:04:22 - 0:00 [oracle]skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 3915970 2>&1' timed out after 60 seconds*** 2012-11-03 17:09:58.294*** 2012-11-03 17:10:08.495Waited for process J000 to initialize for 80 seconds*** 2012-11-03 17:10:08.495Dumping diagnostic information for J000:OS pid = 3915970loadavg : 2.36 2.38 2.64swap info: free_mem = 10389.09M rsv = 64.00M alloc = 239.23M avail = 16384.00M swap_free = 16144.77M F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD1040005 A oracle 3915970 1 0 60 20 105d26590 89196 17:04:22 - 0:00 [oracle]skgpgpstack: read() for cmd /bin/sh -c '/usr/bin/procstack 3915970 2>&1' timed out after 60 seconds*** 2012-11-03 17:12:20.685*** 2012-11-03 17:12:30.886Waited for process J000 to initialize for 90 seconds*** 2012-11-03 17:12:30.886Dumping diagnostic information for J000:还有个trc文件太大暂时不放了 。
the recommended patch. 3. Apply the patch in a test environment. 4. Retest the issue. 5. Migrate the solution as appropriate to other environments.
The issue got the same symptoms ,call stacks ,version with the bug .
The issue has been documented in bug 10207611 which is duplicated to bug 6270137
thank you
标志了以后用
你这个是HPUX的系统 我这是AIX的系统 还是有些不同的。