记一次Oracle Clusterware安装成功后的故障处理
1. 环境
[grid@rac1 rac1]$ cat /etc/issueRed Hat Enterprise Linux Server release 5.8 (Tikanga)Kernel \r on an \m
2. 问题描述在安装RAC的过程中, 成功安装好grid (clusterware) 后关闭了各节点. 在下次开启各节点后, 检查crs资源状态, 出现如下错误:
[grid@rac1 ~]$ crs_stat -t -vCRS-0184: Cannot communicate with the CRS daemon.
3. 分析解决
检查crs状态:
[grid@rac1 ~]$ crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4535: Cannot communicate with Cluster Ready Services # 无法与CRS通信CRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online
查看crsd对应日志:
2014-11-21 15:18:13.490: [GIPCXCPT][1002185440] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)2014-11-21 15:18:13.492: [GIPCXCPT][1002185440] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)2014-11-21 15:18:13.498: [ OCRASM][1002185440]proprasmo: Error in open/create file in dg [DATA] # 打开磁盘组失败[ OCRASM][1002185440]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokgeORA-15077: could not locate ASM instance serving a required diskgroup # 没有ASM实例2014-11-21 15:18:13.498: [ OCRASM][1002185440]proprasmo: kgfoCheckMount returned [7]2014-11-21 15:18:13.498: [ OCRASM][1002185440]proprasmo: The ASM instance is down # ASM实例处于关闭状态2014-11-21 15:18:13.499: [ OCRRAW][1002185440]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.2014-11-21 15:18:13.499: [ OCRRAW][1002185440]proprioo: No OCR/OLR devices are usable2014-11-21 15:18:13.499: [ OCRASM][1002185440]proprasmcl: asmhandle is NULL2014-11-21 15:18:13.499: [ OCRRAW][1002185440]proprinit: Could not open raw device 2014-11-21 15:18:13.499: [ OCRASM][1002185440]proprasmcl: asmhandle is NULL2014-11-21 15:18:13.499: [ OCRAPI][1002185440]a_init:16!: Backend init unsuccessful : [26]2014-11-21 15:18:13.499: [ CRSOCR][1002185440] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokgeORA-15077: could not locate ASM instance serving a required diskgroup] [7]2014-11-21 15:18:13.499: [ CRSD][1002185440][PANIC] CRSD exiting: Could not init OCR, code: 262014-11-21 15:18:13.499: [ CRSD][1002185440] Done.
日志信息表明, ASM实例未能启动, 导致crsd进程无法启动
尝试手动启动ASM实例:[grid@rac1 ~]$ asmcmdConnected to an idle instance.ASMCMD> startupORA-27154: post/wait create failedORA-27300: OS system dependent operation:semget failed with status: 28ORA-27301: OS failure message: No space left on deviceORA-27302: failure occurred at: sskgpsemsperConnected to an idle instance.
上述信息表明, 失败的操作是semget.
semget的任务是获得信号量集(get set of semaphores), 这里的No space left on device并不是指存储空间，而是信号量资源.检查系统中的信号量使用情况:[grid@rac1 ~]$ ipcs------ Shared Memory Segments --------key shmid owner perms bytes nattch status 0x00000000 3407873 root 644 80 2 0x00000000 3440643 root 644 16384 2 0x00000000 3473412 root 644 280 2 ------ Semaphore Arrays --------key semid owner perms nsems ------ Message Queues --------key msqid owner perms used-bytes messages
未发现异常. 继续检查内核参数中的semmns:
root@rac1 ~]# sysctl -a|grep semkernel.sem = 250 100 32 128
四个参数分别为:
semmsl---每个信号集包含的信号数，该值应比ORACLE进程的最大数大10左右semmns---系统中的信号数semopm---每个信号操作呼叫的最大操作数semmni---信号集标识符数,用于控制可随时创建的信号集数加大系统中的信号量(/etc/sysctl.conf):kernel.sem = 256 32768 100 228
重新启动ASM实例:
ASMCMD> startupORA-03113: end-of-file on communication channelConnected to an idle instance.
因着急继续做实验, 此时直接对两个节点进行了重新启动, 重启后ASM实例正常启动, crs资源状态正常, 问题得到解决.
后来实验结束后查询ORA-03113, 导致该错误的可能原因有:1) Unix核心参数设置不当2) Oracle执行文件权限不正确／环境变量问题3) 客户端通信不能正确处理4) 数据库服务器崩溃/操作系统崩溃/进程被kill5) Oracle 内部错误6) 特定SQL、PL/SQL引起的错误7) 空间不够8) 防火墙的问题
但因错误环境已消失, 未能排查解决, 很是遗憾, 仅留做以后参考.
4. 参考
1) [Oracle 11g RAC CRS-4535/ORA-15077] 2) [ASM启动报错ORA-27300, ORA-27301 and ORA-27302: failure occurred at: sskgpsemsper] 3) [DBA手记:共享内存无法正常释放的处理] 4) [ORA-03113: end-of-file on communication channel 错误定位过程]