一：版本信息
操作系統版本：AIX 61009
數據庫版本：11.2.0.3.11(RAC)
二：錯誤描述
1) crsctl stat res -t命令查看crs資源狀態的時候，發現"ora.LISTENER.lsnr"資源狀態異常：
ora.LISTENER.lsnr
ONLINE OFFLINE ****1 ##實例1
ONLINE OFFLINE ****2 ##實例2
2)檢查監聽狀態正常
lsnrctl LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 11-NOV-2014 10:53:26 Copyright (c) 1991, 2011, Oracle. All rights reserved. Welcome to LSNRCTL, type "help" for information. LSNRCTL> set password Password: The command completed successfully LSNRCTL> status Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production Start Date 06-NOV-2014 17:22:05 Uptime 4 days 17 hr. 31 min. 48 sec Trace Level off Security ON: Password SNMP OFF Listener Parameter File /oracle/app/11.2.0/grid/network/admin/listener.ora Listener Log File /oracle/app/11.2.0/grid/log/diag/tnslsnr/bidb4/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=***.***.**.**)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=***.***.**.***)(PORT=1521))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Service "bidb" has 1 instance(s). Instance "bidb2", status READY, has 1 handler(s) for this service... The command completed successfully
3)檢查監聽日志，發現目前監聽還能正常接受連接
su - grid
cd $ORACLE_HOME/log/diag/tnslsnr/主機名/listener/alert
</msg> <msg time='2014-11-11T10:46:02.602+08:00' org_id='oracle' comp_id='tnslsnr' type='UNKNOWN' level='16' <span style="color:#ff0000;">host_id='bidb4'</span>##數據庫主機名 <span style="color:#ff0000;">host_addr='***.***.**.**>##</span>數據庫主機ip地址 <txt>11-NOV-2014 10:46:02 * (CONNECT_DATA=(SERVICE_NAME=bidb)(CID=(PROGRAM=osh)(HOST=EGSNS-49-24.nd.cmgame.com)(USER=dsadm))(SERVER=dedicated)(INSTANCE_NAME=bidb2)) * (ADDRESS=(PROTOCOL=tcp)(<span style="color:#ff0000;">HOST=***.***.**.**)(</span>PORT=61378)) * establish * bidb * 0 </txt> ##連接發起的IP地址
4)檢查 CRS alert日志發現有如下報錯：
/oracle/app/11.2.0/grid/bin/oraagent.bin(7274512)]CRS-5016:Process "/oracle/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/oracle/app/11.2.0/grid/bin/oraagent.bin" f or action "start" failed: details at "(:CLSN00010:)" in "/oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/oraagent_grid.log" 2014-11-07 15:03:21.139: [/oracle/app/11.2.0/grid/bin/oraagent.bin(7274512)]CRS-5016:Process "/oracle/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/oracle/app/11.2.0/grid/bin/oraagent.bin" f or action "start" failed: details at "(:CLSN00010:)" in "/oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/oraagent_grid.log" 2014-11-07 15:03:21.267: <span style="color:#ff0000;">[/oracle/app/11.2.0/grid/bin/oraagent.bin(7274512)]CRS-5016:Process "/oracle/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/oracle/app/11.2.0/grid/bin/oraagent.bin" f or action "check" failed: details at "(:CLSN00010:)" in "/oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/oraagent_grid.log" 2014-11-07 15:08:01.278:</span>
##標紅的部分報，監聽檢查報錯，這個就跟我的問題聯系上了。根據提示 查看oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/下的oraagent_grid日志(注意可能你要找的信息已經被刷出當前日志了，根據時間點查看相應日志。例如我需要的相關信息就在oraagent_grid.l02日志文件中)
5)查看oraagent_grid.l02日志
根據時間點，查找與"[ora.LISTENER.lsnr]"相關的日志，如下：
2014-11-07 15:03:21.143: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] <span style="color:#ff0000;"><span style="color:#ff0000;">LsnrAgent::check </span>{ ##listener檢查
</span>2014-11-07 15:03:21.143: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] <span style="color:#ff0000;">lsnrctl status LISTENER</span>
2014-11-07 15:03:21.143: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] getOracleHomeAttrib: oracle_home = /oracle/app/11.2.0/grid
2014-11-07 15:03:21.143: [ AGFW][2314] {2:2661:31941} Agent sending reply for: RESOURCE_START[ora.LISTENER.lsnr bidb4 1] ID 4098:431717
2014-11-07 15:03:21.144: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] getOracleHomeAttrib: oracle_home = /oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Utils::getCrsHome crsHome /oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Utils::execCmd 1 USR_ORA_ENV:ORACLE_BASE=/oracle/app/grid oracleHome:/oracle/app/11.2.0/grid
CrsHome:/oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Utils::getCrsHome crsHome /oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Utils::getCrsHome crsHome /oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Adding Environment Variables ORACLE_HOME=/oracle/app/11.2.0/grid
2014-11-07 15:03:21.145: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Adding Environment Variables TNS_ADMIN=/oracle/app/11.2.0/grid/network/admin/
2014-11-07 15:03:21.146: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Adding Environment variable from USR_ORA_ENV ORACLE_BASE=/oracle/app/grid
2014-11-07 15:03:21.146: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl.
2014-11-07 15:03:21.146: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] getOracleHomeAttrib: oracle_home = /oracle/app/11.2.0/grid
2014-11-07 15:03:21.265: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 07-NOV-
2014 15:03:21
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)Copyright (c) 1991, 2011, Oracle. All rights reserved.
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)<span style="color:#ff0000;"><span style="color:#ff0000;">TNS-01169: The listener has not recognized the password </span>##檢查報錯(如果用過監聽密碼的人，因該對<span style="color:#ff0000;">TNS-01169</span>不會陌生，如果對監聽器設置了密碼，對監聽進行操作時，需要輸入密碼)
</span>2014-11-07 15:03:21.266: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)Utils:execCmd scls_process_join() uret 1
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] execCmd ret = 1
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 07-NOV-
2014 15:03:21
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)Copyright (c) 1991, 2011, Oracle. All rights reserved.
2014-11-07 15:03:21.267: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.268: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
2014-11-07 15:03:21.268: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)<span style="color:#ff0000;"><span style="color:#ff0000;">TNS-01169: The listener has not recognized the password </span>##檢查報錯
</span>2014-11-07 15:03:21.268: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] (:CLSN00010:)
2014-11-07 15:03:21.268: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] return val (clsagfwret) = 1
2014-11-07 15:03:21.268: [ora.LISTENER.lsnr][4888] {2:2661:31941} [check] LsnrAgent::check }
2014-11-07 15:03:21.269: [ AGFW][2314] {2:2661:31941} <span style="color:#ff0000;"><span style="color:#ff0000;">ora.LISTENER.lsnr bidb4 1 state changed from: STARTING to: OFFLINE </span>##檢查失敗後將"<span style="color:#ff0000;">ora.LISTENER.lsnr</span>"資源狀態改成<span style="color:#ff0000;">OFFLINE </span></span>
三：問題原因
通過上面的分析，可以發現是因為為監聽設置了密碼，然後導致CRS在檢查監聽狀態的時候報錯(TNS-01169)，然後CRS將ora.LISTENER.lsnr資源狀態修改為OFFLINE
四：解決方案
目前還沒有找到解決方案，找到方案後會更新
##注：雖然我們看到crs資源狀態為offline，但是其實監聽運行正常，不會影響業務。
##原創作品，轉載請注明出處！
