Oracle RAC database instance startup exception analysis IPC send timeout

In the near future , A user is restarting RAC Database instance of a node , Found that the startup speed is very slow . At the same time, the business department feedback connection RAC The business of the surviving node is also affected .

Through the analysis of logs , When starting the database ,Reconfiguration Slow speed ,Reconfiguration Report a mistake later IPC Send timeout detected. Sender: ospid 53884 [oracle@test2 (LMD0)], Thus, the node expulsion of database instance group appears ;Wed Apr 13 19:28:02 2022
Instance termination initiated by instance 2 with reason 1.
Instance 2 received a reconfig event from its cluster manager indicating that this instance is supposed to be down
Please check instance 2's alert log and LMON trace file for more details.
Please also examine the CSS log files.
LMON (ospid: 47523): terminating the instance due to error 481

therefore , There is a problem in the process of troubleshooting IPC Send timeout Why ; There are BUG There may be RAC Node load reason , You can refer to MOS The troubleshooting steps in the document check the information of the system item by item ：

Instance Evicted After LMON to LMON IPC Send timeout Due to Storage Issue (Doc ID 2080029.1)
    "ipc send timeout" Precedes Database Instance Crash or Eviction (Doc ID 1951216.1)
    While Evicting One of the Instance, the Remaining instances Terminated by LMON with "LMON is running too slowly and in the middle of reconfiguration" (Doc ID 1949505.1)

The relevant logs are as follows ：

1.
2022-04-13 18:57:29.215  node 1 The cluster software was restarted manually ,
 The database instance was also started successfully ,
Wed Apr 13 18:58:32 2022
QMNC started with pid=100, OS id=52025 
Completed: ALTER DATABASE OPEN /* db agent *//* {1:49652:2} */


2. node 2 RECONFIG Nodes in process 1 abnormal 
-- node 2
Wed Apr 13 19:22:26 2022
Starting ORACLE instance (normal)

-- node 1：
Wed Apr 13 19:28:00 2022
IPC Send timeout detected. Receiver ospid 47526 [
Wed Apr 13 19:28:00 2022
Errors in file /oracle/app/diag/rdbms/testnew/test1/trace/test1_lmd0_47526.trc:
Wed Apr 13 19:28:02 2022
Instance termination initiated by instance 2 with reason 1.
Instance 2 received a reconfig event from its cluster manager indicating that this instance is supposed to be down
Please check instance 2's alert log and LMON trace file for more details.
Please also examine the CSS log files.
LMON (ospid: 47523): terminating the instance due to error 481
System state dump requested by (instance=1, osid=47523 (LMON)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/diag/rdbms/testnew/test1/trace/test1_diag_47507_20220413192802.trc
Wed Apr 13 19:28:03 2022
ORA-1092 : opitsk aborting process
Instance terminated by LMON, pid = 47523

-- node 2：
Wed Apr 13 19:28:00 2022
IPC Send timeout detected. Sender: ospid 53884 [oracle@test2 (LMD0)]
Receiver: inst 1 binc 429458022 ospid 47526
IPC Send timeout to 1.0 inc 4 for msg type 65521 from opid 11
Wed Apr 13 19:28:02 2022
Communications reconfiguration: instance_number 1
Wed Apr 13 19:28:02 2022
Dumping diagnostic data in directory=[cdmp_20220413192802], requested by (instance=1, osid=47523 (LMON)), summary=[abnormal instance termination].
Reconfiguration started (old inc 4, new inc 8)




#############################
3. Node to check 2 start-up ,Reconfiguration In the process ,IPC Send timeout  Why -- This is also the node 2 The reason why it feels slow when starting manually  ; At the same time node 1 stay 19:34 Start the times ORA-00240 error  , We should comprehensively check the network and storage conditions at that time and the load of nodes , Reference resources MOS On file .

Wed Apr 13 19:34:49 2022
Errors in file /oracle/app/diag/rdbms/testnew/test1/trace/test1_dbw0_247773.trc  (incident=168173):
ORA-00240: control file enqueue held for more than 120 seconds

Instance Evicted After LMON to LMON IPC Send timeout Due to Storage Issue (Doc ID 2080029.1)
	"ipc send timeout" Precedes Database Instance Crash or Eviction (Doc ID 1951216.1)
	While Evicting One of the Instance, the Remaining instances Terminated by LMON with "LMON is running too slowly and in the middle of reconfiguration" (Doc ID 1949505.1)