If you set the CLUSTERLOG environment variable, the cluster will create a logfile that contains diagnostic information using the path specified. Important events during the operation of the Cluster Service will be logged in this file. Because so many different events occur, the logfile may be somewhat cryptic or hard to read. This document gives some hints about how to read the logfile and information about what items to look for.
Note Each time you attempt to start the Cluster Service, the log will be cleared and a new logfile started. Each component of MSCS that places an entry in the logfile will indicate itself by abbreviation in square brackets. For example, the Node Manager component would be abbreviated [NM]. Logfile entries will vary from one cluster to another. As a result, other logfiles may vary from excerpts referenced in this document.
Note Log entry lines in the following sections have been wrapped for space constraints in this document. The lines do not normally wrap.
Near the beginning of the logfile, notice the build number of MSCS, followed by the operating system version number and service pack level. If you call for support, engineers may ask for this information:
082::14-21:29:26.625 Cluster Service started - Cluster Version 1.224.
082::14-21:29:26.625 OS Version 4.0.1381 - Service Pack 3.
Following the version information, some initialization steps occur. Those steps are followed by an attempt to join the cluster, if one node already exists in a running state. If the Cluster Service could not detect any other cluster members, it will attempt to form the cluster. Consider the following log entries:
0b5::12-20:15:23.531 We're initing Ep...
0b5::12-20:15:23.531 [DM]: Initialization
0b5::12-20:15:23.531 [DM] DmpRestartFlusher: Entry
0b5::12-20:15:23.531 [DM] DmpStartFlusher: Entry
0b5::12-20:15:23.531 [DM] DmpStartFlusher: thread created
0b5::12-20:15:23.531 [NMINIT] Initializing the Node Manager...
0b5::12-20:15:23.546 [NMINIT] Local node name = NODEA.
0b5::12-20:15:23.546 [NMINIT] Local node ID = 1.
0b5::12-20:15:23.546 [NM] Creating object for node 1 (NODEA)
0b5::12-20:15:23.546 [NM] node 1 state 1
0b5::12-20:15:23.546 [NM] Initializing networks.
0b5::12-20:15:23.546 [NM] Initializing network interface facilities.
0b5::12-20:15:23.546 [NMINIT] Initialization complete.
0b5::12-20:15:23.546 [FM] Starting worker thread...
0b5::12-20:15:23.546 [API] Initializing
0a9::12-20:15:23.546 [FM] Worker thread running
0b5::12-20:15:23.546 [lm] :LmInitialize Entry.
0b5::12-20:15:23.546 [lm] :TimerActInitialize Entry.
0b5::12-20:15:23.546 [CS] Initializing RPC server.
0b5::12-20:15:23.609 [INIT] Attempting to join cluster MDLCLUSTER
0b5::12-20:15:23.609 [JOIN] Spawning thread to connect to sponsor
192.88.80.114
06c::12-20:15:23.609 [JOIN] Asking 192.88.80.114 to sponsor us.
0b5::12-20:15:23.609 [JOIN] Waiting for all connect threads to terminate.
06c::12-20:15:32.750 [JOIN] Sponsor 192.88.80.114 is not available,
status=1722.
0b5::12-20:15:32.750 [JOIN] All connect threads have terminated.
0b5::12-20:15:32.750 [JOIN] Unable to connect to any sponsor node.
0b5::12-20:15:32.750 [INIT] Failed to join cluster, status 53
0b5::12-20:15:32.750 [INIT] Attempting to form cluster MDLCLUSTER
0b5::12-20:15:32.750 [Ep]: EpInitPhase1
0b5::12-20:15:32.750 [API] Online read only
04b::12-20:15:32.765 [RM] Main: Initializing.
Note that the cluster service attempts to join the cluster. If it cannot connect with an existing member, the software decides to form the cluster. The next series of steps attempts to form groups and resources necessary to accomplish this task. It is important to note that the cluster service must arbitrate control of the quorum disk.
0b5::12-20:15:32.781 [FM] Creating group a1a13a86-0eaf-11d1-8427-
0000f8034599
0b5::12-20:15:32.781 [FM] Group a1a13a86-0eaf-11d1-8427-0000f8034599
contains a1a13a87-0eaf-11d1-8427-0000f8034599.
0b5::12-20:15:32.781 [FM] Creating resource a1a13a87-0eaf-11d1-8427-
0000f8034599
0b5::12-20:15:32.781 [FM] FmpAddPossibleEntry adding 1 to a1a13a87-0eaf-
11d1-8427-0000f8034599 possible node list
0b5::12-20:15:32.781 [FMX] Found the quorum resource a1a13a87-0eaf-11d1-
8427-0000f8034599.
0b5::12-20:15:32.781 [FM] All dependencies for a1a13a87-0eaf-11d1-8427-
0000f8034599 created
0b5::12-20:15:32.781 [FM] arbitrate for quorum resource id a1a13a87-0eaf-
11d1-8427-0000f8034599.
0b5::12-20:15:32.781 FmpRmCreateResource: creating resource a1a13a87-0eaf-
11d1-8427-0000f8034599 in shared resource monitor
0b5::12-20:15:32.812 FmpRmCreateResource: created resource a1a13a87-0eaf-
11d1-8427-0000f8034599, resid 1363016
0dc::12-20:15:32.828 Physical Disk <Disk D:>: Arbitrate returned status 0.
0b5::12-20:15:32.828 [FM] FmGetQuorumResource successful
0b5::12-20:15:32.828 FmpRmOnlineResource: bringing resource a1a13a87-0eaf-
11d1-8427-0000f8034599 (resid 1363016) online.
0b5::12-20:15:32.843 [CP] CppResourceNotify for resource Disk D:
0b5::12-20:15:32.843 [GUM] GumSendUpdate: Locker waiting
type 0 context 8
0b5::12-20:15:32.843 [GUM] Thread 0xb5 UpdateLock wait on Type 0
0b5::12-20:15:32.843 [GUM] DoLockingUpdate successful, lock granted to 1
0b5::12-20:15:32.843 [GUM] GumSendUpdate: Locker dispatching seq 388
type 0 context 8
0b5::12-20:15:32.843 [GUM] GumpDoUnlockingUpdate releasing lock ownership
0b5::12-20:15:32.843 [GUM] GumSendUpdate: completed update seq 388
type 0 context 8
0b5::12-20:15:32.843 [GUM] GumSendUpdate: Locker waiting
type 0 context 9
0b5::12-20:15:32.843 [GUM] Thread 0xb5 UpdateLock wait on Type 0
0b5::12-20:15:32.843 [GUM] DoLockingUpdate successful, lock granted to 1
0b5::12-20:15:32.843 [GUM] GumSendUpdate: Locker dispatching seq 389
type 0 context 9
0b5::12-20:15:32.843 [GUM] GumpDoUnlockingUpdate releasing lock ownership
0b5::12-20:15:32.843 [GUM] GumSendUpdate: completed update seq 389
type 0 context 9
0b5::12-20:15:32.843 FmpRmOnlineResource: Resource a1a13a87-0eaf-11d1-
8427-0000f8034599 pending
0e1::12-20:15:33.359 Physical Disk <Disk D:>: Online, created registry
watcher thread.
090::12-20:15:33.359 [FM] NotifyCallBackRoutine: enqueuing event
04d::12-20:15:33.359 [FM] WorkerThread, processing transition event for
a1a13a87-0eaf-11d1-8427-0000f8034599, oldState = 129, newState = 2.
04d::12-20:15:33.359 [FM] HandleResourceTransition: Resource Name =
a1a13a87-0eaf-11d1-8427-0000f8034599 old state=129 new state=2
04d::12-20:15:33.359 [DM] DmpQuoObjNotifyCb: Quorum resource is online
04d::12-20:15:33.375 [DM] DmpQuoObjNotifyCb: Own quorum resource, try open
the quorum log
04d::12-20:15:33.375 [DM] DmpQuoObjNotifyCb: the name of the quorum file
is D:\MSCS\quolog.log
04d::12-20:15:33.375 [lm] LogCreate : Entry FileName=D:\MSCS\quolog.log
MaxFileSize=0x00010000
04d::12-20:15:33.375 [lm] LogpCreate : Entry
In this case, the node forms the cluster group and quorum disk resource, gains control of the disk, and opens the quorum logfile. From here, the cluster performs operations with the logfile, and proceeds to form the cluster. This involves configuring network interfaces and bringing them online.
0b5::12-20:15:33.718 [NM] Beginning form process.
0b5::12-20:15:33.718 [NM] Synchronizing node information.
0b5::12-20:15:33.718 [NM] Creating node objects.
0b5::12-20:15:33.718 [NM] Configuring networks & interfaces.
0b5::12-20:15:33.718 [NM] Synchronizing network information.
0b5::12-20:15:33.718 [NM] Synchronizing interface information.
0b5::12-20:15:33.718 [dm] DmBeginLocalUpdate Entry
0b5::12-20:15:33.718 [dm] DmBeginLocalUpdate Exit, pLocalXsaction=0x00151c20 dwError=0x00000000
0b5::12-20:15:33.718 [NM] Setting database entry for interface a1a13a7f-
0eaf-11d1-8427-0000f8034599
0b5::12-20:15:33.718 [dm] DmCommitLocalUpdate Entry
0b5::12-20:15:33.718 [dm] DmCommitLocalUpdate Exit, dwError=0x00000000
0b5::12-20:15:33.718 [dm] DmBeginLocalUpdate Entry
0b5::12-20:15:33.875 [dm] DmBeginLocalUpdate Exit,
pLocalXsaction=0x00151c20 dwError=0x00000000
0b5::12-20:15:33.875 [NM] Setting database entry for interface a1a13a81-
0eaf-11d1-8427-0000f8034599
0b5::12-20:15:33.875 [dm] DmCommitLocalUpdate Entry
0b5::12-20:15:33.875 [dm] DmCommitLocalUpdate Exit, dwError=0x00000000
0b5::12-20:15:33.875 [NM] Matched 2 networks, created 0 new networks.
0b5::12-20:15:33.875 [NM] Resynchronizing network information.
0b5::12-20:15:33.875 [NM] Resynchronizing interface information.
0b5::12-20:15:33.875 [NM] Creating network objects.
0b5::12-20:15:33.875 [NM] Creating object for network a1a13a7e-0eaf-11d1-
8427-0000f8034599
0b5::12-20:15:33.875 [NM] Creating object for network a1a13a80-0eaf-11d1-
8427-0000f8034599
0b5::12-20:15:33.875 [NM] Creating interface objects.
0b5::12-20:15:33.875 [NM] Creating object for interface a1a13a7f-0eaf-
11d1-8427-0000f8034599.
0b5::12-20:15:33.875 [NM] Registering network a1a13a7e-0eaf-11d1-8427-
0000f8034599 with cluster transport.
0b5::12-20:15:33.875 [NM] Registering interfaces for network a1a13a7e-
0eaf-11d1-8427-0000f8034599 with cluster transport.
0b5::12-20:15:33.875 [NM] Registering interface a1a13a7f-0eaf-11d1-8427-
0000f8034599 with cluster transport, addr 9.9.9.2, endpoint 3003.
0b5::12-20:15:33.890 [NM] Instructing cluster transport to bring network
a1a13a7e-0eaf-11d1-8427-0000f8034599 online.
0b5::12-20:15:33.890 [NM] Creating object for interface a1a13a81-0eaf-
11d1-8427-0000f8034599.
0b5::12-20:15:33.890 [NM] Registering network a1a13a80-0eaf-11d1-8427-
0000f8034599 with cluster transport.
0b5::12-20:15:33.890 [NM] Registering interfaces for network a1a13a80-
0eaf-11d1-8427-0000f8034599 with cluster transport.
0b5::12-20:15:33.890 [NM] Registering interface a1a13a81-0eaf-11d1-8427-
0000f8034599 with cluster transport, addr 192.88.80.190, endpoint 3003.
0b5::12-20:15:33.890 [NM] Instructing cluster transport to bring network
a1a13a80-0eaf-11d1-8427-0000f8034599 online.
After initializing network interfaces, the cluster will continue formation with the enumeration of cluster nodes. In this case, as a newly formed cluster, the cluster will contain only one node. If this session had been joining an existing cluster, the node enumeration would show two nodes. Next, the cluster will bring the Cluster IP address and Cluster Name resources online.
0b5::12-20:15:34.015 [FM] OnlineGroup: setting group state to Online for
f901aa29-0eaf-11d1-8427-0000f8034599
069::12-20:15:34.015 IP address <Cluster IP address>: Created NBT
interface \Device\NetBt_If6 (instance 355833456).
0b5::12-20:15:34.015 [FM] FmpAddPossibleEntry adding 1 to a1a13a87-0eaf-
11d1-8427-0000f8034599 possible node list
0b5::12-20:15:34.015 [FM] FmFormNewClusterPhase2 complete.
.
.
.
0b5::12-20:15:34.281 [INIT] Successfully formed a cluster.
09c::12-20:15:34.281 [lm] :ReSyncTimerHandles Entry.
09c::12-20:15:34.281 [lm] :ReSyncTimerHandles Exit gdwNumHandles=3
0b5::12-20:15:34.281 [INIT] Cluster Started! Original Min WS is 204800,
Max WS is 1413120.
08c::12-20:15:34.296 [CPROXY] clussvc initialized
069::12-20:15:40.421 IP address <Cluster IP Address>: IP Address
192.88.80.114 on adapter DC21X41 online
.
.
.
04d::12-20:15:40.421 [FM] OnlineWaitingTree, a1a13a84-0eaf-11d1-8427-
0000f8034599 depends on a1a13a83-0eaf-11d1-8427-0000f8034599. Start first
04d::12-20:15:40.421 [FM] OnlineWaitingTree, Start resource a1a13a84-0eaf-
11d1-8427-0000f8034599
04d::12-20:15:40.421 [FM] OnlineResource: a1a13a84-0eaf-11d1-8427-
0000f8034599 depends on a1a13a83-0eaf-11d1-8427-0000f8034599. Bring online
first.
04d::12-20:15:40.421 FmpRmOnlineResource: bringing resource a1a13a84-0eaf-
11d1-8427-0000f8034599 (resid 1391032) online.
04d::12-20:15:40.421 [CP] CppResourceNotify for resource Cluster Name
04d::12-20:15:40.421 [GUM] GumSendUpdate: Locker waiting
type 0 context 8
04d::12-20:15:40.437 [GUM] Thread 0x4d UpdateLock wait on Type 0
04d::12-20:15:40.437 [GUM] DoLockingUpdate successful, lock granted to 1
076::12-20:15:40.437 Network Name <Cluster Name>: Bringing resource
online...
04d::12-20:15:40.437 [GUM] GumSendUpdate: Locker dispatching seq 411
type 0 context 8
04d::12-20:15:40.437 [GUM] GumpDoUnlockingUpdate releasing lock ownership
04d::12-20:15:40.437 [GUM] GumSendUpdate: completed update seq 411
type 0 context 8
04d::12-20:15:40.437 [GUM] GumSendUpdate: Locker waiting
type 0 context 11
.
.
.
076::12-20:15:43.515 Network Name <Cluster Name>: Registered server name
MDLCLUSTER on transport \Device\NetBt_If6.
076::12-20:15:46.578 Network Name <Cluster Name>: Registered workstation
name MDLCLUSTER on transport \Device\NetBt_If6.
076::12-20:15:46.578 Network Name <Cluster Name>: Network Name MDLCLUSTER
is now online
Following these steps, the cluster will attempt to bring other resources and groups online. The logfile will continue to increase in size as the cluster service runs. Therefore, it may be a good idea to enable this option when you are having problems, rather than leaving it on for days or weeks at a time.
After reviewing a successful startup of the Cluster Service, you may want to examine some errors that may appear because of various failures. The following examples illustrate possible log entries for four different failures.
If the cluster attempts to form and cannot connect to the quorum disk, entries similar to the following may appear in the logfile. Because of the failure, the cluster cannot form, and the Cluster Service terminates.
0b9::14-20:59:42.921 [RM] Main: Initializing.
08f::14-20:59:42.937 [FM] Creating group a1a13a86-0eaf-11d1-8427-
0000f8034599
08f::14-20:59:42.937 [FM] Group a1a13a86-0eaf-11d1-8427-0000f8034599
contains a1a13a87-0eaf-11d1-8427-0000f8034599.
08f::14-20:59:42.937 [FM] Creating resource a1a13a87-0eaf-11d1-8427-
0000f8034599
08f::14-20:59:42.937 [FM] FmpAddPossibleEntry adding 1 to a1a13a87-0eaf-
11d1-8427-0000f8034599 possible node list
08f::14-20:59:42.937 [FMX] Found the quorum resource a1a13a87-0eaf-11d1-
8427-0000f8034599.
08f::14-20:59:42.937 [FM] All dependencies for a1a13a87-0eaf-11d1-8427-
0000f8034599 created
08f::14-20:59:42.937 [FM] arbitrate for quorum resource id a1a13a87-0eaf-
11d1-8427-0000f8034599.
08f::14-20:59:42.937 FmpRmCreateResource: creating resource a1a13a87-0eaf-
11d1-8427-0000f8034599 in shared resource monitor
08f::14-20:59:42.968 FmpRmCreateResource: created resource a1a13a87-0eaf-
11d1-8427-0000f8034599, resid 1362616
0e9::14-20:59:43.765 Physical Disk <Disk D:>: SCSI, error reserving disk,
error 21.
0e9::14-20:59:54.125 Physical Disk <Disk D:>: SCSI, error reserving disk,
error 21.
0e9::14-20:59:54.140 Physical Disk <Disk D:>: Arbitrate returned status
21.
08f::14-20:59:54.140 [FM] FmGetQuorumResource failed, error 21.
08f::14-20:59:54.140 [INIT] Cleaning up failed form attempt.
08f::14-20:59:54.140 [INIT] Failed to form cluster, status 3213068.
08f::14-20:59:54.140 [CS] ClusterInitialize failed 21
08f::14-20:59:54.140 [INIT] The cluster service is shutting down.
08f::14-20:59:54.140 [evt] EvShutdown
08f::14-20:59:54.140 [FM] Shutdown: Failover Manager requested to shutdown
groups.
08f::14-20:59:54.140 [FM] DestroyGroup: destroying a1a13a86-0eaf-11d1-
8427-0000f8034599
08f::14-20:59:54.140 [FM] DestroyResource: destroying a1a13a87-0eaf-11d1-
8427-0000f8034599
08f::14-20:59:54.140 [OM] Deleting object Physical Disk
08f::14-20:59:54.140 [FM] Resource a1a13a87-0eaf-11d1-8427-0000f8034599
destroyed.
08f::14-20:59:54.140 [FM] Group a1a13a86-0eaf-11d1-8427-0000f8034599
destroyed.
08f::14-20:59:54.140 [Dm] DmShutdown
08f::14-20:59:54.140 [DM] DmpShutdownFlusher: Entry
08f::14-20:59:54.156 [DM] DmpShutdownFlusher: Setting event
062::14-20:59:54.156 [DM] DmpRegistryFlusher: got 0
062::14-20:59:54.156 [DM] DmpRegistryFlusher: exiting
0ca::14-20:59:54.156 [FM] WorkItem, delete resource <Disk D:> status 0
0ca::14-20:59:54.156 [OM] Deleting object Disk Group 1 (a1a13a86-0eaf-
11d1-8427-0000f8034599)
0e7::14-20:59:54.375 [CPROXY] clussvc terminated, error 0.
0e7::14-20:59:54.375 [CPROXY] Service Stopping...
0b9::14-20:59:54.375 [RM] Going away, Status = 1, Shutdown = 0.
02c::14-20:59:54.375 [RM] PollerThread stopping. Shutdown = 1, Status = 0,
WaitFailed = 0, NotifyEvent address = 196.
0e7::14-20:59:54.375 [CPROXY] Cleaning up
0b9::14-20:59:54.375 [RM] RundownResources posting shutdown notification.
0e7::14-20:59:54.375 [CPROXY] Cleanup complete.
0e3::14-20:59:54.375 [RM] NotifyChanges shutting down.
0e7::14-20:59:54.375 [CPROXY] Service Stopped.
Perhaps the most meaningful lines from these are
0e9::14-20:59:43.765 Physical Disk <Disk D:>: SCSI, error reserving disk,
error 21.
0e9::14-20:59:54.125 Physical Disk <Disk D:>: SCSI, error reserving disk,
error 21.
0e9::14-20:59:54.140 Physical Disk <Disk D:>: Arbitrate returned status
21.
Note The error code on these logfile entries is 21. You can issue net helpmsg 21 from the command line and receive the explanation of the error status code. Status code 21 means "The device is not ready." This indicates a possible problem with the device. In this case, the device was turned off, and the error status correctly indicates the problem.
In this example, the drive has failed or has been reformatted from the SCSI controller. As a result, the cluster service cannot locate a drive with the specific signature it is looking for.
0b8::14-21:11:46.515 [RM] Main: Initializing.
074::14-21:11:46.531 [FM] Creating group a1a13a86-0eaf-11d1-8427-
0000f8034599
074::14-21:11:46.531 [FM] Group a1a13a86-0eaf-11d1-8427-0000f8034599
contains a1a13a87-0eaf-11d1-8427-0000f8034599.
074::14-21:11:46.531 [FM] Creating resource a1a13a87-0eaf-11d1-8427-
0000f8034599
074::14-21:11:46.531 [FM] FmpAddPossibleEntry adding 1 to a1a13a87-0eaf-
11d1-8427-0000f8034599 possible node list
074::14-21:11:46.531 [FMX] Found the quorum resource a1a13a87-0eaf-11d1-
8427-0000f8034599.
074::14-21:11:46.531 [FM] All dependencies for a1a13a87-0eaf-11d1-8427-
0000f8034599 created
074::14-21:11:46.531 [FM] arbitrate for quorum resource id a1a13a87-0eaf-
11d1-8427-0000f8034599.
074::14-21:11:46.531 FmpRmCreateResource: creating resource a1a13a87-0eaf-
11d1-8427-0000f8034599 in shared resource monitor
074::14-21:11:46.562 FmpRmCreateResource: created resource a1a13a87-0eaf-
11d1-8427-0000f8034599, resid 1362696
075::14-21:11:46.671 Physical Disk <Disk D:>: SCSI,
Performing bus rescan.
075::14-21:11:51.843 Physical Disk <Disk D:>: SCSI, error attaching to
signature 71cd0549, error 2.
075::14-21:11:51.843 Physical Disk <Disk D:>: Unable to attach to
signature 71cd0549. Error: 2.
074::14-21:11:51.859 [FM] FmGetQuorumResource failed, error 2.
074::14-21:11:51.859 [INIT] Cleaning up failed form attempt.
In this case, the most important logfile entries are
075::14-21:11:51.843 Physical Disk <Disk D:>: SCSI,
error attaching to signature 71cd0549, error 2.
075::14-21:11:51.843 Physical Disk <Disk D:>: Unable to attach to
signature 71cd0549. Error: 2.
Status code 2 means "The system cannot find the file specified." The error in this case may mean that it cannot find the disk, or that, because of some kind of problem, it cannot locate the quorum logfile that should be on the disk.
If another computer on the network has the same IP address as the cluster IP address resource, the resource will be prevented from going online. Further, the cluster name will not be registered on the network, as it depends on the IP address resource. Because this name is the network name used for cluster administration, you will not be able to administer the cluster using this name, in this type of failure. However, you may be able to use the computer name of the cluster node to connect with Cluster Administrator. Additionally, you may be able to connect locally from the console using the loopback address. The following sample entries are from a cluster logfile during this type of failure:
0b9::14-21:32:59.968 IP Address <Cluster IP Address>: The IP address is
already in use on the network, status 5057.
0d2::14-21:32:59.984 [FM] NotifyCallBackRoutine: enqueuing event
03e::14-21:32:59.984 [FM] WorkerThread, processing transition event for
a1a13a83-0eaf-11d1-8427-0000f8034599, oldState = 129, newState = 4.03e
.
.
.
03e::14-21:32:59.984 FmpHandleResourceFailure: taking resource a1a13a83-
0eaf-11d1-8427-0000f8034599 and dependents offline
03e::14-21:32:59.984 [FM] TerminateResource: a1a13a84-0eaf-11d1-8427-
0000f8034599 depends on a1a13a83-0eaf-11d1-8427-0000f8034599. Terminating
first
0d3::14-21:32:59.984 Network Name <Cluster Name>: Terminating name
MDLCLUSTER...
0d3::14-21:32:59.984 Network Name <Cluster Name>: Name MDLCLUSTER is
already offline.
.
.
.
03e::14-21:33:00.000 FmpRmTerminateResource: a1a13a84-0eaf-11d1-8427-0000f8034599 is now offline
0c7::14-21:33:00.000 IP Address <Cluster IP Address>: Terminating
resource...
0c7::14-21:33:00.000 IP Address <Cluster IP Address>: Address
192.88.80.114 on adapter DC21X41 offline.
If you evict a node from a cluster, the cluster software on that node must be reinstalled to gain access to the cluster again. If you start the evicted node, and the Cluster Service attempts to join the cluster, entries similar to the following may appear in the cluster logfile:
032::26-16:11:45.109 [INIT] Attempting to join cluster MDLCLUSTER
032::26-16:11:45.109 [JOIN] Spawning thread to connect to sponsor
192.88.80.115
040::26-16:11:45.109 [JOIN] Asking 192.88.80.115 to sponsor us.
032::26-16:11:45.109 [JOIN] Spawning thread to connect to sponsor 9.9.9.2
032::26-16:11:45.109 [JOIN] Spawning thread to connect to sponsor
192.88.80.190
099::26-16:11:45.109 [JOIN] Asking 9.9.9.2 to sponsor us.
032::26-16:11:45.109 [JOIN] Spawning thread to connect to sponsor NODEA
098::26-16:11:45.109 [JOIN] Asking 192.88.80.190 to sponsor us.
032::26-16:11:45.125 [JOIN] Waiting for all connect threads to terminate.
092::26-16:11:45.125 [JOIN] Asking NODEA to sponsor us.
040::26-16:12:18.640 [JOIN] Sponsor 192.88.80.115 is not available
(JoinVersion), status=1722.
098::26-16:12:18.640 [JOIN] Sponsor 192.88.80.190 is not available
(JoinVersion), status=1722.
099::26-16:12:18.640 [JOIN] Sponsor 9.9.9.2 is not available
(JoinVersion), status=1722.
098::26-16:12:18.640 [JOIN] JoinVersion data for sponsor 157.57.224.190 is
invalid, status 1722.
099::26-16:12:18.640 [JOIN] JoinVersion data for sponsor 9.9.9.2 is
invalid, status 1722.
040::26-16:12:18.640 [JOIN] JoinVersion data for sponsor 157.58.80.115 is
invalid, status 1722.
092::26-16:12:18.703 [JOIN] Sponsor NODEA is not available (JoinVersion),
status=1722.
092::26-16:12:18.703 [JOIN] JoinVersion data for sponsor NODEA is invalid,
status 1722.
032::26-16:12:18.703 [JOIN] All connect threads have terminated.
032::26-16:12:18.703 [JOIN] Unable to connect to any sponsor node.
032::26-16:12:18.703 [INIT] Failed to join cluster, status 0
032::26-16:12:18.703 [INIT] Attempting to form cluster MDLCLUSTER
.
.
.
032::26-16:12:18.734 [FM] arbitrate for quorum resource id 24acc093-1e28-
11d1-9e5d-0000f8034599.
032::26-16:12:18.734 [FM] FmpQueryResourceInfo:initialize the resource
with the registry information
032::26-16:12:18.734 FmpRmCreateResource: creating resource 24acc093-1e28-
11d1-9e5d-0000f8034599 in shared resource monitor
032::26-16:12:18.765 FmpRmCreateResource: created resource 24acc093-1e28-
11d1-9e5d-0000f8034599, resid 1360000
06d::26-16:12:18.812 Physical Disk <Disk G:>: SCSI,
error attaching to signature b2320a9b, error 2.
06d::26-16:12:18.812 Physical Disk <Disk G:>: Unable to attach to
signature b2320a9b. Error: 2.
032::26-16:12:18.812 [FM] FmGetQuorumResource failed, error 2.
032::26-16:12:18.812 [INIT] Cleaning up failed form attempt.
032::26-16:12:18.812 [INIT] Failed to form cluster, status 2.
032::26-16:12:18.828 [CS] ClusterInitialize failed 2
The node attempts to join the existing cluster, but has invalid credentials, because it was previously evicted. Therefore, the existing node refuses to communicate with it. The node may attempt to form its own version of the cluster, but cannot gain control of the quorum disk, because the existing cluster node maintains ownership. Examination of the logfile on the existing cluster node reveals that the Cluster Service posted entries to reflect the failed attempt to join:
0c4::29-18:13:31.035 [NMJOIN] Processing request by node 2 to begin
joining.
0c4::29-18:13:31.035 [NMJOIN] Node 2 is not a member of this cluster.
Cannot join.