Diagnoses and Treatment of Black Hole Routers

• Microsoft Windows NT Workstation versions 3.5 3.51 and 4.0
• Microsoft Windows NT Server versions 3.5, 3.51 and 4.0

SUMMARY

On a TCP/IP wide area network (WAN), communication over some routes may fail if intermediate network segments have packet sizes smaller than the communicating hosts, and routers do not send appropriate ICMP responses to this condition. A router that causes this condition is sometimes known as a "black hole" router. The Ping utility, a standard utility installed with the Microsoft Windows NT TCP/IP protocol, can be used to find black hole routers. Some recommendations are provided to work around or fix problems with black hole routers.

MORE INFORMATION

When a network router receives a packet larger than the Maximum Transfer Unit (MTU) of the next network segment, and that packet's IP layer "don’t fragment" bit is flagged, the router should send an ICMP destination unreachable message back to the sending host. When this does not happen, packets can be dropped, causing a variety of errors that will vary with the application that is communicating over the failed link. These errors will not occur when an application connects to a computer on a local subnet. The problem may seem intermittent, but on closer examination, it can be duplicated, such as in having a client read a large file from a remote host.

The Ping utility can be used to find black hole routers by using the -f (do not fragment) and the -l (buffer size) parameters. Setting the -f parameter will cause the Ping utility to send an ICMP echo packet with the IP "do not fragment" bit set. The -l parameter sets the buffer, or payload, size of the ICMP Echo packet. The largest buffer that can be sent unfragmented equals the MTU minus the IP and ICMP headers (MTU-28) of the smallest MTU along a route. For example, because Ethernet has an MTU of 1500 bytes, under the best circumstances, Ping could echo an unfragmented packet with an ICMP buffer of 1472 bytes. The syntax for Ping in this case would be:

   Ping <machine name or IP address> -f -l 1472

By changing the -l parameter on successive Pings, the largest unfragmented packet that will travel a specific route can be found. The smallest MTU in general use is 576 bytes, so you should be able to safely start with an ICMP buffer of 548, then work up from there. For example, if "Ping <host name or address> -f -l 972" returns packets and "Ping <host name or address> -f -l 973" fails, the largest MTU that can be used over that route is 1000 (972+28).

To fix or work around black hole routers, there are four possible solutions.

1. Enable PMTU Black Hole Detection on Windows NT hosts that will be communicating over a wide area connection, as documented in Knowledge Base article Q136970. In this case, Windows NT 3.51 Service Pack 2 or greater or Windows NT 4.0 should be used.

2. Configure intermediate routers to send ICMP type 3 code 4 (destination unreachable don't fragment (DF) bit sent and fragmentation required) messages. This may require upgrading router software or firmware, router configuration or router replacement.

3. Disable PMTU discovery on Windows NT hosts that communicate over troublesome routes. This will configure the default MTU to 576 bytes. This could cause significant degradation in network performance.

4. Set the MTU of the host interface to be the largest the black hole router can handle. This guarantees the largest possible packet size will be sent over that connection, but will cause local traffic, and traffic over routed connections without problems, to use smaller packets than they would otherwise. This workaround assumes that you have determined the MTU and the state of all possible links that could be used by the host in question.

   ARTICLE-ID: Q120642
   TITLE     : TCP/IP & NBT Configuration Parameters for Windows NT 3.5

   ARTICLE-ID: Q128797
   TITLE     : Unable to Transfer Files Across DEC 250 and DEC 500 Routers

   ARTICLE-ID: Q136970
   TITLE     : PMTU Black Hole Detection Algorithm Change for Windows NT
               3.51

   ARTICLE-ID: Q138575
   TITLE     : Communication Fails Through Ethernet Segment Between FDDI
               Rings

   ARTICLE-ID: Q140375
   TITLE     : Default MTU Size for Different Network Topology