Up till know, the discussion on the benefits of increasing the MTU above the standard 1,500 bytes limit has been limited to the local area network. We now proceed to investigate whether bottlenecks at higher layers can benefit from an Internet wide increase of the MTU. More specifically, we focus on the performance benefits such an increase would have on the backbone of reliable connection oriented applications - TCP.

TCP, extensively researched and modeled, has been shown to have an upper bound throughput limit formulated by the following:



Where:

• MSS - maximum segment size. This is the maximum amount of data the receiving end can handle in one piece. This is determined by the available space in the receiving buffer, and the maximum MTU acceptable on both communicating parties. In most cases, MSS equals MTU minus the header overhead, which is 40 bytes.
• RTT - The average round trip time of packets, from packet transmission to packet acknowledgement.
• Packet_Loss_Rate - the rate at which packets have to be retransmitted due to expiration of TCP timers (At which a loss event is recorded).

It can easily be seen that MTU seems to dominate the throughput through linear dependency. Packet loss rate may increase as MTU is increased, but it does so at a sub-linear rate, and has the inverse square effect on the throughput.

While round trip time and packet loss rate are rather small at the local area network, they seem to grow erratic as the packet traverses a wide area network. Moreover, there is little the source system can do to remedy this. The only way to enhance WAN performance seems to be an increase in MTU.

To illustrate throughput improvement on MTU increase, we take a TCP connection form New-York to Los-Angeles having an RTT of 40msec and a loss rate of 0.01%. Considering an MTU of 1,500 bytes, this connection would have a throughput bound of 6.5 Mbits/second. Considering an MTU of 9,000 bytes, as most jumbo frame recommendations suggest, TCP throughput would be upper bound by 40 Mbits/second.

Another strong result from TCP throughput bound comes from its relation with the packet loss rate. Should we set out to achieve a certain throughput using 1,500 byte frames, we would need a packet loss rate 36 times better than would be by using 9,000 byte frames and achieving the same throughput. This alone should be a great encouragement for using jumbo size MTU on most last-mile wireless connections.

If greater encouragement is needed for an increasing MTU to enhance Internet wide performance, nothing beats the economy of such an increase. ISPs are at a constant race to recruit new customers, while maintaining the levels of service to avoid customer churn. One of the most common methods of maintaining these levels of service would be upgrading the ISP Network Access Points (NAP) - devices where many local area and wide area networks intersect. Enhanced performance on these NAPs has a potential of relieving bottlenecks. Replacing a NAP based on 4,000 bytes MTU FDDI with a NAP based on 9,000 bytes ATM OC3 would yield a throughput increase slightly greater than 50%, not justifying the expense. The cost of a 50 port ATM OC12 switch (The more ports the better at a NAP) would be outrageous, compared with the cost of a 64 port 9KB MTU Gigabit Ethernet capable of delivering the same 50% more throughput per port at about a 1/3 of the cost. This solution enables the old FDDI 4KB packets to continue and be used, while only slightly reducing throughput for IP over ATM MTU of 9,180 bytes.

To summarize, transporting beyond the local area network requires large MTU to improve throughput. Core Internet infrastructure should be particularly careful not to limit the MTU to the 10Mbits Ethernet 1,500 byte limit.