I/O Buffer Size

Every time an application asks to send or receive data, the system must undergo a certain amount of overhead to check the parameters and buffers specified to the call. These checks cost cycles, so it is best to do as few I/O calls as possible to send the necessary amount of data.

The easiest way to minimize the number of I/O calls is to give each call a big chunk of data. For example, if you need to send 128KB of data, don't do 128 calls to send() with 1K in each call. Instead, do just a couple of calls with 32K or 64K of data. This way you get the data transferred with fewer I/O calls.

However, it is best not to specify too much data to any one I/O call because the data must be locked down in physical memory in order to be used. High performance applications and services use buffer sizes in the range of 8K to 64K to balance between the physical memory cost of the locked-down buffer and the CPU overhead of each I/O call. The best way to determine a specific number for your application is experimentation.

Another problem with small I/O is the effect of Nagling. Nagling, which occurs only with TCP/IP, is an attempt by the transport protocol to coalesce small buffers into larger buffers so that there are not a lot of tiny packets on the physical network. While Nagling helps to reduce small packets on the network, it does it by delaying sends until there is more data available to send. With small buffers passed to send() these delays can significantly impact performance, slowing down the application by as much as 10,000%.