Asynchronous Disk I/O on Windows NT Appears as Synchronous

ID: Q156932

The information in this article applies to:

Microsoft Win32 Application Programming Interface (API), used with:
- Microsoft Windows NT versions 3.51, 4.0
Microsoft Windows 2000

SUMMARY

File I/O on Windows NT can be synchronous or asynchronous. The default behavior for I/O is synchronous: an I/O function is called and returns when the I/O is complete. Asynchronous I/O, on the other hand, allows an I/O function to return execution back to the caller immediately, but the I/O is not assumed to be complete until some future time. The operating system notifies the caller when the I/O is complete. Alternatively, the caller can determine the status of the outstanding I/O operation by utilizing services of the operating system.

The advantage of asynchronous I/O is that the caller has time to do other work or issue more requests while the I/O operation is being completed. The term Overlapped I/O is often used for Asynchronous I/O and Non-overlapped I/O for Synchronous I/O. This article uses the terms Asynchronous and Synchronous for I/O operations under Windows NT. This article assumes the reader has certain familiarity with the File I/O APIs such as CreateFile, ReadFile, WriteFile.

Often, asynchronous I/O operations behave just as synchronous I/O. Certain conditions that this article discusses in the later sections make the I/O operations complete synchronously. The caller has no time for background work because the I/O functions don't return until the I/O is complete.

There are several APIs that are related to synchronous and asynchronous I/O. ReadFile and WriteFile are used as examples in this article, but ReadFileEx and WriteFileEx could also be used. Although this article discusses disk I/O specifically, many of the principles can be applied to other types of I/O, such as serial I/O or network I/O; such issues are, however, not discussed here. Note that Windows 95 doesn't support asynchronous I/O on disk devices, but does support asynchronous operation with other types of I/O devices. Since Windows 95 doesn't support asynchronous I/O for disk devices, its behavior is not covered in this article.

MORE INFORMATION

Setup Asynchronous I/O

The FILE_FLAG_OVERLAPPED flag must be specified in CreateFile when the file is opened. This flag allows I/O operations on the file to be performed asynchronously. Here is an example:


   HANDLE hFile;

   hFile = CreateFile(szFileName,
                      GENERIC_READ,
                      0,
                      NULL,
                      OPEN_EXISTING,
                      FILE_FLAG_NORMAL | FILE_FLAG_OVERLAPPED,
                      NULL);

   if (hFile == INVALID_HANDLE_VALUE)
      ErrorOpeningFile();

Take care when coding for asynchronous I/O because the system reserves the right to make an operation synchronous if it needs to. So, a program should be written to correctly handle an I/O operation that may be completed either synchronously or asynchronously. The example code provided takes this into consideration.

There are many things a program can do while waiting for asynchronous operations to complete, such as queuing additional operations or doing background work. For example, the following code properly handles overlapped and non-overlapped completion of a read operation. It does nothing more than wait for the outstanding I/O to complete:


   if (!ReadFile(hFile,
                 pDataBuf,
                 dwSizeOfBuffer,
                 &NumberOfBytesRead,
                 &osReadOperation )
   {
      if (GetLastError() != ERROR_IO_PENDING)
      {
         // some other error occurred while reading the file
         ErrorReadingFile();
         ExitProcess(0);
      }
      else
         // operation has been queued and
         // will complete in the future.
         fOverlapped = TRUE;
   }
   else
      // operation has completed immediately.
      fOverlapped = FALSE;

   if (fOverlapped)
   {
      // wait for the operation to complete before continuing.
      // you could do some background work if you wanted to.
      if (GetOverlappedResult( hFile,
                               &osReadOperation,
                               &NumberOfBytesTransferred,
                               TRUE))
         ReadHasCompleted(NumberOfBytesTransferred);
      else
         // operation has completed, but it failed
         ErrorReadingFile();
   }
   else
      ReadHasCompleted(NumberOfBytesRead);

Note that &NumberOfBytesRead passed into ReadFile is different than &NumberOfBytesTransferred passed into GetOverlappedResult. If an operation has been made asynchronous, then GetOverlappedResult is used to determine the actual number of bytes transferred in the operation once it has completed. The &NumberOfBytesRead passed into ReadFile is meaningless. If, on the other hand, an operation is completed immediately, then &NumberOfBytesRead passed into ReadFile is valid for the number of bytes read. In this case, the OVERLAPPED structure passed into ReadFile should be ignored. It should not be used with GetOverlappedResult or WaitForSingleObject.

Another caveat with asynchronous operation is that an OVERLAPPED structure must not be reused until its pending operation has completed. In other words, if you have three outstanding I/O operations, you should be using three OVERLAPPED structures. Reuse of an OVERLAPPED structure will cause unpredictable results in the I/O operations and may cause data corruption. In addition, before an OVERLAPPED structure can be used for the first time or reused after a prior operation has completed, it should be properly initialized so no left-over data affects the new operation.

The same type of restriction applies to the data buffer used in an operation. A data buffer must not be read or written until its corresponding I/O operation has completed: Reading or writing the buffer may cause errors and corrupt data.

Asynchronous I/O Still Appears to be Synchronous

If you've followed the instructions above, but your I/O operations still tend to all complete synchronously in the order issued and actually none of the ReadFile operations ever return FALSE with GetLastError() returning ERROR_IO_PENDING, then this leaves no time for any background work. Why does this happen?

There are a number of reasons why I/O operations complete synchronously even if you have coded for asynchronous operation:

Compression

One obstruction to asynchronous operation is NTFS compression. The file system driver will not access compressed files asynchronously; instead all operations are just made synchronous. This does not apply to files that are compressed with utilities similar to COMPRESS or PKZIP.

Extending a File

Another reason that I/O operations are completed synchronously is the operations themselves. On Windows NT, any write operation to a file that extends its length will be synchronous

Cache

Most I/O drivers (disk, communications, etc.) have special case code where, if an I/O request can be completed "immediately," the operation will be completed and the ReadFile or WriteFile function will return TRUE. In all ways, these types of operations appear to be synchronous. For a disk device, the typical case where an I/O request can be completed "immediately" is when the data is cached in memory.

Data Isn't in Cache

The cache scheme can work against you, however, if the data is not in the cache. The Windows NT cache is implemented internally using file mappings. The memory manager in Windows NT does not provide an asynchronous page fault mechanism to manage the file mappings used by the cache manager. The cache manager can, however, tell if the requested page is in memory, so if you issue an asynchronous cached read, and the pages are not in memory, the file system driver assumes that you do not want your thread blocked and the request will be handled by a limited pool of worker threads. Control is returned to your program after your ReadFile call with the read still pending. This works fine for a small number of requests, but since the pool of worker threads is limited (currently three on a 16MB system), there will still only be a few requests queued to the disk driver at a given time. If you issue a lot of I/O operations for data that is not in the cache, the cache manager and memory manager become saturated and your requests are made synchronous.

The behavior of the cache manager can also be influenced based on whether you access a file sequentially or randomly. Benefits of the cache are seen most when accessing files sequentially. The FILE_FLAG_SEQUENTIAL_SCAN flag in the CreateFile call will optimize the cache for this type of access. However, if you access files in a random fashion, use the FILE_FLAG_RANDOM_ACCESS flag in CreateFile to instruct the cache manager to optimize its behavior for random access.

Don't Use the Cache

The FILE_FLAG_NO_BUFFERING flag has the most effect on the behavior of the file system for asynchronous operation. This is the best way to guarantee that I/O requests are actually asynchronous. It instructs the file system to not utilize any cache mechanism at all. Warning: There are some restrictions to using this flag which have to do with the data buffer alignment and the device's sector size. See the function reference in the documentation for the CreateFile function for more information on using this flag properly.

SAMPLE CODE

The following file is available for download from the Microsoft Download Center. Click the file name below to download the file:

Asynczip.exe

For more information about how to download files from the Microsoft Download Center, please visit the Download Center at the following Web address

http://www.microsoft.com/downloads/search.asp

and then click How to use the Microsoft Download Center.

Sample code associated with this article demonstrates the use of the flags and APIs discussed. The code is run as a console application on Windows NT. The following command line switches controls its behavior:


Asynchio
Usage: asynchio [options]

Options:
   /fFilePattern  Files to use for I/O.
   /s    Specifies synchronous operation.
   /n    Specifies that no buffering should be used
   /r    Use FILE_FLAG_RANDOM_ACCESS
   /l    Use FILE_FLAG_SEQUENTIAL_SCAN
   /o###    Issue ### operations
   /e    First read entire file, then issue smaller reads
   /?    Display this usage message.

Example: asynchio /f*.bmp /n

Default operation of this program is for asynchronous, buffered operation. 500 I/O operations are requested by default.

Real World Test Results

Following are some test results from the sample code. The magnitude of the numbers is not important here and will vary from machine to machine, but the relationship of the numbers compared to each other illuminates the general affect of the flags on performance. You should see similar results:


Asynchronous, unbuffered I/O:  asynchio /f*.dat /n

   Operations completed out of the order in which they were requested.
   500 requests queued in 0.224264 seconds.
   500 requests completed in 4.982481 seconds.

This test demonstrates that the above program issued 500 I/O requests quickly and was given a significant amount of time to do other work or issue more requests.


Synchronous, unbuffered I/O: asynchio /f*.dat /s /n

   Operations completed in the order issued.
   500 requests queued and completed in 4.495806 seconds.

This test demonstrates that the above program spent 4.495880 seconds calling ReadFile to complete its operations, whereas the former test only spent 0.224264 seconds to issue the same requests. In the second test, there was no "extra" time for the program to do any background work.


Asynchronous, buffered I/O: asynchio /f*.dat

   Operations completed in the order issued.
   500 requests issued and completed in 0.251670 seconds.

This test demonstrates the synchronous nature of the cache. All reads were issued and completed in 0.251670 seconds. In other words asynchronous requests were completed synchronously. This test also demonstrates the high performance of the cache manager when data is in the cache.


Synchronous, buffered I/O: asynchio /f*.dat /s

   Operations completed in the order issued.
   500 requests and completed in 0.217011 seconds.

This test demonstrates the same results as above. Note that synchronous reads from the cache complete slightly faster than asynchronous reads from the cache. This test also demonstrates the high performance of the cache manager when data is in the cache.

CONCLUSION

The decision on which method is best is left to you because it depends entirely on the type, size, and number of operations that your program does.

The default file access without specifying any special flags to CreateFile is a synchronous and cached operation. Note that you do get some automatic asynchronous behavior in this mode because the file system driver does predictive asynchronous read-ahead and asynchronous lazy writing of modified data. Although this doesn't make the application[ASCII 146]s I/O asynchronous, it is the ideal case for the vast majority of simple applications.

If, on the other hand, your application is not simple, you may need to do some profiling and performance monitoring to determine the best method, similar to the tests illustrated above. Profiling the time spent in the ReadFile or WriteFile API and then comparing this time to how long it takes for actual I/O operations to complete is extremely useful. If the majority of the time is spent in actually issuing the I/O, then your I/O is being completed synchronously. However, if the time spent issuing I/O requests is relatively small compared to the time it takes for the I/O operations to complete, then your operations are being treated asynchronously. The sample code mentioned above uses the QueryPerformanceCounter API to do its own internal profiling.

Performance monitoring can help determine how efficiently your program is using the disk and the cache. Tracking any of the performance counters for the Cache object will indicate the performance of the cache manager. Tracking the performance counters for the Physical Disk or Logical Disk objects will indicate the performance of the disk systems.

There are several utilities that are helpful in performance monitoring: PerfMon and DiskPerf are two that are extremely useful. The diskperf -y command must be issued in order to enable the system to collect data on the performance of the disk systems, so be sure to do it first. After issuing the command, a reboot will be required in order to start the data collection.

REFERENCES

For more information on these utilities and performance monitoring in general, see the volume "Optimizing Windows NT" in the Windows NT Resource Kit documentation.

Additional query words: ReadFile ReadFileEx WriteFile kbdss WriteFileEx GetOverlappedResult Asynchronous Synchronous Overlapped Non-overlapped kbfile

Keywords : kbfile kbAPI kbFileIO kbKernBase kbNTOS351 kbNTOS400 kbWinOS2000 kbSDKWin32 kbWinOS95 kbDSupport kbGrpKernBase
Version : WINDOWS:; winnt:3.51,4.0
Platform : WINDOWS winnt
Issue type : kbinfo