You can select any combination of Anonymous, Basic, and NTLM authentication. If Anonymous authentication is checked, the request will be handled without any actual authentication and IIS will execute the request in the context of the IUSR_<servername> account. If, for some reason, the IUSR_<servername> account does not have access to the resource, IIS will send back an access-denied error to the client indicating that the client needs to use one of the other authentication schemes. This scenario could occur if you limited access to the actual ISAPI DLL file to a specific user, such as User1. IIS would receive the initial anonymous request and attempt to launch the ISAPI DLL under the IUSR_<servername> user context, only to get an access-denied error from the NTFS file system. IIS would respond to the client with a message saying that access was denied and the client needs to submit the request using either the Basic or NTLM authentication schemes (depending on which one is enabled, possibly even both). The client can then resubmit the request with the Basic authentication credentials or with the initial NTLM challenge/response sequence. If either of these responses provides validation of the User1 account, then IIS will impersonate the User1 account and successfully launch the ISAPI DLL.
It is worth noting that if both NTLM and Basic authentication schemes are checked, IIS responds to requests indicating that both schemes are acceptable. It is up to the client to determine which authentication schemes it supports and to respond appropriately. Browsers, such as Internet Explorer, that support both Basic and NTLM Authentication will respond using the first supported authentication scheme indicated by IIS.