After a user inputs a search string, it must be put into a form that the Search component, and eventually a stored procedure, can use. One of the steps in this process is called tokenizing. Briefly, this is how it works:
A library user inputs a search string in Library.asp, which includes Search.asp and DSearch.asp.
In Search.asp, the Search object is instantiated and its SearchString (Let) property is assigned the search string input by the user. This SearchString property does several things. When the property is set, it calls the Search object's private Tokenize method, which builds (and returns) an array of tokens from the original string. These tokens include not only the words in the input search string, now all separated by spaces, but also the operators AND, OR, and ~ (the tilde character), which was substituted for NEAR. Strings inside double quotes count as a single token, whether or not they contain spaces.
To do its job, Tokenize calls the GetToken method, which helps it keep track of its position in the search string. When the property is retrieved, the SearchType property determines the format of the final string. For instance, if SearchType is set to ftsContains, each token in the string is verified to be separated by a separator keyword. If one is not present, AND is used.