SO_SSL_RSA_ENCRYPT_HOOK

This WSAIoctl command is used to get/set the RSA encryption hook that SSL will use for the socket. The lpvInBuffer points to the following structure:

struct sslrsaencrypthook {
    int (*func)(void *arg, int blockType, char *dest, int *destlen,
    char *src, int srclen);
    void *arg;
};
 

This hook is used when the SSL implementation requires an RSA private or public key encryption. The blockType determines if the operation is a private or public key operation (only block types 0x01 and 0x02 are used by SSL; see PKCS#1 for more information). The dest and destlen values are used to return the encryption results. destlen is an input/output parameter with the input value defining the maximum storage area available at dest, and the output value containing the actual stored length of the encryption results. The src and srclen values define the input data. The following values are returned by the hook:

#define SSL_REH_OK                0
    #define SSL_REH_BAD_TYPE          1
    #define SSL_REH_BAD_LEN           2
 

SSL_REH_OK is returned if the encryption succeded. SSL_REH_BAD_TYPE is returned if the blockType value is unacceptable. SSL_REH_BAD_LEN is returned if the srclen is too large to be encrypted in a single encryption, or if the dstlen value is too small to hold the encryption results.