SO_SSL_RSA_DECRYPT_HOOK

This WSAIoctl command is used to get/set the RSA decryption hook that SSL will use for the socket. The lpvInBuffer points to the following structure:

struct sslrsadecrypthook {
    int (*func)(void *arg, int blockType, char *dest, int *destlen,
    char *src, int srclen);
    void *arg;
};
 

This hook is used when the SSL implementation requires an RSA private or public key decryption. The blockType determines if the operation is a private or public key operation (only block types 0x01 and 0x02 are used by SSL; see PKCS#1 for more information). The dest and destlen values are used to return the decryption results. destlen is an input/output parameter with the input value defining the maximum storage area available at dest, and the output value containing the actual stored length of the decryption results. The src and srclen values define the input data. The following values are returned by the hook:

#define SSL_RDH_OK           0
    #define SSL_RDH_BAD_TYPE     1
    #define SSL_RDH_BAD_LEN      2
 

SSL_RDH_OK is returned if the decryption succeeded. SSL_RDH_BAD_TYPE is returned if the blockType value is unacceptable. SSL_RDH_BAD_LEN is returned if the srclen doesn't match the key's modulus length, or if the dstlen value is too small to hold the decryption results.