Ncryptdecrypt fails to decrypt data that is encrypted by openssl with RSA_PKCS1_OAEP_PADDING

I have difficulty to decrypt data being encrypted using OpenSSL, RSA and RSA_PKCS1_OAEP_PADDING padding option.

What I am doing is to load the key from Windows KSP:

m_hSystemStore = CertOpenStore(CERT_STORE_PROV_SYSTEM, 0, NULL, CERT_SYSTEM_STORE_LOCAL_MACHINE, m_storeName.c_str());

m_pCertWithKeys = CertFindCertificateInStore(m_hSystemStore, SupportedEncodings, 0, CERT_FIND_SUBJECT_STR, m_certName.c_str(), NULL);

// Obtain the private key from the certificate.
DWORD m_KeyContextSpec = 0;
HCRYPTPROV_OR_NCRYPT_KEY_HANDLE m_hKeyContextFull;
CryptAcquireCertificatePrivateKey(m_pCertWithKeys, CRYPT_ACQUIRE_SILENT_FLAG | CRYPT_ACQUIRE_PREFER_NCRYPT_KEY_FLAG, NULL, &m_hKeyContextFull, &m_KeyContextSpec, &m_KeyContextMustBeReleased);

and call NCryptDecrypt like:

BCRYPT_OAEP_PADDING_INFO paddingInfo = { 0 };
DWORD cbDecryptedMessage;
BYTE* pbDecryptedMessage = NULL;

paddingInfo.pszAlgId = BCRYPT_SHA1_ALGORITHM;

// Calculate the required buffer
NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, NULL, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);

// After required buffer is allocated...
NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, pbDecryptedMessage, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);

It fails with NTE_INVALID_PARAMETER (0x80090027). I tried different flags but none of them works.

Note: All error checkings have been removed from code for readability.

The data is being encrypted with same key (public part) like:

RsaPublicEncrypt(size - 42, blk, output, Rsa, RSA_PKCS1_OAEP_PADDING)

and can be decrypted using SoftHSM successfully.

Is there any limitation for CNG to decrypt data being encrypted by OpenSSL?
Is there any idea what am I doing wrong?

Thanks.

What I have tried:

I did not change anything in OpenSSL side, because that part works and we can decrypt data using SoftHSM with same key, but:
* I have tried different flags in NCryptDecrypt
* Different algorithm for padding
* Different algorithm to generate the key
None of them worked so far.

Tags: OpenSSL, Encryption