Click here to Skip to main content
14,447,535 members

Ncryptdecrypt fails to decrypt data that is encrypted by openssl with RSA_PKCS1_OAEP_PADDING

Amir Dashti asked:

Open original thread
I have difficulty to decrypt data being encrypted using OpenSSL, RSA and RSA_PKCS1_OAEP_PADDING padding option.

What I am doing is to load the key from Windows KSP:
m_hSystemStore = CertOpenStore(CERT_STORE_PROV_SYSTEM, 0, NULL, CERT_SYSTEM_STORE_LOCAL_MACHINE, m_storeName.c_str());

m_pCertWithKeys = CertFindCertificateInStore(m_hSystemStore, SupportedEncodings, 0, CERT_FIND_SUBJECT_STR, m_certName.c_str(), NULL);

// Obtain the private key from the certificate.
DWORD m_KeyContextSpec = 0;
CryptAcquireCertificatePrivateKey(m_pCertWithKeys, CRYPT_ACQUIRE_SILENT_FLAG | CRYPT_ACQUIRE_PREFER_NCRYPT_KEY_FLAG, NULL, &m_hKeyContextFull, &m_KeyContextSpec, &m_KeyContextMustBeReleased);

and call NCryptDecrypt like:
BCRYPT_OAEP_PADDING_INFO paddingInfo = { 0 };
DWORD cbDecryptedMessage;
BYTE* pbDecryptedMessage = NULL;

paddingInfo.pszAlgId = BCRYPT_SHA1_ALGORITHM;

// Calculate the required buffer
NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, NULL, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);

// After required buffer is allocated...
NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, pbDecryptedMessage, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);

It fails with NTE_INVALID_PARAMETER (0x80090027). I tried different flags but none of them works.

Note: All error checkings have been removed from code for readability.

The data is being encrypted with same key (public part) like:
RsaPublicEncrypt(size - 42, blk, output, Rsa, RSA_PKCS1_OAEP_PADDING)

and can be decrypted using SoftHSM successfully.

Is there any limitation for CNG to decrypt data being encrypted by OpenSSL?
Is there any idea what am I doing wrong?


What I have tried:

I did not change anything in OpenSSL side, because that part works and we can decrypt data using SoftHSM with same key, but:
* I have tried different flags in NCryptDecrypt
* Different algorithm for padding
* Different algorithm to generate the key
None of them worked so far.
Tags: Encryption, OpenSSL


When answering a question please:
  1. Read the question carefully.
  2. Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
  3. If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
  4. Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.
Let's work to help developers, not make them feel stupid.
Please note that all posts will be submitted under the The Code Project Open License (CPOL).

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100