I developed a client and server that use the OpenSSL to communicate using TLS via TCP, and it is working very well.
But I need to communicate via Namedpipe, and there is the question:
When I create a namedpipe, I call:
HANDLE hPipe = CreateNamedPipe(...);
After the Namedpipe is connected, I need call:
int SSL_set_fd(SSL *ssl, int fd);
As we can see, the "hPipe" is a HANDLE, but I need convert this HANDLE to int. So I tried to use the below to get a "int" then call SSL_set_fd(ssl, fd):
int fd = _open_osfhandle(reinterpret_cast<intptr_t>(hPipe), 0);
The return of _open_osfhandle() will be "3", but OpenSSL isn't accepting when I try call "SSL_accept()".
Am I doing something wrong?
Is there other way to use OpenSSL with Namedpipe?
Note: I'm using the Microsoft Visual Studio, C++.