Click here to Skip to main content
15,903,854 members
Articles / Security / Encryption

Applied OpenSSL: CTR Mode in File Encryption

Rate me:
Please Sign up or sign in to vote.
4.68/5 (14 votes)
22 Feb 2011CPOL6 min read 65.1K   2K   39   8
In this article, the use of open source OpenSSL library for file and disk encryption is described


Information security on PC has become more and more popular, so I want to cover some issues of this topic – in particular, the using of ciphers to prevent data stealing from a physically removed hard disk.

Cipher is the sequence of steps to transform original text (plain text) to cipher text, which completely can’t be read by an unauthorized person or the efforts to crack it will cost more than the information in the plain text costs.

Ciphers are documented and well-known, but the problem is how to make the good implementation, or in our situation, how to find and use the open source library. I will explain what advantages OpenSSL library has and how to use it.

File Encryption Methods

You can secure single files or folders by the user-mode applications, which will encrypt or decrypt information by your command. You can also use the complex kernel-mode on-the-fly encryption solutions based on the virtual drives (legacy disk device), encrypted file systems, file system filters and storage filter driver (full disk encryption), which work synchronously with OS read/write requests.


Fig. 1. Application levels to crypt information.

Cipher types by type of the key:

Symmetric key ciphers are preferable because encryption and decryption are performed by a single solution. At the same time, you should remember that the process can be successful only if nobody can get access to the processed data and the sources of the solution (this should be guaranteed by the solution itself).

Cipher types by type of input data

  • Block ciphers – Encrypt and decrypt blocks of data of fixed size
  • Stream ciphers - Encrypt and decrypt continuous streams of data (message)

Block ciphers are preferable because of the way the data is stored on HDD: it is divided into the sectors of fixed size.

In this article, I want to describe the problem of cipher implementation in file encryption.

For each kind of security application, we should choose cipher and in case of the file encryption, corresponding block cipher mode. Each case requires different cipher mode:

  • Legacy Disk Device and File System level could use any block cipher, because on this level, read and write requests are aligned to the sector size and you control and have access to all data. These disk encryption solutions use XTS and CMC cipher mode.
  • File System filters have less freedom to work with files, because of permissions, which are set on file creation by user or system. Also, on this level, you closely connected to pure documented behavior of the cache manager and file system. Any wrong action with data, data length or permissions could cause errors or unexpected behavior. The best cipher mode for this purpose is CTR, which offers real random access to the file data. It allows you to encrypt or decrypt only the current chunk of non-granulated data, and only it.
  • User-mode applications can use any block cipher, but while using ECB or CBC, the applications should store the header in files, which contains the real file size, and should granulate the file size to the block size of cipher. Another approach is to use CTR, CFB or OFB. They differ only by error propagation in changed cipher text, so I would prefer CTR.


Crypto++ library is well-known, use of Crypto++ is described very well in the article “Applied Crypto++: Block Ciphers”. This library is very useful and has a big amount of ciphers and wrappers, but it also has a couple of disadvantages, which can prevent its usage:

  • Porting through compilers (different msvc and gcc)
  • Problems with using in driver projects

I would like to introduce the OpenSSL implementation, which has solid C style easily portable for different compilers and platforms.

Main folders for our purpose are crypto and include. The crypto folder contains all ciphers and the include folder contains library headers with definitions and constants:


Fig. 2. Listing of OpenSSL folder.

Each cipher has its own subfolder in the crypto folder. Implementation of each mode for the cipher is located in the same folder, but some ciphers use common cipher mode implementation, which can be found in the modes folder. You can use all of them or easily copy a folder with the required algorithm and use it separately. I pick the AES cipher:


Fig. 3. Listing of crypto folder.

You should copy files to the new library project, change some include paths and build the library with AES cipher. See the example attached to the article.

To use AES cipher, you should initialize AES key from the initialization vector with the help of AES_set_encrypt_key:

InitializeAesKey ( const unsigned char* aKey, 
	unsigned int KeyLength, CSecurityKey* key )
    AES_KEY bfKey;
    ::memset( &bfKey, 0, sizeof (bfKey) );

    AES_set_encrypt_key( aKey, KeyLength, &bfKey );
    *key = CSecurityKey ( (char*)&bfKey, sizeof(AES_KEY));

To encrypt or decrypt data with CTR, you also need three variables:

void AES_ctr128_encrypt(...
			unsigned char ivec[AES_BLOCK_SIZE],
			unsigned char ecount_buf[AES_BLOCK_SIZE],
			unsigned int *num )
  • ivec – Counter of the CTR mode
  • ecount_buf – Variable used for encrypted ivec
  • num – Index of byte from start of the data block to encrypt


  • If you want to use CTR from the beginning of the file, you should use nullified ivec if other, you should initialize your ivec with the value that corresponds to the size of the working block of data (for 128bit cipher, to crypt from the byte 48, you should increment ivec 3 times – it is only for example because this method is not very fast for the big files)
  • If you crypt data from the beginning of the block of your block cipher, you should use nullified ecount_buf and num if other, you should manually assign ecount_buf to the encrypted ivec for this data block, assign num to the index of byte from the beginning of the data block, and assign ivec to the value of the next data block.

For example, let’s take the 128bit cipher. To start encryption from the 50th byte, we should:

  • Perform ivec incrementation 3 times
  • Set ecount_buf to the encrypted value of ivec
  • Set num to 2 (as the 50th byte is the 2nd one from the beginning of the 4th block – 50=16*3+2)
  • Perform ivec incrementation

You can find the example of using OpenSSL implementation in the attached files. There is solution for 2008 Visual Studio with 2 projects:

  • OpenSSLAes – Library with the AES cipher plus CTR mode
  • Crypt_Mode_CTR – Executable test file. You can encrypt or decrypt any file you want and see the result.

Input file:


Encrypted file:




This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Written By
Chief Technology Officer Apriorit Inc.
United States United States
ApriorIT is a software research and development company specializing in cybersecurity and data management technology engineering. We work for a broad range of clients from Fortune 500 technology leaders to small innovative startups building unique solutions.

As Apriorit offers integrated research&development services for the software projects in such areas as endpoint security, network security, data security, embedded Systems, and virtualization, we have strong kernel and driver development skills, huge system programming expertise, and are reals fans of research projects.

Our specialty is reverse engineering, we apply it for security testing and security-related projects.

A separate department of Apriorit works on large-scale business SaaS solutions, handling tasks from business analysis, data architecture design, and web development to performance optimization and DevOps.

Official site:
Clutch profile:
This is a Organisation

33 members

Written By
Software Developer (Senior) ApriorIT
Ukraine Ukraine
Senior Software Developer of Apriorit Inc.
My favorite tasks are multithreading, networking and WDK.

... But in free time : beer, meat, travelling and photo...)

LinkedIn Profile : Wineblat Eugene

Comments and Discussions

Questionthis is great Pin
Southmountain25-Oct-21 10:47
Southmountain25-Oct-21 10:47 
QuestionHow can I make a decryption with AES CTR? Pin
Member 876675513-Aug-12 19:03
Member 876675513-Aug-12 19:03 
GeneralMy vote of 5 Pin
Manoj Kumar Choubey15-Apr-12 23:32
professionalManoj Kumar Choubey15-Apr-12 23:32 
GeneralMy vote of 5 Pin
Ccello3-Feb-12 7:10
Ccello3-Feb-12 7:10 
GeneralMistake Pin
mmikkone22-Feb-11 2:59
mmikkone22-Feb-11 2:59 
GeneralMy vote of 4 Pin
Pierfabio4-Jan-11 3:47
Pierfabio4-Jan-11 3:47 
GeneralMy vote of 5 Pin
Tage Lejon20-Dec-10 17:28
Tage Lejon20-Dec-10 17:28 
GeneralMy vote of 5 Pin
Tage Lejon8-Dec-10 21:16
Tage Lejon8-Dec-10 21:16 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.