Fededim.Extensions.Configuration.Protected.DataProtectionAPI: The Ultimate Integration between ASP.NET Configuration and Data Protection API





5.00/5 (5 votes)
Fededim.Extensions.Configuration.Protected is an improved ConfigurationBuilder which allows partial or full encryption of configuration values stored inside any possible ConfigurationSource and fully integrated in the ASP.NET Core architecture.
Introduction
Almost one month ago, I posted this article ProtectedJson about an improved ConfigurationSource
and ConfigurationProvider
for JSON files which allowed partial or full encryption of configuration values using Data Protection API. Some comments came through and one which was not so "meaningful" had a question whether my package also supported environment variables.
Even though we could wonder why someone ever wants to encrypt/decrypt environment variables, this question made me have an epiphany: can what I had already done for JSON files also be extended to other configuration sources ? After a small proof of concept project after work, the answer was yes and after some rework, I published a new package Fededim.Extensions.Configuration.Protected which is, most of all, an improvement and an extension of ProtectedJson
to support encryption/decryption of configuration values stored inside ANY configuration source.
Key Features
- Encrypt partially or fully a configuration value
- Works with any existing and (hopefully) future
ConfigurationSource
andConfigurationProvider
(successfully tested with framework's builtin providers likeCommandLine, EnvironmentVariables, Json, Xml and InMemoryCollection
) - Transparent in memory decryption of encrypted values without almost any additional line of code
- Supports a global configuration and an eventual custom override for any
ConfigurationSource
- Supports almost any NET framework (net6.0, netstandard2.0 and net462)
- Pluggable easily into any existing NET / NET Core project
- Supports automatic re-decryption on configuration reload if underlying
IConfigurationProvider
supports it - Supports per configuration value encryption derived subkey (called "subpurposes")
- Supports pluggable encryption/decryption with different providers implementing a standard interface
IProtectProvider
(since version 1.0.12, keep in mind that implementing a secure and robust encryption/decryption provider requires a deep knowledge of security!).
Background
ASP.NET Configuration is the standard .NET Core way of storing application configuration data through hierarchical key-value pairs inside a variety of configuration sources (usually JSON files, but also environment variables, XML file, in memory dictionaries, command line parameters or any custom provider you would like to implement). While .NET Framework used a single source (usually, a XML file which was intrinsically more verbose), .NET Core can use multiple ordered configuration sources, which gets "merged" allowing the concept of overriding of the value of a key in a configuration source with the same one present in a subsequent configuration source. This is useful because in software development, there are usually multiple environments (Development, Integration, PRE-Production and Production) and each environment has its own custom settings (for example, API endpoints, database connection strings, different configuration variables, etc.). In .NET Core, this management is straightforward, in fact, you usually have two JSON files:
- appsettings.json: which contains the configuration parameters common to all environments
- appsettings.<environment name>.json: which contains the configuration parameters specific to the particular environment
ASP.NET Core apps usually configure and launch a host. The host is responsible for app startup, configuring dependency injection and background services, configuring logging, lifetime management and obviously configuring application configuration. This is done mainly in two ways:
- Implicitly, by using one of the framework provided methods like
WebApplication.CreateBuilder
orHost.CreateDefaultBuilder
(usually called inside the Program.cs source file) which substantially do:- Read and parse command line arguments
- Retrieve environment name respectively from
ASPNETCORE_ENVIRONMENT
andDOTNET_ENVIRONMENT
environment variable (set either in the operating system variables or passed directly in the command line with--environment
argument). - Read and parse two JSON configuration files named appsettings.json and appsettings.<environment name>.json.
- Read and parse the environment variables.
- Call the delegate
Action<Microsoft.Extensions.Hosting.HostBuilderContext, Microsoft.Extensions.Configuration.IConfigurationBuilder>
ofConfigureAppConfiguration
where you can configure the app configuration throughIConfigurationBuilder
parameter
- Explicitly by instantiating the
ConfigurationBuilder
class and using one of the provided extensions methods:AddCommandLine
: to request the parsing of command line parameters (either by--
or-
or/
)AddJsonFile
: to request the parsing of a JSON file specifying whether it is mandatory or optional and whether it should be reloaded automatically whenever it changes on filesystem.AddXmlFile
: to request the parsing of an XML file specifying whether it is mandatory or optional and whether it should be reloaded automatically whenever it changes on filesystem.AddEnvironmentVariables
: to request the parsing of environment variables- etc.
In essence, every Add<xxxx>
extension method adds a ConfigurationSource
to specify the source of key-value pairs (CommandLine, Json File, Environment Variables, etc.) and an associated ConfigurationProvider
used to load and parse the data from the source into the Providers
list of IConfigurationRoot
interface which is returned as a result of the Build
method on ConfigurationBuilder
class as you can see the picture below.
(Inside configuration.Providers
, there are six sources: CommandLineConfigurationProvider
, two JsonConfigurationProvider
for both appsettings.json and appsettings.<environment name>.json, a XMLConfigurationProvider
for the XML file appsettings.xml, a MemoryConfigurationProvider
for the provided dictionary and finally EnvironmentVariableConfigurationProvider
for the environment variables).
As I wrote earlier, the order in which the Add<xxxx> extension methods are called is important because when the IConfigurationRoot
class retrieves a key value, it uses the GetConfiguration method which cycles the Providers
list in a reversed order trying to return the first one which contains the queried key, thus simulating a "merge" of all configuration sources (LIFO order, Last In First Out).
Using the Code
You find all the source code on my Github repository, the code is based on .NET 6.0 and Visual Studio 2022. Inside the solution file, there are five projects, two about the older ProtectedJson
and three about this new Protected
package, let's talk about the last four:
Fededim.Extensions.Configuration.Protected
: Essentially this package implements the core logic to integrate with Microsoft ASP.NET Core Configuration API, delegating all the encryption/encryption code to a pluggable external provider implementing the interfacesIProtectProvider
(dealing with the core logic of encryption/decryption) andIProtectProviderConfigurationData
,ProtectedConfigurationBuilder
, the enabling class for decryption of any configuration value stored inside any configuration source.ProtectedConfigurationProvider
(the main class actually responsible for decryption of configuration values),ConfigurationBuilderExtensions
which contains the extension methods forIConfigurationBuilder
interface (WithProtectedConfigurationOptions
used to specify a particular configuration which applies only to a specificConfigurationSource
) andProtectFileOptions/ProtectFileProcessors
(XmlProtectFileProcessor, JsonProtectFileProcessor, JsonWithCommentsProtectFileProcessor, RawProtectFileProcessor
) used to read, decode, encrypt and re-encode source files. This package has been published on NuGet.Org.
A standard provider based on Microsoft Data Protection API is provided by the companion package Fededim.Extensions.Configuration.Protected.DataProtectionAPI, you can reference just this package in your project if you do not plan to develop another encryption/decryption provider (again keep in mind that implementing a secure and robust encryption/decryption provider requires a deep knowledge of security!).
Fededim.Extensions.Configuration.Protected.DataProtectionAPI
: This is the standard Microsoft Data Protection API encryption/decryption provider which implements the interfaceIProtectProvider
and the abstract classIProtectProviderConfigurationData
respectively with its classesDataProtectionAPIProtectProvider
andDataProtectionAPIProtectConfigurationData.
Again this package has been published on NuGet.Org.
Fededim.Extensions.Configuration.Protected.DataProtectionAPITest
: This axUnit
test project which tests thoroughly the two above packages in order to improve the reliability and the code quality. It creates sample data for allConfigurationSources
provided by MS .NET (a JSON file, a XML file, environment variables, an in-memory dictionary and command line arguments) containing a 2*fixed set of keys (10000), one in plaintext with random datatype and value and another with the same value but encrypted. It loads then the sample data withProtectedConfigurationBuilder
in order to decrypt it and tests that all plaintext values are the same as those that have been decrypted.
Fededim.Extensions.Configuration.Protected.ConsoleTest
: This is a console application which shows how to useProtectedConfigurationBuilder
by reading and parsing six encrypted bespoke configuration sources and converting them to a strongly type class calledAppSettings
. The decryption happens flawlessly and automatically without almost any line of code, let's see how.
To use the automatic decryption feature, you have just to replace the call new ConfigurationBuilder()
with a call to new ProtectedConfigurationBuilder()
passing to it the configuration of the pluggable encryption/decryption provider. After having done that, you can add any existing configuration source by using the standard methods like AddCommandLine
, AddJsonFile
, AddXmlFile
, AddInMemoryCollection
, AddEnvironmentVariables
or even future configuration sources since this package should support all of them as long as the implementation of the GetChildKeys of Microsoft.Extensions.Configuration.ConfigurationProvider does not change (keep reading below to understand the reason). The constructor for ProtectedConfigurationBuilder
takes just one parameter, a configuration class derived from the IProtectProviderConfigurationData
, an abstract class (implemented by one of the available providers) which is used for specifying the configuration options and plugging the providers into ProtectedConfigurationBuilder.
There are four common fundamental parameters which must be specified by every provider:
ProtectedRegex
: It is a regular expression which specifies the tokenization tag which encloses the encrypted data to be decrypted; it must define a named group calledprotectedData
(and optionally two additional groups calledsubPurposePattern
andsubPurpose
for specifying a per configuration value subkey). Ifnull
, this parameter assumes the default value:public const String DefaultProtectedRegexString = "Protected(?<subPurposePattern>(:{(?<subPurpose>[^:}]+)})?):{(?<protectedData>.*?)}";
The above regular expression essentially searches in a lazy way (so it can retrieve all the occurrences inside a value) for any
string
matching the pattern'Protected:{<subPurpose>}:{<encrypted data>}'
and extracts the<encrypted data>
substring storing it inside a group namedprotectedData
. There is also an optional part called <subPurposePattern>
(made up of:{<subPurpose>}
)<encrypted data>
substring in a group calledprotectedData
and the <subPurposePattern>
and<subPurpose>
substrings in two groups called respectivelysubPurposePattern
andsubPurpose.
-
ProtectRegex
: It is a regular expression which specifies the tokenization tag which encloses the data to be encrypted; again it must define a named group called this timeprotectData
(and optionally two additional groups calledsubPurposePattern
andsubPurpose
for specifying a per configuration value subkey). Ifnull
, this parameter assumes the default value (e.g.Protect:{<subPurpose>}:{<data to be encrypted>}
):public const String DefaultProtectRegexString = "Protect(?<subPurposePattern>(:{(?<subPurpose>[^:}]+)})?):{(?<protectData>.*?)}";
-
DefaultProtectedReplaceString:
It is a string expression used to transform the plaintext tokenization into the encrypted tokenization (e.g. fromProtect:{<subPurpose>}:{<data to be encrypted>}
intoProtected:{<subPurpose>}:{<encrypted data>}
). It contains two placeholders${subPurposePattern}
and${protectedData}
which gets substituted respectively with the subPurposePattern (if present) and the encrypted data. Ifnull
, this parameter assumes the default value:
public const String DefaultProtectedReplaceString = "Protected${subPurposePattern}:{${protectedData}}";
IProtectProvider
: this is a standard interface which must be implemented by the pluggable provider providing the encryption/decryption services.
By default a standard encryption/decryption provider based on Microsoft Data Protection API is provided in the companion package Fededim.Extensions.Configuration.Protected.DataProtectionAPI. Its configuration class implementing IProtectProviderConfigurationData
is called DataProtectionAPIProtectConfigurationData whose constructor takes these additional inherent parameters besides the above three common ProtectedRegex,ProtectRegex
and DefaultProtectedReplaceString
ones:
dataProtectionServiceProvider
: This is aIServiceProvider
interface needed to instance theIDataProtectionProvider
of Data Protection API in order to decrypt the data. This parameter is mutually exclusive to the next one.dataProtectionConfigureAction
: This is anAction<IDataProtectionBuilder>
used to configure the Data Protection API in standard NET Core. Again, this parameter is mutually exclusive to the previous one.purposeString
: used to specify explicitly a purpose string to use for encryption. Data Protection API supports multiple encryption keys which are derived from the configured master key and strictly connected to one or morepurpose
string passed to theCreateProtector
API, for further information please read here and here.keyNumber
: this parameter is alternative topurposeString
for convenience (we are more used to think of different key numbers rather than different purposes), it is used to specify a key index (the default value is 1) to use for encryption, under the hood the provided index is used to construct thepurpose
string$"Key{keyNumber}" (
seeProtectedConfigurationBuilder.ProtectedConfigurationBuilderKeyNumberPurpose
method)
The dataProtectionServiceProvider
and dataProtectionConfigureAction
parameters are somewhat a drawback, because they represent a reconfiguration of another dependency injection for instantiating the IDataProtectionProvider
needed to decrypt the data.
In fact, in a standard NET Core application, usually the dependency injection is configured after having read and parsed the configuration file (so all configuration sources and providers do not use DI), but in this case, I was compelled since the only way to access Data Protection API is through DI. Moreover, when configuring the dependency injection, the parsed configuration usually gets binded to a strongly typed class by using services.Configure<<strongly typed settings class>>(configuration)
so it's a dog chasing its tail (for decrypting configuration you need DI, for configuring DI, you need the configuration parsed in order to bound it to a strongly typed class). The only solution I came up for now is reconfiguring a second DI IServiceProvider
just for the Data Protection API and use it inside ProtectedConfigurationProvider
. To configure the second DI IServiceProvider
, you have two options:
- You create it by yourself (by instantiating a
ServiceCollection and
callingAddDataProtection
on it) - You let
ProtectedConfigurationProvider
to create it by passing adataProtectionConfigureAction
parameter. In this case, in order to avoid duplicated code, the configuration of Data Protection API can be performed inside a commonprivate
method calledConfigureDataProtection
, e.g.:
private static void ConfigureDataProtection(IDataProtectionBuilder builder)
{
builder.UseCryptographicAlgorithms(new AuthenticatedEncryptorConfiguration
{
EncryptionAlgorithm = EncryptionAlgorithm.AES_256_CBC,
ValidationAlgorithm = ValidationAlgorithm.HMACSHA256,
}).SetDefaultKeyLifetime(TimeSpan.FromDays(365*15)).PersistKeysToFileSystem
(new DirectoryInfo("..\\..\\Keys"));
}
Here, I chose to use AES 256 symmetric encryption with HMAC SHA256 as digital signature function. Moreover, I ask to store the master encryption key metadata (key, iv, hash algorithm, etc.) in an XML file inside the Keys folder of the console app (by default, keys are stored in a particular location according to MS key management documentation, note that all these APIs are provided by default by the Data Protection API).
So when you start the app for the first time, the Data Protection API creates automatically the master encryption key and stores it in the Keys folder, in the following runs, it loads it from this XML file. This configuration however is not the best approach from the security viewpoint because the key is stored in plain text, if you want to encrypt the master key at rest, you can use the ProtectKeysWithDpapi
extension method (works only in Windows and in this case, it would be encrypted with Windows DPAPI) or ProtectKeysWithCertificate
to encrypt it with a certificate installed on the computer. Please note that even though you can use different encryption keys in the Keys folder, there is just one master key from which all encryption keys are derived using the purpose string crafted from either the keyNumber
parameter or the purposeString
parameter specified either in the ProtectedConfigurationBuilder
or in the WithProtectedConfigurationOptions
extension method.
In the console application, I add the six configuration sources in the following order to exemplify the merging feature of ASP.NET Core Configuration and also the use of encrypted values:
AddCommandLine
: to add the command line arguments.AddJsonFile
: to add the two json files appsetting.json and appsettings.development.json, the second one has thereloadOnChange
flag set totrue
in order to allow the reload of json file whenever it changes on the filesystem.If you look at the ConnectionStrings section of appsetting.json, there are three keys:
PlainTextConnectionString
: As the name states, it contains a plaintext connection stringPartiallyEncryptedConnectionString
: As the name states, it contains a mixture of plain text and multipleProtect:{<data to encrypt>}
tokenization tags. On every run, these tokens get automatically encrypted and replaced with theProtected:{<encrypted data>}
token after the call to the extension methodIProtectProviderConfigurationData.ProtectFiles.
FullyEncryptedConnectionString
: As the name states, it contains a singleProtect:{<data to encrypt>}
token spanning the whole connection string which gets totally encrypted after the first run.
If you look at Nullable section of appsetting.development.json, you can find some interesting keys:
Int, DateTime, Double, Bool
: These keys contain respectively an integer, a datetime, a double and a boolean but they are all stored as astring
using a singleProtect:{<data to encrypt>}
tag. Hey wait, how is this possible?Well, chiefly, all the
ConfigurationProviders
convert initially anyConfigurationSource
into aDictionary<String,String>
in theirLoad
method (please see the property Data of the framework ConfigurationProvider base abstract class, theLoad
method also flattens all the hierarchical path to the key into astring
separated by a colon, so for exampleNullable->Int
becomesNullable:Int
). Only later, this dictionary gets converted and binded to a strongly typed class.The decryption process of
ProtectedConfigurationProvider
happens in the middle, so it's transparent for the user and moreover is available on any simple variable type (DateTime
,bool
, etc.). For now, the full encryption of a whole array is not supported, but you can however encrypt a single element converting the array to an array of strings (have a look atDoubleArray
key, note also that in this key it is exemplified the use of per configuration value subkey, in fact the array contains twice the same3.14
value ["Protect:{3.14}"
and"Protect:{%customSubPurpose%}:{3.14}"
] but it is encrypted with two different keys and different salt.
AddXmlFile
: to add the XML file appsettings.xmlAddInMemoryCollection
: to add an in memory dictionaryAddEnvironmentVariables
: to add environment variables
The main code of the Protected.ConsoleTest
console application is:
public static void Main(String[] args)
{
args = new String[] { "--EncryptedCommandLinePassword", "Protect:{secretArgPassword!\\*+?|{[()^$.#}", "--PlainTextCommandLinePassword", "secretArgPassword!\\*+?|{[()^$.#" };
// define the DI services: setup Data Protection API
var servicesDataProtection = new ServiceCollection();
ConfigureDataProtection(servicesDataProtection.AddDataProtection());
var serviceProviderDataProtection = servicesDataProtection.BuildServiceProvider();
// creates all the DataProtectionAPIProtectConfigurationData classes specifying three different provider configurations
// standard configuration using key number purpose
var standardProtectConfigurationData = new DataProtectionAPIProtectConfigurationData(serviceProviderDataProtection);
// standard configuration using key number purpose overridden with a custom tokenization
var otherProtectedTokenizationProtectConfigurationData = new DataProtectionAPIProtectConfigurationData(serviceProviderDataProtection,2, protectRegexString: "OtherProtect(?<subPurposePattern>(:{(?<subPurpose>[^:}]+)})?):{(?<protectData>.+?)}", protectedRegexString: "OtherProtected(?<subPurposePattern>(:{(?<subPurpose>[^:}]+)})?):{(?<protectedData>.+?)}", protectedReplaceString: "OtherProtected${subPurposePattern}:{${protectedData}}");
// standard configuration using string purpose
var magicPurposeStringProtectConfigurationData = new DataProtectionAPIProtectConfigurationData(serviceProviderDataProtection, "MagicPurpose");
// activates JsonWithCommentsProtectFileProcessor
ConfigurationBuilderExtensions.UseJsonWithCommentsProtectFileOption();
// define in-memory configuration key-value pairs to be encrypted
var memoryConfiguration = new Dictionary<String, String>
{
["EncryptedInMemorySecretKey"] = "Protect:{InMemory MyKey Value}",
["PlainTextInMemorySecretKey"] = "InMemory MyKey Value",
["TransientFaultHandlingOptions:Enabled"] = bool.FalseString,
["Logging:LogLevel:Default"] = "Protect:{Warning}",
["UserDomain"] = "Protect:{DOMAIN\\USER}",
["EncryptedInMemorySpecialCharacters"] = "Protect:{\\!*+?|{[()^$.#}",
["PlainTextInMemorySpecialCharacters"] = "\\!*+?|{[()^$.#"
};
// define an environment variable to be encrypted
Environment.SetEnvironmentVariable("EncryptedEnvironmentPassword", "Protect:{SecretEnvPassword\\!*+?|{[()^$.#}");
Environment.SetEnvironmentVariable("PlainTextEnvironmentPassword", "SecretEnvPassword\\!*+?|{[()^$.#");
// encrypts all configuration sources (must be done before reading the configuration)
// encrypts all Protect:{<data>} token tags inside command line argument (you can use also the same method to encrypt String, IEnumerable<String>, IDictionary<String,String> value of any configuration source
var encryptedArgs = standardProtectConfigurationData.ProtectConfigurationValue(args);
// encrypts all Protect:{<data>} token tags inside im-memory dictionary
magicPurposeStringProtectConfigurationData.ProtectConfigurationValue(memoryConfiguration);
// encrypts all Protect:{<data>} token tags inside .json files and all OtherProtect:{<data>} inside .xml files
var encryptedJsonFiles = standardProtectConfigurationData.ProtectFiles(".");
var encryptedXmlFiles = otherProtectedTokenizationProtectConfigurationData.ProtectFiles(".", searchPattern: "*.xml");
// encrypts all Protect:{<data>} token tags inside environment variables
magicPurposeStringProtectConfigurationData.ProtectEnvironmentVariables();
// please check that all configuration source defined above are encrypted (check also Environment.GetEnvironmentVariable("SecretEnvironmentPassword") in Watch window)
// note the per key purpose string override in file appsettings.development.json inside Nullable:DoubleArray contains two elements one with "Protect:{3.14}" and one with "Protect:{%customSubPurpose%}:{3.14}", even though the value is the same (3.14) they are encrypted differently due to the custom key purpose string
// note the per key purpose string override in file appsettings.xml inside TransientFaultHandlingOptions contains two elements AutoRetryDelay with "OtherProtect:{00:00:07}" and AutoRetryDelaySubPurpose with "OtherProtect:{sUbPuRpOsE}:{00:00:07}", even though the value is the same (00:00:07) they are encrypted differently due to the custom key purpose string
Debugger.Break();
// define the application configuration using almost all possible known ConfigurationSources
var configuration = new ProtectedConfigurationBuilder(standardProtectConfigurationData) // global configuration
.AddCommandLine(encryptedArgs)
.AddJsonFile("appsettings.json")
.AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("DOTNETCORE_ENVIRONMENT")}.json", false, true)
.AddXmlFile("appsettings.xml").WithProtectedConfigurationOptions(otherProtectedTokenizationProtectConfigurationData) // overrides global configuration for XML file
.AddInMemoryCollection(memoryConfiguration).WithProtectedConfigurationOptions(magicPurposeStringProtectConfigurationData) // overrides global configuration for in-memory collection file
.AddEnvironmentVariables().WithProtectedConfigurationOptions(magicPurposeStringProtectConfigurationData) // overrides global configuration for enviroment variables file
.Build();
// define other DI services: configure strongly typed AppSettings configuration class (must be done after having read the configuration)
var services = new ServiceCollection();
services.Configure<AppSettings>(configuration);
var serviceProvider = services.BuildServiceProvider();
// retrieve the strongly typed AppSettings configuration class, we use IOptionsMonitor in order to be notified on any reloads of appSettings
var optionsMonitor = serviceProvider.GetRequiredService<IOptionsMonitor<AppSettings>>();
var appSettings = optionsMonitor.CurrentValue;
optionsMonitor.OnChange(appSettingsReloaded =>
{
// this breakpoint gets hit when the appsettings have changed due to a configuration reload, please check that the value of "Int" property inside appSettingsReloaded class is different from the one inside appSettings class
// note that also there is an unavoidable framework bug on ChangeToken.OnChange which could get called multiple times when using FileSystemWatchers see https://github.com/dotnet/aspnetcore/issues/2542
// see also the remarks section of FileSystemWatcher https://learn.microsoft.com/en-us/dotnet/api/system.io.filesystemwatcher.created?view=net-8.0#remarks
Console.WriteLine($"OnChangeEvent: appsettings.{Environment.GetEnvironmentVariable("DOTNETCORE_ENVIRONMENT")}.json has been reloaded! appSettings Int {appSettings.Int} appSettingsReloaded {appSettingsReloaded.Int}");
Debugger.Break();
});
// please check that all values inside appSettings class are actually decrypted with the right value, make a note of the value of "Int" property it will change on the next second breakpoint
Debugger.Break();
// added some simple assertions to test that decrypted value is the same as original plaintext one
Debug.Assert(appSettings.EncryptedCommandLinePassword == appSettings.PlainTextCommandLinePassword);
Debug.Assert(appSettings.EncryptedEnvironmentPassword == appSettings.PlainTextEnvironmentPassword);
Debug.Assert(appSettings.EncryptedInMemorySecretKey == appSettings.PlainTextInMemorySecretKey);
// appsettings.json assertions
Debug.Assert(appSettings.EncryptedJsonSpecialCharacters == appSettings.PlainTextJsonSpecialCharacters);
Debug.Assert(appSettings.ConnectionStrings["PartiallyEncryptedConnectionString"].Contains("(local)\\SECONDINSTANCE"));
Debug.Assert(appSettings.ConnectionStrings["PartiallyEncryptedConnectionString"].Contains("Secret_Catalog"));
Debug.Assert(appSettings.ConnectionStrings["PartiallyEncryptedConnectionString"].Contains("secret_user"));
Debug.Assert(appSettings.ConnectionStrings["PartiallyEncryptedConnectionString"].Contains("secret_password"));
Debug.Assert(appSettings.ConnectionStrings["FullyEncryptedConnectionString"].Contains("Data Source=server1\\THIRDINSTANCE; Initial Catalog=DB name; User ID=sa; Password=pass5678; MultipleActiveResultSets=True;"));
// appsettings.development.json assertions
Debug.Assert(appSettings.Nullable.DateTime.Value.ToUniversalTime() == new DateTime(2016, 10, 1, 20, 34, 56, 789, DateTimeKind.Utc));
Debug.Assert(appSettings.Nullable.Double == 123.456);
Debug.Assert(appSettings.Nullable.Int == 98765);
Debug.Assert(appSettings.Nullable.Bool == true);
Debug.Assert(appSettings.Nullable.DoubleArray[1] == 3.14);
Debug.Assert(appSettings.Nullable.DoubleArray[3] == 3.14);
// appsettings.xml assertions
Debug.Assert(appSettings.TransientFaultHandlingOptions["AutoRetryDelay"] == appSettings.TransientFaultHandlingOptions["AutoRetryDelaySubPurpose"]);
Debug.Assert(appSettings.Logging.LogLevel["Microsoft"] == "Warning");
Debug.Assert(appSettings.EncryptedXmlSecretKey == appSettings.PlainTextXmlSecretKey);
// multiple configuration reload example (in order to check that the ReloadToken re-registration works)
int i = 0;
while (i++ < 5)
{
// updates inside appsettings.<environment>.json the property "Int": <whatever>, --> "Int": "Protected:{<random number>},"
var environmentAppSettings = File.ReadAllText($"appsettings.{Environment.GetEnvironmentVariable("DOTNETCORE_ENVIRONMENT")}.json");
environmentAppSettings = new Regex("\"Int\":.+?,").Replace(environmentAppSettings, $"\"Int\": \"{standardProtectConfigurationData.ProtectConfigurationValue($"Protect:{{{new Random().Next(0, 1000000)}}}")}\",");
File.WriteAllText($"appsettings.{Environment.GetEnvironmentVariable("DOTNETCORE_ENVIRONMENT")}.json", environmentAppSettings);
// wait 5 seconds for the reload to take place, please check on this breakpoint that the value of "Int" property has changed in appSettings class and it is the same of appSettingsReloaded
Thread.Sleep(5000);
appSettings = optionsMonitor.CurrentValue;
Console.WriteLine($"ConfigurationReloadLoop: appSettings Int {appSettings.Int}");
Debugger.Break();
}
}
The above code is quite simple and commented, if you launch it in Debug mode, it will automatically break into most significant points by using Debugger.Break()
:
- The first one happens after all six configuration sources values have been encrypted by replacing the default tokenization tag
Protect:{<data to encrypt>}
withProtected:{<encrypted data>}
(for files, please check those inside bin/Debug/net6.0 folder and not inside the solution dir which remain unchanged)- CommandLine arguments
args
gets encrypted inencryptedArgs
variable by using the provided extension methodIProtectProviderConfigurationData.ProtectConfigurationValue
- The appsettings.*json files have been backed up in a .bak file and encrypted by using the provided extension method
IProtectProviderConfigurationData.ProtectFiles
- The appsettings.xml file has been backed up in a .bak file and has been encrypted by using the provided extension method
IProtectProviderConfigurationData.ProtectFiles
- The environment variables have got encrypted by using
IProtectProviderConfigurationData.ProtectEnvironmentVariables
- CommandLine arguments
- The second one is the one actually showing the use of
ProtectedConfigurationBuilder
, if you watch in the debugger theappSettings
strongly typed class, you will notice that it magically and automatically contains the decrypted values with the right data type even though the encrypted keys are always stored in configuration sources asstrings
. Please note that I usedIOptionsMonitor<AppSettings>
insteadIOptions<AppSettings>
because I wanted to test also the autodecryption feature on reload of a configuration file whenever it changes on filesystem.
- The third one, the fith one, the seventh one, etc. is inside the
IOptionsMonitor.OnChange
event and happens just after having updated the "Int
" property with a random encrypted integer inside appsettings.development.json file (take note of this value, you will need it again in the next breakpoint), you can see that theappSettingsReloaded
variable contains a different value from the one ofappSettings
variable.
- The fourth one, the sixth one, the eighth one, etc. is after
IOptionsMonitor.OnChange
event and happens after re-assignment toappSettings
variable of the current strongly typed configuration class fromIOptionsMonitor
, you can check thatappSettings
contains the same new value for "Int
" property as that ofappSettingsReloaded
variable at previous breakpoint.
So summing up in order to use this package, we had just to use to replace the call new ConfigurationBuilder()
with a call to new ProtectedConfigurationBuilder()
, pass the Data Protection API configuration and an eventual custom tokenization tag, and everything works flawlessly in a transparent way. Moreover, all the decryption happens in memory and nothing is stored on disk for any reason.
Implementation Details
I explain here the main points of the implementation:
Code inside Fededim.Extensions.Configuration.Protected package
- Extensions methods defined inside
ConfigurationBuilderExtensions
:ProtectFiles
is an extension method ofIProtectProviderConfigurationData
which gets called and scans all files according to the supplied extensionsearchPattern
inside the suppliedpath
forProtect:{<data to encrypt>}
tokens, encrypts enclosed data, performs the replacement withProtected:{<encrypted data>}
and saves back the file after having created an optional backup of the original file with the .bak extension. Again, if you do not like the default tokenization regular expression, you can pass your own one with the constraint that it must extract the<data to encrypt>
substring in a group calledprotectData
and optionally the <subPurposePattern>
and<subPurpose>
substrings in two groups called respectivelysubPurposePattern
andsubPurpose.
/// <summary>
/// Perform wildcard search of files in path encrypting any data using the specified <see cref="protectProviderConfigurationData"/>
/// </summary>
/// <param name="protectProviderConfigurationData">an IProtectProviderConfigurationData interface obtained from a one of the supported providers</param>
/// <param name="path">directory to be searched</param>
/// <param name="searchPattern">wildcard pattern to filter files</param>
/// <param name="searchOption">search options</param>
/// <param name="backupOriginalFile">boolean which indicates whether to make a backupof original file with extension .bak</param>
/// <returns>a list of filenames which have been successfully encrypted</returns>
/// <exception cref="ArgumentException"></exception>
public static IList<String> ProtectFiles(this IProtectProviderConfigurationData protectProviderConfigurationData, String path, String searchPattern = "*.json", SearchOption searchOption = SearchOption.TopDirectoryOnly, bool backupOriginalFile = true)
{
protectProviderConfigurationData.CheckConfigurationIsValid();
var result = new List<String>();
foreach (var f in Directory.EnumerateFiles(path, searchPattern, searchOption))
{
var fileContent = File.ReadAllText(f);
var replacedContent = fileContent;
foreach (var protectFileOption in ProtectFilesOptions)
if (protectFileOption.FilenameRegex.Match(f).Success)
{
replacedContent = protectFileOption.ProtectFileProcessor.ProtectFile(fileContent, protectProviderConfigurationData.ProtectRegex, (value) => ProtectConfigurationValue(protectProviderConfigurationData, value));
break;
}
if (replacedContent != fileContent)
{
if (backupOriginalFile)
File.Copy(f, f + ".bak", true);
File.WriteAllText(f, replacedContent);
result.Add(f);
}
}
return result;
}
The format of input files is decoded, encrypted and re-encoded according to the processor specified in the public static property ConfigurationBuilderExtensions.ProtectFilesOptions
, by default three processors are provided, two for JSON files (e.g. \\ becomes \, etc., see JsonProtectFileProcessor
and JsonWithCommentsProtectFileProcessor
class in ProtectFileProcessors.cs
) and one for XML files (e.g. > becomes >, etc., see XmlProtectFileProcessor
class again in ProtectFileProcessors.cs
) and one for RAW text files (no decoding is done, see RawProtectFileProcessor
class again in ProtectFileProcessors.cs
). You can add as many processors as you wish by adding records to the ConfigurationBuilderExtensions.ProtectFilesOptions
list specifying a filenameRegex
which if matched applies the associated ProtectFileProcessor
, e.g. a class which must implement the IProtectFileProcessor
interface (see below). This interface has essentially just one method ProtectFile
which essentially takes in input 3 parameters passed to it by the ConfigurationBuilderExtensions.ProtectFiles
method:
rawFileText
: this is the raw input file as a stringprotectRegex
: this is the configured protected regex which must be matched in file configuration values in order to choose whether to encrypt or not the data.protectFunction:
this is the protect function which encrypts data taking the plaintext data as input and producing encrypted base64 string as output.
This ProtectFile
method must decode, encrypt and re-encode the input file and return it as a string as final output. Note that all the file processors are processed in FIFO (First-In First-Out) order so keep this in mind when configuring your additional handlers.
/// <summary>
/// This is the interface which must be implemented by a custom ProtectFileProcessor. It contains a single method <see cref="ProtectFile"/> used to decode, encrypt and re-encode the input file and return it as string.
/// </summary>
public interface IProtectFileProcessor
{
/// <summary>
/// This method actually implements a custom ProtectFileProcessor which must decode, encrypt and re-encode the input file and return it as string.
/// </summary>
/// <param name="rawFileText">The is the raw input file as a string</param>
/// <param name="protectRegex">This is the configured protected regex which must be matched in file values in order to choose whether to encrypt or not the data.</param>
/// <param name="protectFunction">This is the protect function taking the plaintext data as input and producing encrypted base64 data as output</param>
/// <returns>the encrypted re-encoded file as a string</returns>
String ProtectFile(String rawFileText, Regex protectRegex, Func<String, String> protectFunction);
}
ProtectEnvironmentVariables
is an extension method ofIProtectProviderConfigurationData
which encrypts the environment variables with the same criteria./// <summary> /// Encrypts all the environment variables using the specified <see cref="protectProviderConfigurationData"/> (used for environment variables) /// </summary> /// <param name="protectProviderConfigurationData">an IProtectProviderConfigurationData interface obtained from a one of the supported providers</param> public static void ProtectEnvironmentVariables(this IProtectProviderConfigurationData protectProviderConfigurationData, EnvironmentVariableTarget environmentTarget = EnvironmentVariableTarget.User) { var environmentVariables = Environment.GetEnvironmentVariables(environmentTarget); foreach (String key in environmentVariables.Keys) Environment.SetEnvironmentVariable(key, protectProviderConfigurationData.ProtectConfigurationValue(environmentVariables[key].ToString())); }
ProtectConfigurationValue
is an extension method ofIProtectProviderConfigurationData
which encrypts astring
, anIEnumerable<string>
, astring[]
or aDictionary<string,string>
with the same criteria. The ultimate method responsible for the actual encryption isProtectConfigurationValueInternal
(it is the only private method since static classes do not have protected members)
/// <summary>
/// Encrypts the String value using the specified <see cref="protectProviderConfigurationData"/>
/// </summary>
/// <param name="protectProviderConfigurationData">an IProtectProviderConfigurationData interface obtained from a one of the supported providers</param>
/// <param name="value">a String literal which needs to be encrypted</param>
/// <returns>the encrypted configuration value</returns>
/// <exception cref="ArgumentException"></exception>
public static String ProtectConfigurationValue(this IProtectProviderConfigurationData protectProviderConfigurationData, String value)
{
return ProtectConfigurationValueInternal(protectProviderConfigurationData, value);
}
/// <summary>
/// internal method actually performing the encryption using the specified <see cref="protectProviderConfigurationData"/>
/// </summary>
/// <param name="protectProviderConfigurationData">an IProtectProviderConfigurationData interface obtained from a one of the supported providers</param>
/// <param name="value">a String literal which needs to be encrypted</param>
/// <returns></returns>
private static String ProtectConfigurationValueInternal(IProtectProviderConfigurationData protectProviderConfigurationData, String value)
{
if (value == null)
return null;
protectProviderConfigurationData.CheckConfigurationIsValid();
return protectProviderConfigurationData.ProtectRegex.Replace(value, (me) =>
{
var subPurposePresent = !String.IsNullOrEmpty(me.Groups["subPurpose"]?.Value);
var protectProvider = protectProviderConfigurationData.ProtectProvider;
if (subPurposePresent)
protectProvider = protectProviderConfigurationData.ProtectProvider.CreateNewProviderFromSubkey(me.Groups["subPurpose"].Value);
return protectProviderConfigurationData.ProtectedReplaceString.Replace("${subPurposePattern}", subPurposePresent ? me.Groups["subPurposePattern"].Value : String.Empty).Replace("${protectedData}", protectProvider.Encrypt(me.Groups["protectData"].Value));
});
}
/// <summary>
/// Encrypts the Dictionary<String, String> initialData using the specified <see cref="protectProviderConfigurationData"/> (used for in-memory collections)
/// </summary>
/// <param name="protectProviderConfigurationData">an IProtectProviderConfigurationData interface obtained from a one of the supported providers</param>
/// <param name="initialData">a Dictionary<String, String> whose values need to be encrypted</param>
public static void ProtectConfigurationValue(this IProtectProviderConfigurationData protectProviderConfigurationData, Dictionary<String, String> initialData)
{
if (initialData != null)
foreach (var key in initialData.Keys.ToList())
initialData[key] = protectProviderConfigurationData.ProtectConfigurationValue(initialData[key]);
}
/// <summary>
/// Encrypts the IEnumerable<String> arguments using the specified <see cref="protectProviderConfigurationData"/>
/// </summary>
/// <param name="protectProviderConfigurationData">an IProtectProviderConfigurationData interface obtained from a one of the supported providers</param>
/// <param name="arguments">a IEnumerable<String> whose elements need to be encrypted</param>
/// <returns>a newer encrypted IEnumerable<String></returns>
public static IEnumerable<String> ProtectConfigurationValue(this IProtectProviderConfigurationData protectProviderConfigurationData, IEnumerable<String> arguments)
{
return arguments?.Select(argument => protectProviderConfigurationData.ProtectConfigurationValue(argument));
}
/// <summary>
/// Encrypts the String[] arguments using the specified <see cref="protectProviderConfigurationData"/> (used for command-line arguments)
/// </summary>
/// <param name="protectProviderConfigurationData">an IProtectProviderConfigurationData interface obtained from a one of the supported providers</param>
/// <param name="arguments">a String array whose elements need to be encrypted</param>
/// <returns>a newer encrypted String[] array</returns>
public static String[] ProtectConfigurationValue(this IProtectProviderConfigurationData protectProviderConfigurationData, String[] arguments)
{
return arguments?.Select(argument => protectProviderConfigurationData.ProtectConfigurationValue(argument)).ToArray();
}
WithProtectedConfigurationOptions:
It is an extension method ofIConfigurationBuilder
which allows to override the Data Protection or tokenization tag configuration for a particularConfigurationSource
(e.g., the last one added). Note that this method is a little bit hacky: I was not able to change the return type ofProtectedConfigurationBuilder.Add
, otherwise theIConfigurationBuilder
interface wouldn't be implemented; thusWithProtectedConfigurationOptions
extends the standardIConfigurationBuilder
interface and converts it toIProtectedConfigurationBuilder
interface and calls theProtectedConfigurationBuilder.WithProtectedConfigurationOptions
method, if instead the providedIConfigurationBuilder
is not an instance ofIProtectedConfigurationBuilder
it raises an exception remembering to replace thenew ConfigurationBuilder
instantiation with newProtectedConfigurationBuilder
. This method takes as input just one parameter: aIProtectProviderConfigurationData
class configured properly which overrides the global configuration specified in theProtectedConfigurationBuilder
constructor./// <summary> /// WithProtectedConfigurationOptions is a helper method used to override the ProtectedGlobalConfigurationData for a particular provider (e.g. the last one added) /// </summary> /// <param name="configurationBuilder">the IConfigurationBuilder instance</param> /// <param name="protectProviderLocalConfigurationData">a regular expression which captures the data to be decrypted in a named group called protectedData</param> /// <returns>The <see cref="IConfigurationBuilder"/> interface for method chaining</returns> /// <exception cref="ArgumentException">if configurationBuilder is not an instance of ProtectedConfigurationBuilder class</exception> public static IConfigurationBuilder WithProtectedConfigurationOptions(this IConfigurationBuilder configurationBuilder, IProtectProviderConfigurationData protectProviderLocalConfigurationData) { var protectedConfigurationBuilder = configurationBuilder as IProtectedConfigurationBuilder; if (protectedConfigurationBuilder != null) return protectedConfigurationBuilder.WithProtectedConfigurationOptions(protectProviderLocalConfigurationData); else throw new ArgumentException("Please use ProtectedConfigurationBuilder instead of ConfigurationBuilder class!", nameof(configurationBuilder)); }
ProtectedConfigurationBuilder
implements theIConfigurationBuilder
interface likeConfigurationBuilder
framework class (from which part of the implementation is borrowed), the main difference is theBuild
method which elementally proxies through composition theIConfigurationProvider
returned from theIConfigurationSource.Build
method by passing it as a constructor parameter to the core class responsible for in memory transparent decryption:ProtectedConfigurationProvider
. It also performs the merge between the custom configuration specified for theIConfigurationSource
being converted intoIConfigurationProvider
(if you want to know how, check theProtectProviderConfigurationData.Merge
static
method) and the global configuration specified in theProtectedConfigurationBuilder
constructor.
/// <summary>
/// Builds an <see cref="IConfiguration"/> with keys and values from the set of configuration sources registered in <see cref="Sources"/>.
/// </summary>
/// <returns>An <see cref="IConfigurationRoot"/> with keys and values from the providers generated by registered configuration sources.</returns>
public virtual IConfigurationRoot Build()
{
var providers = new List<IConfigurationProvider>();
foreach (IConfigurationSource source in _sources)
{
IConfigurationProvider provider = source.Build(this);
// if we have a custom configuration we move the index from the ConfigurationSource object to the newly created ConfigurationProvider object
ProtectProviderLocalConfigurationData.TryGetValue(source.GetHashCode(), out var protectedConfigurationData);
if (protectedConfigurationData != null)
{
ProtectProviderLocalConfigurationData[provider.GetHashCode()] = protectedConfigurationData;
ProtectProviderLocalConfigurationData.Remove(source.GetHashCode());
}
providers.Add(CreateProtectedConfigurationProvider(provider));
}
return new ConfigurationRoot(providers);
}
/// <summary>
/// CreateProtectedConfigurationProvider creates a new ProtectedConfigurationProvider using the composition approach
/// </summary>
/// <param name="provider">an existing IConfigurationProvider to instrument in order to perform the decryption of the encrypted keys</param>
/// <returns>a newer decrypted <see cref="IConfigurationProvider"/> if we have a valid protected configuration data, otherwise it returns the existing original undecrypted provider</returns>
protected virtual IConfigurationProvider CreateProtectedConfigurationProvider(IConfigurationProvider provider)
{
// this code is an initial one of when I was thinking of casting IConfigurationProvider to ConfigurationProvider (all MS classes derive from this one)
// in order to retrieve all configuration keys inside DecryptChildKeys using the Data property (through reflection since it is protected) without using the recursive "hack" of GetChildKeys
// it has been commented because it is not needed anymore, but I keep it as workaround for accessing all configuration keys just in case MS changes the implementation of GetChildKeys "forbidding" the actual way
//var providerType = provider.GetType();
//if (!providerType.IsSubclassOf(typeof(ConfigurationProvider)))
// return provider;
// we merge ProtectedProviderGlobalConfigurationData and ProtectProviderLocalConfigurationData
var actualProtectedConfigurationData = ProtectProviderLocalConfigurationData.ContainsKey(provider.GetHashCode()) ? ProtectProviderConfigurationData.Merge(ProtectedProviderGlobalConfigurationData, ProtectProviderLocalConfigurationData[provider.GetHashCode()]) : ProtectedProviderGlobalConfigurationData;
// we use composition to perform decryption of all provider values
return new ProtectedConfigurationProvider(provider, actualProtectedConfigurationData);
}
- IProtectProviderConfigurationData This class is an abstract class for specifying the configuration options and plugging the providers into
ProtectedConfigurationBuilder.
/// <summary> /// an abstract class for specifying the configuration data of the encryption/decryption provider /// </summary> public abstract class IProtectProviderConfigurationData { public const String DefaultProtectRegexString = "Protect(?<subPurposePattern>(:{(?<subPurpose>[^:}]+)})?):{(?<protectData>.*?)}"; public const String DefaultProtectedRegexString = "Protected(?<subPurposePattern>(:{(?<subPurpose>[^:}]+)})?):{(?<protectedData>.*?)}"; public const String DefaultProtectedReplaceString = "Protected${subPurposePattern}:{${protectedData}}"; /// <summary> /// The actual provider performing the encryption/decryption, <see cref="IProtectProvider"/> interface /// </summary> public IProtectProvider ProtectProvider { get; protected set; } /// <summary> /// a regular expression which captures the data to be decrypted in a named group called protectData /// </summary> public Regex ProtectedRegex { get; protected set; } /// <summary> /// a regular expression which captures the data to be encrypted in a named group called protectData /// </summary> public Regex ProtectRegex { get; protected set; } /// <summary> /// a string replacement expression which captures the substitution which must be applied for transforming unencrypted tokenization <see cref="DefaultProtectRegexString" /> into an encrypted tokenization <see cref="DefaultProtectedRegexString" /> /// </summary> public String ProtectedReplaceString { get; protected set; } /// <summary> /// A helper overridable method for checking that the configuation data is valid (e.g. ProtectProvider is not null, ProtectedRegex and ProtectRegex contains both a regex group named protectedData) /// </summary> public virtual void CheckConfigurationIsValid() { ProtectRegex = ProtectRegex ?? new Regex(DefaultProtectRegexString); if (!ProtectRegex.GetGroupNames().Contains("protectData")) throw new ArgumentException("ProtectRegex must contain a group named protectedData!", nameof(ProtectRegex)); ProtectedRegex = ProtectedRegex ?? new Regex(DefaultProtectedRegexString); if (!ProtectedRegex.GetGroupNames().Contains("protectedData")) throw new ArgumentException("ProtectedRegex must contain a group named protectedData!", nameof(ProtectedRegex)); ProtectedReplaceString = !String.IsNullOrEmpty(ProtectedReplaceString) ? ProtectedReplaceString : DefaultProtectedReplaceString; if (!ProtectedReplaceString.Contains("${protectedData}")) throw new ArgumentException("ProtectedReplaceString must contain ${protectedData}!", nameof(ProtectedReplaceString)); if (ProtectProvider == null) throw new ArgumentException("ProtectProvider must not be null!", nameof(ProtectProvider)); } }
As we can see the
IProtectProviderConfigurationData
fundamentally contains four properties and one method:ProtectedRegex
: It is a regular expression which specifies the tokenization tag which encloses the encrypted data to be decrypted; it must define a named group calledprotectedData
(and optionally two additional groups calledsubPurposePattern
andsubPurpose
for specifying a per configuration value subkey). Ifnull
, this parameter assumes the default value:public const String DefaultProtectedRegexString = "Protected(?<subPurposePattern>(:{(?<subPurpose>[^:}]+)})?):{(?<protectedData>.*?)}";
The above regular expression essentially searches in a lazy way (so it can retrieve all the occurrences inside a value) for any
string
matching the pattern'Protected:{<subPurpose>}:{<encrypted data>}'
and extracts the<encrypted data>
substring storing it inside a group namedprotectedData
. There is also an optional part called <subPurposePattern>
(made up of:{<subPurpose>}
)<encrypted data>
substring in a group calledprotectedData
and the <subPurposePattern>
and<subPurpose>
substrings in two groups called respectivelysubPurposePattern
andsubPurpose.
-
ProtectRegex
: It is a regular expression which specifies the tokenization tag which encloses the data to be encrypted; again it must define a named group called this timeprotectData
(and optionally two additional groups calledsubPurposePattern
andsubPurpose
for specifying a per configuration value subkey). Ifnull
, this parameter assumes the default value (e.g.Protect:{<subPurpose>}:{<data to be encrypted>}
):public const String DefaultProtectRegexString = "Protect(?<subPurposePattern>(:{(?<subPurpose>[^:}]+)})?):{(?<protectData>.*?)}";
-
DefaultProtectedReplaceString:
It is a string expression used to transform the plaintext tokenization into the encrypted tokenization (e.g. fromProtect:{<subPurpose>}:{<data to be encrypted>}
intoProtected:{<subPurpose>}:{<encrypted data>}
). It contains two placeholders${subPurposePattern}
and${protectedData}
which gets substituted respectively with the subPurposePattern (if present) and the encrypted data. Ifnull
, this parameter assumes the default value:
public const String DefaultProtectedReplaceString = "Protected${subPurposePattern}:{${protectedData}}";
ProtectProvider:
a class implementing theIProtectProvider
interface for specifying a pluggable logic for encryption/decryption. This interface is made up of three methods:Encrypt
: which takes a plain text string and returns an encrypted base64 string.-
Decrypt
: which takes a base64 encrypted string and returns the plain text string -
CreateNewProviderFromSubkey
: for supporting the per configuration value encryption derived subkey. It takes a subkey string parameter which should be used as encryption subkey to create a new derivedIProtectProvider
interface.
/// <summary>
/// A standard interface which must implement by an encryption/decryption provider
/// </summary>
public interface IProtectProvider
{
/// <summary>
/// This method encrypts a plain-text string
/// </summary>
/// <param name="plainTextValue">the plain-text string to be encrypted</param>
/// <returns>the encrypted string</returns>
String Encrypt(String plainTextValue);
/// <summary>
/// This method decrypts an encrypted string
/// </summary>
/// <param name="encryptedValue">the encrypted string to be decrypted</param>
/// <returns>the decrypted string</returns>
String Decrypt(String encryptedValue);
/// <summary>
/// This methods create a new <see cref="IProtectProvider"/> for supporting per configuration value encryption subkey (e.g. "subpurposes")
/// </summary>
/// <param name="subkey">the per configuration value encryption subkey</param>
/// <returns>a derived <see cref="IProtectProvider"/> based on the <see cref="subkey"/> parameter</returns>
IProtectProvider CreateNewProviderFromSubkey(string subkey);
}
CheckConfigurationIsValid
:
it is a base virtual (e.g. overrideable) method which basically checks that theIProtectProviderConfigurationData
configuration is valid, if not it raises an exception with the details. It is called every time you add anIProtectProviderConfigurationData
toProtectedConfigurationBuilder,
either globally in the constructor either locally with the extension methodWithProtectedConfigurationOptions.
Moreover it is also called duringProtectProviderConfigurationData.Merge
to check that the resulting configuration is valid and when performing encryption withIProtectProviderConfigurationData.Protect...
methods.
ProtectedConfigurationProvider
: This class performs the actual transparent decryption of encrypted configuration values stored inside any existing configuration source. It should also support even future configuration sources as long as the implementation of the GetChildKeys of Microsoft.Extensions.Configuration.ConfigurationProvider does not change (I principally use it to enumerate all possible configuration keys). Essentially, it takes in its constructor as input theIConfigurationProvider
that needs to be decrypted and acts as a proxy:-
It redefines the
Load
method by calling after inputIConfigurationProvider.Load
the method responsible for the actual decryptionDecryptChildKeys
. This method initially used the standard methodGetChildKeys
offered by theIConfigurationProvider
interface to enumerate all the existing configuration keys and decrypt them by matching theProtectedRegex
stored insideProtectProviderConfigurationData
and usingDecrypt
method ofProtectProvider
. If thesubPurpose
group is present in the regex, a temporary derivedProtectProvider
is created usingCreateNewProviderFromSubkey
method and used to perform decryption. With version 1.0.15 of the package I added a faster method for decrypting the child keys, it usesreflection
just to retrieve the key-value dictionary inside theConfigurationProvider.Data
protected property (reflection should be avoided as much as possible, but in this case it is needed since the property is not public, moreover it is only used to retrieve only the dictionary, not the key-values entries). The speed increase is around 3000 times (yes 3k times!) and it should be supported by all Microsoft providedConfigurationProviders
(e.g.XmlConfigurationProvider
,JsonConfigurationProvider
,MemoryConfigurationProvider
,CommandLineConfigurationProvider
,EnvironmentVariablesConfigurationProvider
) . This method is hacky but it is used in a safe mode, e.g. if theIConfigurationProvider
objectConfigurationProvider
class and the class does not contain a property namedData
the old slower recursive method is used./// <summary> /// Calls the underlying provider Load method in order to /// load configuration values and then decrypts them by calling /// DecryptChildKeys helper method /// </summary> public virtual void Load() { Provider.Load(); // call DecryptChildKeys after Load DecryptChildKeys(); } /// <summary> /// Static PropertyInfo of protected property Data of Microsoft.Extensions.Configuration.ConfigurationProvider class (even though it is protected and not available here, you can use reflection in order to retrieve its value) /// </summary> public static PropertyInfo ConfigurationProviderDataProperty = typeof(ConfigurationProvider).GetProperty("Data", BindingFlags.NonPublic | BindingFlags.Instance); /// <summary> /// Hacky and fastest, tough safe method which gives access to the provider Data dictionary in readonly mode (it could be null in the future or for other providers not deriving from ConfigurationProvider, be sure to always check that it is not null!) /// </summary> public IReadOnlyDictionary<String, String> ProviderDataReadOnly { get { IDictionary<String, String> providerData = ProviderData; if (providerData != null) return new ReadOnlyDictionary<String, String>(providerData); return null; } } /// <summary> /// Hacky and fastest, tough safe method which gives access to the provider Data dictionary (it could be null in the future or for other providers not deriving from ConfigurationProvider, be sure to always check that it is not null!) /// </summary> protected IDictionary<String, String> ProviderData { get { IDictionary<String, String> providerData = null; if (Provider is ConfigurationProvider && ConfigurationProviderDataProperty != null) providerData = ConfigurationProviderDataProperty.GetValue(Provider) as IDictionary<string, string>; return providerData; } } /// <summary> /// This is a helper method actually responsible for the decryption of all configuration values. It decrypts all values using just IConfigurationBuilder interface methods so it should work on any existing or even future IConfigurationProvider <br /><br /> /// Note: unluckily there Data dictionary property of ConfigurationProvider is not exposed on the interface IConfigurationProvider, but we can manage to get all keys by using the GetChildKeys methods, look at its implementation <see href="https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.Extensions.Configuration/src/ConfigurationProvider.cs#L61-L94"/> <br /><br /> /// The only drawback of this method is that it returns the child keys of the level of the hierarchy specified by the parentPath parameter (it's at line 84 in MS source code "Segment(kv.Key, parentPath.Length + 1)" <see href="https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.Extensions.Configuration/src/ConfigurationProvider.cs#L84"/>) <br /> /// So you have to use a recursive function to gather all existing keys and also to issue a distinct due to the way the GetChildKeys method has been implemented <br /> /// </summary> /// <param name="parentPath"></param> protected void DecryptChildKeys(String parentPath = null) { IDictionary<String, String> dataProperty; // this is a hacky yet safe way to speed up key enumeration // we access the Data dictionary of ConfigurationProvider using reflection avoiding enumerating all keys with recursive function // the speed improvement is more 3000 times! if (((dataProperty = ProviderData) != null)) { foreach (var key in dataProperty.Keys.ToList()) { if (!String.IsNullOrEmpty(dataProperty[key])) Provider.Set(key, ProtectProviderConfigurationData.ProtectedRegex.Replace(dataProperty[key], me => { var subPurposePresent = !String.IsNullOrEmpty(me.Groups["subPurpose"]?.Value); IProtectProvider protectProvider = ProtectProviderConfigurationData.ProtectProvider; if (subPurposePresent) protectProvider = protectProvider.CreateNewProviderFromSubkey(me.Groups["subPurpose"].Value); return protectProvider.Decrypt(me.Groups["protectedData"].Value); })); } } else { foreach (var key in Provider.GetChildKeys(new List<String>(), parentPath).Distinct()) { var fullKey = parentPath != null ? $"{parentPath}:{key}" : key; if (Provider.TryGet(fullKey, out var value)) { if (!String.IsNullOrEmpty(value)) Provider.Set(fullKey, ProtectProviderConfigurationData.ProtectedRegex.Replace(value, me => { var subPurposePresent = !String.IsNullOrEmpty(me.Groups["subPurpose"]?.Value); IProtectProvider protectProvider = ProtectProviderConfigurationData.ProtectProvider; if (subPurposePresent) protectProvider = protectProvider.CreateNewProviderFromSubkey(me.Groups["subPurpose"].Value); return protectProvider.Decrypt(me.Groups["protectedData"].Value); })); } else DecryptChildKeys(fullKey); } } }
It creates its own
ReloadToken
if the underlyingIConfigurationProvider
supports it, it returns it in theGetReloadToken
method and finally it registers a callback to the inputIConfigurationProvider
reload token using the framework static utility methodChangeToken.OnChange
class in order to be notified of any configuration change re-performing the decryption in order to support to automatic decryption of values on configuration reloadpublic ProtectedConfigurationProvider(IConfigurationProvider provider, ProtectedConfigurationData protectedConfigurationData) { Provider = provider; ProtectedConfigurationData = protectedConfigurationData; RegisterReloadCallback(); } /// <summary> /// Registers a reload callback which redecrypts all values if the underlying IConfigurationProvider supports it /// </summary> protected void RegisterReloadCallback() { // check if underlying provider supports reloading if (Provider.GetReloadToken() != null) { // Create our reload token ReloadToken = new ConfigurationReloadToken(); // registers Provider on Change event using framework static utility method ChangeToken.OnChange in order to be notified of configuration reload and redecrypts subsequently the needed keys ProviderReloadTokenRegistration = ChangeToken.OnChange(() => Provider.GetReloadToken(), (configurationProvider) => { var protectedConfigurationProvider = configurationProvider as ProtectedConfigurationProvider; // redecrypts all needed keys protectedConfigurationProvider.DecryptChildKeys(); // notifies all subscribes OnReload(); }, this); } } /// <summary> /// Returns our reload token /// </summary> /// <returns>the <see cref="ReloadToken"/></returns> public IChangeToken GetReloadToken() { return ReloadToken; } /// <summary> /// Dispatches all the callbacks waiting for the reload event from /// this configuration provider (and creates a new ReloadToken) /// </summary> protected virtual void OnReload() { ConfigurationReloadToken previousToken = Interlocked.Exchange(ref ReloadToken, new ConfigurationReloadToken()); previousToken.OnReload(); }
-
Code inside Fededim.Extensions.Configuration.Protected.DataProtectionAPI package
DataProtectionAPIProtectConfigurationData
: This class chiefly stores either the global configuration or theConfigurationSource
specific one for Data Protection API and the eventual custom tokenization regular expressions. It also sets up another dependency injection provider in the main constructor (see above for the reason) and it provides two overloads one forkeyNumber
and one forpurposeString
. Besides these two main constructors there are several other overloads provided for usability./// <summary> /// Main constructor for DataProtectionAPIProtectConfigurationData using a key number /// </summary> /// <param name="protectRegexString">a regular expression which captures the data to be encrypted in a named group called protectData</param> /// <param name="protectedRegexString">a regular expression which captures the data to be decrypted in a named group called protectedData</param> /// <param name="protectedReplaceString">a string replacement expression which captures the substitution which must be applied for transforming unencrypted tokenization into an encrypted tokenization</param> /// <param name="dataProtectionServiceProvider">a service provider configured with Data Protection API, this parameters is mutually exclusive to dataProtectionConfigureAction</param> /// <param name="dataProtectionConfigureAction">a configure action to setup the Data Protection API, this parameters is mutually exclusive to dataProtectionServiceProvider</param> /// <param name="keyNumber">a number specifying the index of the key to use</param> /// <exception cref="ArgumentException">if dataProtectionServiceProvider or dataProtectionServiceProvider is null or not well configured</exception> public DataProtectionAPIProtectConfigurationData(String protectRegexString = null, String protectedRegexString = null, String protectedReplaceString = null, IServiceProvider dataProtectionServiceProvider = null, Action<IDataProtectionBuilder> dataProtectionConfigureAction = null, int keyNumber = 1) : this(protectRegexString, protectedRegexString, protectedReplaceString, dataProtectionServiceProvider, dataProtectionConfigureAction, DataProtectionAPIProtectConfigurationKeyNumberToString(keyNumber)) { } /// <summary> /// Main constructor for DataProtectionAPIProtectConfigurationData using a purpose string /// </summary> /// <param name="protectRegexString">a regular expression which captures the data to be encrypted in a named group called protectData</param> /// <param name="protectedRegexString">a regular expression which captures the data to be decrypted in a named group called protectedData</param> /// <param name="protectedReplaceString">a string replacement expression which captures the substitution which must be applied for transforming unencrypted tokenization into an encrypted tokenization</param> /// <param name="dataProtectionServiceProvider">a service provider configured with Data Protection API, this parameters is mutually exclusive to dataProtectionConfigureAction</param> /// <param name="dataProtectionConfigureAction">a configure action to setup the Data Protection API, this parameters is mutually exclusive to dataProtectionServiceProvider</param> /// <param name="purposeString">a string used to derive the encryption key</param> /// <exception cref="ArgumentException">if dataProtectionServiceProvider or dataProtectionServiceProvider is null or not well configured</exception> public DataProtectionAPIProtectConfigurationData(String protectRegexString = null, String protectedRegexString = null, String protectedReplaceString = null, IServiceProvider dataProtectionServiceProvider = null, Action<IDataProtectionBuilder> dataProtectionConfigureAction = null, String purposeString = null) { // check that at least one parameter is not null if (dataProtectionServiceProvider == null && dataProtectionConfigureAction == null) throw new ArgumentException("Either dataProtectionServiceProvider or dataProtectionConfigureAction must not be null!"); // if dataProtectionServiceProvider is null and we pass a dataProtectionConfigureAction configure a new service provider if (dataProtectionServiceProvider == null && dataProtectionConfigureAction != null) { var services = new ServiceCollection(); dataProtectionConfigureAction(services.AddDataProtection()); dataProtectionServiceProvider = services.BuildServiceProvider(); } // check that dataProtectionServiceProvider resolves the IDataProtector var dataProtect = dataProtectionServiceProvider.GetRequiredService<IDataProtectionProvider>().CreateProtector(DataProtectionAPIProtectConfigurationStringPurpose(purposeString)); if (dataProtect == null) throw new ArgumentException("Either dataProtectionServiceProvider or dataProtectionConfigureAction must configure the DataProtection services!", dataProtectionServiceProvider == null ? nameof(dataProtectionServiceProvider) : nameof(dataProtectionConfigureAction)); // sets the abstract class base properties and calls CheckConfigurationIsValid if (!String.IsNullOrEmpty(protectRegexString)) ProtectRegex = new Regex(protectRegexString); if (!String.IsNullOrEmpty(protectedRegexString)) ProtectedRegex = new Regex(protectedRegexString); ProtectedReplaceString = protectedReplaceString; ProtectProvider = new DataProtectionAPIProtectProvider(dataProtect); CheckConfigurationIsValid(); }
-
DataProtectionAPIProtectProvider it is the implementation of the
IProtectProvider
interface using the Microsoft Data Protection API (e.g. callsIDataProtector.Unprotect
forDecrypt
,IDataProtector.Protect
forEncrypt
,IDataProtector.CreateProtector
forCreateNewProviderFromSubkey).
/// <summary>
/// The standard Microsoft DataProtectionAPI protect provider for Fededim.Extensions.Configuration.Protected, implementing the <see cref="IProtectProvider"/> interface.
/// </summary>
public class DataProtectionAPIProtectProvider : IProtectProvider
{
public IDataProtector DataProtector { get; }
/// <summary>
/// The main constructor
/// </summary>
/// <param name="dataProtector">the <see cref="IDataProtect"/> interface obtained from Data Protection API</param>
public DataProtectionAPIProtectProvider(IDataProtector dataProtector)
{
DataProtector = dataProtector;
}
/// <summary>
/// This methods create a new <see cref="IProtectProvider"/> for supporting per configuration value encryption subkey (e.g. "subpurposes")
/// </summary>
/// <param name="subkey">the per configuration value encryption subkey</param>
/// <returns>a derived <see cref="IProtectProvider"/> based on the <see cref="subkey"/> parameter</returns>
public IProtectProvider CreateNewProviderFromSubkey(String subkey)
{
return new DataProtectionAPIProtectProvider(DataProtector.CreateProtector(subkey));
}
/// <summary>
/// This method decrypts an encrypted string
/// </summary>
/// <param name="encryptedValue">the encrypted string to be decrypted</param>
/// <returns>the decrypted string</returns>
public String Decrypt(String encryptedValue)
{
return DataProtector.Unprotect(encryptedValue);
}
/// <summary>
/// This method encrypts a plain-text string
/// </summary>
/// <param name="plainTextValue">the plain-text string to be encrypted</param>
/// <returns>the encrypted string</returns>
public String Encrypt(String plainTextValue)
{
return DataProtector.Protect(plainTextValue);
}
}
Points of Interest
I think that the idea of specifying the custom tag through a regex is very witty because it gives every user the flexibility they need to customize the tokenization tag.
It's strange that MS did not plan a method or a property inside IConfigurationProvider
to enumerate all possible keys of a provider,
probably (and hopefully), it will be added in future releases so I could avoid using the non efficient recursive GetChildKeys
method to enumerate all keys.
If you wonder whether you can use this package in your project and it will still be working in the future, I can underline that the only critical point is the enumeration of all configuration keys which is now done by using the GetChildKeys
method. Even if its implementation could be changed by Microsoft, it will always provide what its name states, e.g., the child keys of a configuration key. And even in the remote case that the GetChildKeys
method will be removed from the IConfigurationProvider
interface, you can always access all configuration keys by casting the interface to the ConfigurationProvider
base class and access the protected Data
property through reflection (please see the comment in CreateProtectedConfigurationProvider
method, basically all configuration providers derive from this class), so I am quite confident that the package will be working for many years.
In addition since version 1.0.12 the package allows pluggable encryption/decryption in case you want for whatever reason (alleged Microsoft hidden backdoors, open source product aficionado, etc.) not to use the default provider Fededim.Extensions.Configuration.Protected.DataProtectionAPI
based on Microsoft Data Protection API.
Furthermore since version 1.0.16 of Fededim.Extensions.Configuration.Protected
I implemented the safe casting of the IConfigurationProvider
being decrypted to the ConfigurationProvider
base class in order to make child keys enumeration process unbelievably fast. On top of that I have recently added an xUnit
test project in order to improve the reliability and the software quality of these two packages: on my personal laptop (Intel I9-13900K based) I have successfully tested it with 2*100000 random keys inside five tests cases, each one dealing with the framework provided ConfigurationProviders (CommandLineConfigurationProvider, EnvironmentProtectedConfigurationBuilder
(in order to decrypt 250k encrypted values this step took around 5 seconds in .Net462 and around 3 seconds in net6.0 which is faster) and checking that every decrypted key was equal to the plaintext one. Moreover all the whole set of five test cases was repeated for 1000 iterations (Test Explorer Run Until Failure, unluckily it is not available for the whole suite of tests, I had to do it separately), both for net462 (total runtime 705 minutes) and net6.0 (total runtime 424 minutes) without raising any error as you can see in the pictures below.
Net462 Endurance Test
Net6.0 Endurance Test
Last but not the least, I want to remind everyone that the protected
member access modifier does exist! I have seen it being used rarely as compared to private
, sometimes even inside the .NET framework classes! It should be used instead as the default member access modifier with very little or almost none private
members, since it allows inheritance
which is the basis of object-oriented programming (you never know who wants to extend and customize your classes for their or other general needs).
History
- V1.0.0 (16th December, 2023)
- Initial version
- V1.0.1 (27th December, 2023)
- Added further code in implementation details sections of
ConfigurationBuilderExtensions, ProtectedConfigurationBuilder
andProtectedConfigurationProvider
- Explained better how automatic decryption of values on configuration reload works in implementation details sections
- Added further code in implementation details sections of
- V1.0.2 (30th December, 2023)
- Improved the
ConfigureDataProtection
explanation and added link to Data Protection purpose string documentation - Added point in "Points of Interest" section about missing method or property on MS
IConfigurationProvider
to enumerate all keys - Fixed some typos
- Improved the
- V1.0.3 (7th February, 2024)
- Commented 3 lines of unneeded code in
CreateProtectedConfigurationProvider
method ofProtectedConfigurationBuilder
- Released NuGet package version 1.0.5
- Commented 3 lines of unneeded code in
- V1.0.4 (8th February, 2024)
- Added point in Points of Interest section about the critical points of this package and its likelihood to be still working in the future
- Added point in Points of Interest section about fostering the usage of protected member access modifier
- V1.0.5 (4th May, 2024)
- Released NuGet package version 1.0.6
- Updated paragraphs and code to reflect code changes (e.g.
ConfigurationBuilderExtensions
.IDataProtect.ProtectFiles
andProtected.ConsoleTest
) - Added source code hyperlinks to appsetting.json, appsettings.development.json and appsettings.xml files
- V1.0.6 (8th May, 2024)
- Released NuGet package version 1.0.7
- Updated paragraphs and code to reflect code changes (e.g.
ConfigurationBuilderExtensions
,ProtectedConfigurationProvider. RegisterReloadCallback
andProtected.ConsoleTest
)
- V1.0.7 (31th May, 2024)
- Released NuGet package version 1.0.10
- Updated paragraphs and code to reflect code changes (e.g.
ProtectFiles, ProtectConfigurationValue
andWithProtectedConfigurationOptions
insideConfigurationBuilderExtensions
,ProtectedConfigurationProvider.DecryptChildKeys
,ProtectedConfigurationData
constructors,ProtectedConfigurationBuilder
constructors and regex strings, added a new file ProtectFileProcessors.cs withIFileProtectProcessor
,FilesProtectOptions
,RawFileProtectProcessor
,JsonFileProtectProcessor
,XmlFileProtectProcessor
)
- V1.0.8 (18th June, 2024)
- Released NuGet package version 1.0.13 and extracted Data Protection API encryption/decryption code into a new provider package called Fededim.Extensions.Configuration.Protected.DataProtectionAPI
- Updated different paragraphs and code to reflect code changes (too many to quote them)
- V1.0.9 (26th June, 2024)
- Released Fededim.Extensions.Configuration.Protected NuGet package version 1.0.14 together with Fededim.Extensions.Configuration.Protected.DataProtectionAPI version 1.0.2
- Updated different paragraphs and code to reflect code changes
- Added
xUnit
test project to improve reliability and software quality
- V1.0.10 (28th June, 2024)
- Released Fededim.Extensions.Configuration.Protected NuGet package version 1.0.16 together with Fededim.Extensions.Configuration.Protected.DataProtectionAPI version 1.0.4
- Made child keys enumeration process unbelievably fast (if the provider is derived from
ConfigurationProvider
like allMicrosoft
providedConfigurationProviders
, otherwise the old method is used) - Improved code and speed of
xUnit
test project - Bugfix on
ProtectEnvironmentVariables
(target environment was not passed)
- V1.0.11 (3rd July, 2024)
- Released Fededim.Extensions.Configuration.Protected NuGet package version 1.0.17 together with Fededim.Extensions.Configuration.Protected.DataProtectionAPI version 1.0.5
- Made
ProtectedConfigurationProvider.ProviderData
property safer using as instead of a direct cast - Added virtual to various methods in order to allow extensibility
- Updated code to reflect code changes