Introduction

ProtectedJson is an improved JSON configuration provider which allows partial or full encryption of configuration values stored in appsettings.json files and fully integrated in the ASP.NET Core architecture. Basically, it implements a custom ConfigurationSource and a custom ConfigurationProvider which decrypts all the encrypted data enclosed in a custom tokenization tag inside the JSON values using ASP.NET Core Data Protection API.

Note: This package is obsolete and has been replaced by the more versatile Fededim.Extensions.Configuration.Protected.

Background

ASP.NET Configuration is the standard .NET Core way of storing application configuration data through hierarchical key-value pairs inside a variety of configuration sources (usually JSON files, but also environment variables, key vaults, database tables or any custom provider you would like to implement). While .NET Framework used a single source (usually, a XML file which was intrinsically more verbose), .NET Core can use multiple ordered configuration sources, which gets "merged" allowing the concept of overriding of the value of a key in a configuration source with the same one present in a subsequent configuration source. This is useful because in software development, there are usually multiple environments (Development, Integration, PRE-Production and Production) and each environment has its own custom settings (for example, API endpoints, database connection strings, different configuration variables, etc.). In .NET Core, this management is straightforward, in fact, you usually have two JSON files:

• appsettings.json: which contains the configuration parameters common to all environments.
• appsettings.<environment name>.json: which contains the configuration parameters specific to the particular environment.

ASP.NET Core apps usually configure and launch a host. The host is responsible for app startup, configuring dependency injection and background services, configuring logging, lifetime management and obviously configuring application configuration. This is done mainly in two ways:

• Implicitly, by using one of the framework provided methods like WebApplication.CreateBuilder or Host.CreateDefaultBuilder (usually called inside the Program.cs source file) which substantially do:
  • Read and parse command line arguments
  • Retrieve environment name respectively from ASPNETCORE_ENVIRONMENT and DOTNET_ENVIRONMENT environment variable (set either in the operating system variables or passed directly in the command line with --environment argument).
  • Read and parse two JSON configuration files named appsettings.json and appsettings.<environment name>.json.
  • Read and parse the environment variables.
  • Call the delegate Action<Microsoft.Extensions.Hosting.HostBuilderContext,Microsoft.Extensions.Configuration.IConfigurationBuilder> of ConfigureAppConfiguration where you can configure the app configuraton through IConfigurationBuilder parameter
• Explicitly by instantiating the ConfigurationBuilder class and using one of the provided extensions methods:
  • AddCommandLine: to request of parsing of command line parameters (either by -- or - or /)
  • AddJsonFile: to request the parsing of a JSON file specifying whether it is mandatory or optional and whether it should be reloaded automatically whenever it changes on filesystem.
  • AddEnvironmentVariables: to request the parsing of environment variables
  • etc.

In essence, every Add<xxxx> extension method adds a ConfigurationSource to specify the source of key-value pairs (CommandLine, JSON File, Environment Variables, etc.) and an associated ConfigurationProvider used to load and parse the data from the source into the Providers list of IConfigurationRoot interface which is returned as a result of the Build method on ConfigurationBuilder class as you can see the picture below.

(Inside configuration.Providers, we have four sources: CommandLineConfigurationProvider, two ProtectedJsonConfigurationProvider for both appsettings.json and appsettings.<environment name>.json and finally EnvironmentVariableConfigurationProvider).

As I wrote earlier, the order in which the Add<xxxx> extension methods are called is important because when the IConfigurationRoot class retrieves a key value, it uses the GetConfiguration method which cycles the Providers list in a reversed order trying to return the first one which contains the queried key, thus simulating a "merge" of all configuration sources (LIFO order, Last In First Out).

ProtectedJson is fundamentally a class library which defines a configuration source called ProtectedJsonConfigurationSource which specifies the configuration file and the tokenization tag, and the associated configuration provider ProtectedJsonConfigurationProvider used to parse the JSON file and to decrypt the JSON values enclosed in the tokenization tag; moreover, it provides also standard extensions method for hooking them into IConfigurationBuilder interface (e.g., AddProtectedJsonFile).

Using the Code

You find all the source code on my Github repository, the code is based on .NET 6.0 and Visual Studio 2022. Inside the solution file, there are two projects:

• FDM.Extensions.Configuration.ProtectedJson: This is a class library which implements ProtectedJsonConfigurationSource, ProtectedJsonConfigurationProvider (and their stream corresponding ProtectedJsonStreamConfigurationProvider and ProtectedJsonStreamConfigurationSource) and the extension methods for IConfigurationBuilder interface (AddProtectedJsonFile and its overloads).
• FDM.Extensions.Configuration.ProtectedJson.ConsoleTest: This is a console project which shows how to use JsonProtector by reading and parsing two bespoke configuration files and converting them to a strongly typed class called AppSettings. The decryption happens flawlessly and automatically without almost any line of code, let's see how.

To use ProtectedJson, you have to add how many JSON files you like by using the extension method AddProtectedJsonFile of IConfigurationBuilder which takes these parameters:

• path: specifies the path and filename of the JSON file (standard parameter)
• optional: it is a boolean for specifying whether the JSON file is mandatory or optional (standard parameter)
• reloadOnChange: this is a boolean which indicates that the JSON file (and configuration) should be reloaded automatically whenever the specified file changes on disk (standard parameter).
• protectedRegexString: it is a regular expression string which specifies the tokenization tag which encloses the encrypted data; it must define a named group called protectedData. By default, this parameter assumes the value:

  public const string DefaultProtectedRegexString = "Protected:{(?<protectedData>.+?)}";

  The above regular expression essentially searches in a lazy way (so it can retrieve all the occurrences inside a JSON value) for any string matching the pattern 'Protected:{<encrypted data>}' and extracts the <encrypted data> substring storing it inside a group named protectedData. If you do not like this tokenization, you can replace it to any other one you prefer by crafting a regular expression with the constraint that it extracts the <encrypted data> substring in a group called protectedData.

• serviceProvider: This is a IServiceProvider interface needed to instance the IDataProtectionProvider of Data Protection API in order to decrypt the data. This parameter is mutually exclusive to the next one.
• dataProtectionConfigureAction: This is a Action<IDataProtectionBuilder> used to configure the Data Protection API in standard NET Core. Again, this parameter is mutually exclusive to the previous one.

The last two parameters are somewhat a drawback, because they represent a reconfiguration of another dependency injection for instantiating the IDataProtectionProvider needed to decrypt the data.

In fact, in a standard NET Core application, usually the dependency injection is configured after having read and parsed the configuration file (so all configuration sources and providers do not use DI), but in this case, I was compelled since the only way to access Data Protection API is through DI. Moreover, when configuring the dependency injection, the parsed configuration usually gets binded to a strongly typed class by using services.Configure<<strongly typed settings class>>(configuration) so it's a dog chasing its tail (for decrypting configuration you need DI, for configuring DI you need the configuration parsed in order to bound it to a strongly typed class). The only solution I came up for now is reconfiguring a second DI IServiceProvider just for the Data Protection API and use it inside ProtectedJsonConfigurationProvider. To configure the second DI IServiceProvider you have two options: you create it by yourself (by instantiating a ServiceCollection, calling AddDataProtection on it and passing it to AddProtectedJsonFile) or you let ProtectedJsonConfigurationProvider to create it by passing a dataProtectionConfigureAction parameter to AddProtectedJsonFile. In the console application, in order to avoid duplicated code, the configuration of Data Protection API is performed inside a common private method called ConfigureDataProtection whose implementation is:

private static void ConfigureDataProtection(IDataProtectionBuilder builder)
{
    builder.UseCryptographicAlgorithms(new AuthenticatedEncryptorConfiguration
    {
        EncryptionAlgorithm = EncryptionAlgorithm.AES_256_CBC,
        ValidationAlgorithm = ValidationAlgorithm.HMACSHA256,

    }).SetDefaultKeyLifetime(TimeSpan.FromDays(365*15)).PersistKeysToFileSystem
                                              (new DirectoryInfo("..\\..\\Keys"));
}

Here, I chose to use AES 256 symmetric encryption with HMAC SHA256 as digital signature function. Moreover, I ask to store all the encryption metadata (keys, iv, hash algorithm key) in an XML file inside the Keys folder of the console app (note that all these APIs are provided by default by the Data Protection API). So when you start the app for the first time, the Data Protection API creates automatically the encryption key and stores it in the Keys folder, in the following runs, it loads the key data from this XML file. This configuration however is not the best approach from the security viewpoint because the metadata are stored in plain text, if you are on Windows, you can remove the PersistKeysToFileSystem extension method and in this case, the metadata would be encrypted in turn with another key stored in a secure place inside your computer. I have instead no clue on how Data Protection API manages this in Linux.

In the two appsetting.json and appsettings.development.json files, I define standard key-value pairs in hierarchical way in order to exemplify the merging feature of ASP.NET Core Configuration and also the use of encrypted values.

If you look at the ConnectionStrings section of appsetting.json, there are three keys:

• PlainTextConnectionString: As the name states, it contains a plaintext connection string
• PartiallyEncryptedConnectionString: As the name states, it contains a mixture of plain text and multiple Protect:{<data to encrypt>} tokenization tags. On every run, these tokens get automatically encrypted and replaced with the Protected:{<encrypted data>} token after the call to the extension method IDataProtect.ProtectFiles.
• FullyEncryptedConnectionString: As the name states, it contains a single Protect:{<data to encrypt>} token spanning the whole connection string which gets totally encrypted after the first run.

If you look at Nullable section of appsetting.development.json, you can find some interesting keys:

• Int, DateTime, Double, Bool: These keys contains respectively an integer, a datetime, a double and a boolean but they are all stored as a string using a single Protect:{<data to encrypt>} tag. Hey wait, how is this possible?

  Well, chiefly, all the ConfigurationProviders convert initially any ConfigurationSource into a Dictionary<String,String> in their Load method (please see the property Data of the framework ConfigurationProvider base abstract class, the Load method also flattens all the hierarchical path to the key into a string separated by a colon, so for example Nullable->Int becomes Nullable:Int). Only later, this dictionary gets converted and binded to a strongly typed class.

  The decryption process of ProtectedJsonConfigurationProvider happens in the middle, so it's transparent for the user and moreover is available on any simple variable type (DateTime, bool, etc.). For now, the full encryption of a whole array is not supported, but you can however encrypt a single element converting the array to an array of strings (have a look at DoubleArray key).

The main code is:

public static void Main(string[] args)
{
    // define the DI services: Data Protection API
    var servicesDataProtection = new ServiceCollection();
    ConfigureDataProtection(servicesDataProtection.AddDataProtection());
    var serviceProviderDataProtection = servicesDataProtection.BuildServiceProvider();

    // retrieve IDataProtector interface for encrypting data
    var dataProtector = serviceProviderDataProtection.GetRequiredService
                        <IDataProtectionProvider>().CreateProtector
                        (ProtectedJsonConfigurationProvider.DataProtectionPurpose);

    // encrypt all Protect:{<data>} token tags of all .json files 
    // (must be done before reading the configuration)
    var encryptedFiles = dataProtector.ProtectFiles(".");

    // define the application configuration and read .json files
    var configuration = new ConfigurationBuilder()
            .AddCommandLine(args)
            .AddProtectedJsonFile("appsettings.json", ConfigureDataProtection)
            .AddProtectedJsonFile($"appsettings.{Environment.GetEnvironmentVariable
                   ("DOTNETCORE_ENVIRONMENT")}.json", ConfigureDataProtection)
            .AddEnvironmentVariables()
            .Build();

    // define other DI services: configure strongly typed AppSettings 
    // configuration class (must be done after having read the configuration)
    var services = new ServiceCollection();
    services.Configure<AppSettings>(configuration);
    var serviceProvider = services.BuildServiceProvider();

    // retrieve the strongly typed AppSettings configuration class
    var appSettings = serviceProvider.GetRequiredService
                      <IOptions<AppSettings>>().Value;
    }

The above code is quite simple and commented, if you launch it in Debug mode, put a breakpoint on the last line where the appSettings variable gets retrieved from DI, you will notice:

• the appsettings.*json files have been backed up in a .bak file and have its Protect:{<data to encrypt>} tokens being replaced with their encrypted version (e.g., Protected:{<encrypted data>})
• magically and automatically, the appSettings strongly typed class contains the decrypted values with the right data type even though the encrypted keys are always stored in JSON file as strings.

To use it, we had just to use AddProtectedJsonFile on the IConfigurationBuilder, pass the Data Protection API configuration and everything works flawlessly in a transparent way. Moreover all the decryption happens in memory and nothing is stored on disk for any reason.

Implementation Details

I explain here the main points of the implementation:

• IDataProtect.ProtectFiles is the first extension method which gets called and scans all JSON files inside the supplied directory for Protect:{<data to encrypt>} tokens, encrypts enclosed data, performs the replacement with Protected:{<encrypted data>} and saves back the file after having created an optional backup of the original file with the .bak extension. Again, if you do not like the default tokenization regular expression, you can pass your own one with the constraint that it must extracts the <dato to encrypt> substring in a group called protectData.
• The extension method AddProtectedJsonFile stores input parameters into a ProtectedJsonConfigurationSource object and passes it to the IConfigurationBuilder by calling Add method.
• ProtectedJsonConfigurationSource class derives from the standard JsonConfigurationSource and adds three properties: ProtectedRegex (after having checked that the provided regex string contains a group named protectedData), DataProtectionBuildAction and ServiceProvider. The overridden Build method returns a ProtectedJsonConfigurationProvider passing to it the ProtectedJsonConfigurationSource instance.
• ProtectedJsonConfigurationProvider is the class responsible for the transparent decryption. Essentially:
  • it sets up another dependency injection provider in the constructor (see above for the reason)

    public ProtectedJsonConfigurationProvider
         (ProtectedJsonConfigurationSource source) : base(source)
    {
        // configure data protection
        if (source.DataProtectionBuildAction != null)
        {
            var services = new ServiceCollection();
            source.DataProtectionBuildAction(services.AddDataProtection());
            source.ServiceProvider = services.BuildServiceProvider();
        }
        else if (source.ServiceProvider==null)
            throw new ArgumentNullException(nameof(source.ServiceProvider));
    
        DataProtector = source.ServiceProvider.GetRequiredService
          <IDataProtectionProvider>().CreateProtector(DataProtectionPurpose);
    }

  • it overrides the Load method calling firstly the base class (JsonConfigurationProvider) corresponding method to load and parse input JSON file into the Data property and then cycling all keys, querying and replacing the associated value for all the tokenization tags using the regex Replace method after having decrypted its protectedData group (e.g., <encrypted data>).

    public override void Load()
    {
        base.Load();
    
        var protectedSource = (ProtectedJsonConfigurationSource)Source;
    
      // decrypt needed values
      foreach (var key in Data.Keys.ToList())
      {
          if (!String.IsNullOrEmpty(Data[key]))
          Data[key] = protectedSource.ProtectedRegex.Replace(Data[key], 
               me => DataProtector.Unprotect(me.Groups["protectedData"].Value));
      }
    }

Points of Interest

I think that the idea of specifying the custom tag through a regex is very witty because it gives every user the flexibility they need to customize the tokenization tag. I have also released it as a NuGet package on NuGet.Org.

History

• V1.0 (20^th November, 2023)
  • Initial version
• V1.1 (21^st November, 2023)
  • Added IDataProtect.ProtectFile extension method
  • Replaced regular expression named group from protectionSection to protectedData
  • Improved legibility (there was a lot of basically!) and code
• V1.2 (4^th December, 2023)
  • Targeting multi frameworks: NET 6.0, NET Standard 2.0 and .NET Framework 4.6.2
  • Update NuGet package to 1.0.1