|
So that I don't have to keep duplicating my blog posts on here and over on my main blog platform, I will from now on just post a link to the article here rather than the entire content.
My main platform for posting technical articles is over here[^] if you want to stay up-to-date with my musings, ramblings and thoughts on all things technical. However, I will continue to post on here too, but only links to the main articles themselves.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
|
It's been a while since I wrote my last article, so apologies for that. I'm sure like many of you, it's been a very busy time. Since writing my last article, I've changed jobs (which I alluded to in my last aticle[^]
The last few months have been like a whirlwind. I made the tough decision to leave my last job and have been busy settling into my new job. I've been here for a couple of months now and thoroughly enjoying it. They're a software house who create a fully customisable web based Customer Relationship Management (CRM) application. The tech stack is as follows:
- Visual Studio (.NET Core, .NET Framework, Web API)
- Git workflow in conjunction with Azure DevOps for source control
- SQL Server
- TeamCity build system
- Octopus deployment system (of which they make extensive use as they deploy to their hosted customers as well as many of their on-premise customers too).
They also use Jira when planning their work requirements and for defect tracking.
They build all their own internal tools (such as tools for managing their hosted / on-premise customers). This is where I come in. My role is responsible for building these internal tools. My first project has been to build an application that is capable of synchronising customer licences with their hosted licensing system (another internal tool).
It's been challenging settling in as many of the team work remotely (either entirely or partly). So getting to know the team and form those all important relationships has not been easy. This will be the same for anyone starting a new role so is certainly not unique to my own circumstances. That said, the team have all been very welcoming and have made every effort to answer my questions when I've had them.
With the next lockdown now imminent, I will be working from home myself very soon. I've setup the small spare room (which previously was my man-cave) into an office (complete with desk and chair). If I'm going to work from home, I may as well do it properly.
So I have lots of new things to learn, and lots of new challenges ahead (and lots of future aticles to write). Hopefully it won't be so long till I write my next article (but I'm not making any promises). Until then however, stay safe.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
Throughout my 20+ year career within the software development industry, I have moved jobs for a whole variety of reasons. Often I have moved jobs for several reasons at once. There are many reasons why I have moved, and these will be different from one person to the next. This article will describe a few of the reasons that have played a part in my own personal decisions to move during those 20+ years.
To protect the identity of the companies in question, I will not mention any company names or people. That would be unprofessional and is not the point of this article. I want the article to focus on the reasons why I have felt the need to move on in my career, and is in no way intended to finger point or to pass blame or judgement.
So with that out of the way, let's get on with the list.
- Job security. Surely one of the most important factors is how confident you feel in your role within the business. This confidence (or lack thereof) may stem from a multitude of factors. Do you feel your role is somehow threatened by senior management decisions (cutbacks, downsizing etc). Do you feel the industry or company is threatened by external forces that could lead to your role / department being threatened? Maybe the company has been the target of an acquisition, and the parent company wants to make savings on their investment. Maybe the company has lost one or more key customers. Maybe the company is vulnerable to market forces that have led to a fall in customer demand. Many companies / industries have been highly affected by COVID, and have shown vulnerability to its devastating effects. All of these are perfectly valid reasons for feeling that your job is less than secure.
- Poor management. We've all seen poor management in action. Perhaps we've worked with poor managers, or managers who have made poor decisions. Maybe a manager who has not had the backs of their team and / or been coerced to make a politically expedient decision that didn't go down well with the team. I have worked with all of the previous managers, but only once have I worked at a company where the management structure was so bad that I have felt the need to leave that particular company. From the CEO, to the CTO to the senior management team, this particular company made bad decision after bad decision. This culminated in the entire development team leaving over the course of 12 months. The effects of these decisions had a catastrophic effect on the morale and wellbeing of everyone, and the development team in particular. Promising customers functionality that simply could not be delivered in the timescales that were agreed led to huge stress within those who were tasked with delivering that functionality. In the end, even financial enticements weren't enough to alleviate the pressure, and the team all eventually walked away.
There is an adage that says "People don't leave bad companies, they leave bad managers". There is certainly a lot of truth in that.
- Career progression. We all want to make sure we progress within our chosen profession. That we acquire new knowledge, learn new skills etc. What we don't want is to become stagnant. This is particularly a killer within software development, where if you're not moving forwards you're effectively going backwards. You cannot become stagnant or get complacement within this industry. I have seen many people thrown on the scapheap when their skills were no longer needed. Career progression is absolutely vital within the software development industry. You may feel comfortable in your current company using that legacy technology, but it only takes small ripple to topple you out of the boat and into turbulent waters. All of a sudden you have to try to find a new job when you lack the latest skills the industry demands. Career progression is also about ensuring your own longevity within a fast moving industry.
Future employers may not look very favourably on a software developer who has used the same technology and made little progress in their skill set. I happen to be someone who enjoys learning new skills and technologies and keeping myself abreast of current trends. If you don't have that same drive to be constantly on the cutting edge and learning what is new and emerging, then you may fall by the way side should the unexpected happen. You just don't know what's around the next corner. Always keep learning and trying out new skills. It might save your skin one day.
There are many reasons why I or anyone may decide to move on from a particular company. Ultimately the decision to do so is personal, and there are no wrong or right reasons for doing so. We all need to do what we feel is right at the time. I have never made such a decision lightly, and always reflect on the pros and cons of such a decision, as well as relying on gut instinct. Moving to a new company can breathe new life into your career and give you the opportunity to face new challenges, meet new people, learn new ways of solving problems and new ways of creating software applications. As long as you feel you are moving for the right reasons, then that's what really matters.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
I recently began using Octopus to deploy our ASP.NET Core line-of-business web application. Previously I had configured Team Foundation Services (TFS) to do this. The web application was already being built using TFS as part of our Continuous Integration (CI) pipeline, so I had configured TFS to also take care of the Continuous Deployments (CD) too.
I had three deployments configured. One for Development (which was triggered automatically by the build) and one each for our Staging and Production environments. The former deployment was to an on-premise server and was to give the developers an environment where they could test their changes. This Development environment would always have the latest changes as it was triggered by the build server as part of the CI / CD pipeline. The Staging and Production environments were both Azure web sites and would be triggered manually. We didn't want un-tested changes going into our Staging or Production environments.
In the beginning, the Staging and Production deployments were configured under TFS. This all worked perfectly well until Microsoft updated their Powershell scripts to use the latest RM (Resource Manager) versions. As the TFS deployments to Azure environments use Powershell scripting under the covers, this had the effect of breaking our Azure deployments to our Staging and Production environments. At this point, we could no longer deploy to the Staging or Production environments.
I initially investigated fixing the underlying Powershell scripts to use the updated RM versions, but this involved re-writing some quite complex scripts, and I wasn't comfortable doing this. I then looked to see if Microsoft has provided any updates for this script but there didn't appear to be any. As a temporary workaround I decided to write some FTP scripts to do get the job done. I used WinSCP to deploy to our Staging and Production environments.
I wrote two WinSCP scripts. One for deploying to Staging and one for deploying to Production. They both followed the template below. The heavylifting of the deployment is the synchronize command. This effectively checks for any changes between the local and remote servers and updates to the latest as necessary.
option batch abort
option confirm off
option transfer binary
open ftp:
cd /site/wwwroot
synchronize remote "C:\Builds\TFS\MyWebApp\Latest" /site/wwwroot
close
exit This worked well enough but wasn't bulletproof. It would sometimes fail when unable to copy one of the assemblies. The assembly was being cached by the Azure web server, and so the deployment would fail. I got round this problem by writing a Powershell script to stop / start the Azure web hosting.
The deployment scripts performed the following tasks.
- Stop the Azure web site (Powershell script)
- Deploy the latest changes (using WinSCP)
- Start the Azure website (Powershell script)
I have previously written an article on how to stop / start an Azure web site here.
So although all these scripts worked and deployed to the Azure environments successfully, having to write and maintain these scripts seemed a bit klunky. I wanted something more robust for our CD pipeline than a bunch of hand written scripts.
This is where Octopus comes in. I'd very briefly looked at Octopus at a previous company I worked for. I decided I would use this for our deployments to our Azure environments. I would still keep the on-premise CD pipeline for Development but use Octopus to deploy to our Azure environments. I created an account with Octopus and began configuring our deployments. It took me a morning to get everything working. I won't go through a step-by-step guide as these can already be found within the Octopus documentation on their website (and as every deployment configuration is unique it probably wouldn't be a very useful exercise anyway).
I was very impressed with just how easy and flexible it was to configure our deployments. The documentation was very helpful and all the options and settings all seemed intuitive. If you're familiar other CD IDEs then Octopus will not be difficult to pick up.
I have a script that is part of our CI pipeline that creates a nuget package from our published artifacts and pushes this nuget package to our Octopus environment. I downloaded and installed the Octopus tools onto our build server which then allows me to invoke the octo.exe command.
Here are the two commands the CI pipeline invokes for creating the necessary nuget package and then pushing this to our Octopus environment.
octo.exe pack --id MyWebApp --version %1 --basePath "C:\Builds\TFS\MyWebApp\Latest\websharelatest" --outFolder outputfolder
octo.exe push --package=outputfolder\MyWebApp.%1.nupkg --overwrite-mode=OverwriteExisting --server=https: When we're ready to deploy to our Azure environment(s), we simply create a new release within Octopus and deploy the latest nuget package which our CI pipeline has already made available for us.
All in all, moving to Octopus has totally streamlined our CD pipeline. It is far more robust and uses a first class technology now instead of the myriad klunky scripts it used previously. We haven't had a single issue since switching to Octopus. If you really want to take your deployments to the next level, then I'd highly recommend looking into Octopus.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
There seem to be plenty of articles relating to the creation of a DevOps process, or the creation of a Continuous Integration and / or Continuous Delivery (CI / CD) pipeline. But there seem to be very few that relate to what makes for a good DevOps process. Or in other words, what makes for a good CI / CD pipeline? If you're creating a DevOps process, what makes for a good one? What makes for a good CI / CD pipeline? In this article I will try to describe what makes for a good DevOps process, and what doesn't.
Let's start with a brief summary for those not familiar with the concept of DevOps. In a nutshell, DevOps is a relatively new concept. It's aim is to align and integrate the following core IT disciplines.
- Software Development
- Quality Assurance
- IT Operations
Through a set of processes, tooling and cultural changes, it helps remove / reduce barriers between these disciplines to enhance the quality and reliability of software products, and to accelerate the delivery of those software products.
What makes a good DevOps tool?
Agile tools are very often mistaken for DevOps tools due to their close relationship with the iterative and incremental framework of software development. While a good DevOps tool is inherently Agile, that doesn't necessarily imply that all Agile tools will be fit for purpose within the confines of a DevOps environment.
In reality, there are probably very few Agile development tools that are integrated or flexible enough to provide the full scope of features that DevOps requires. And all within a collaborative environment.
The primary goal of DevOps is to bring together the previously isolated disciplines of software development (whose goal is to create features within the software products), QA (whose goal is to ensure that those features meet customer needs and are fit for purpose) and Operations (whose goal is to main the infrastructure that these disciplines rely upon).
A good DevOps tool therefore provides these disciplines with the necessary insights they require. It provides insights not just for their own disciplines, but so that each discipline can gain insights into the other disciplines. It therefore brings them together by forming part of the overarching development lifecycle.
What makes for a good CI / CD pipeline?
A good DevOps process therefore is one that is closely aligned with Agile (from where the concept originated). What makes for a good CI and / or CD pipeline? I won't create an exhaustive list (that could form the basis for a book), but instead give a few broad tips when building your own pipeline(s).
- Repeatable, reliable, consistent: This is the mantra for a good pipeline. It cannot be overstated how important these attributes are. These are the cornerstones for a good pipeline.
Repeatable - You can run your pipeline as often as necessary and the exact same steps will be executed in the exact same order each and every time.
Reliable - Your pipeline should be robust and should not be reliant upon temporal resources. Build your pipelines using the same engineering rigour and discipline that you would apply to your code. A good pipeline should be able to withstand whatever changes the software developers commit to it during the development of a feature.
Consistent - A good pipeline should be predictable. A pipeline should create only the artifacts that it has been designed to create. It should only run the unit tests that have been designed to be executed. It should only deploy to the endpoints that have been configured. A good pipeline should be consistent and predictable to the point of being boring. In fact, with regards to pipelines, the more boring the better.
- Automation: As far as possible, the entire pipeline should be automated with no manual intervention. Other than manually triggering certain events (such as your deployment into production), the steps involved in your pipelines should all be capable of running without the need for human intervention.
Summary
This has hopefully give you an overview of what makes for a good DevOps process, and what makes for a good CI / CD pipeline. If you create your processes and pipelines with these in mind, you should be on the right tracks.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
Whilst developing the journey logging functionality for our Xamarin mobile app, we needed to give the user the ability to start and stop the app from recording their journey. The functionality was designed to give drivers the ability to record their business journeys so that they could reclaim business mileage expenses. So when they set off on their business journey, they would open the app and set it to begin recording their journey. When they arrived at their destination they would stop the app from recording their journey. For anyone who's ever used a running / cycling app such as Garmin or Strava, the concept is identical.
We already had the functionality for initialising and recording the driver's journey. These were RESTful services that send the data to our backend for storage and processing. The problem came when we tried to plug these RESTful services into the Xamarin app. The Xamarin service methods we wanted to plug into were not async. They were overridden service methods, and we had no control over these whatsoever. Our RESTful services for initialising and recording the journey were both async.
So the question was how to execute async code from inside non async code. Turns out the solution is pretty straight forward. The .NET Framework has already solved this problem, as there are many use cases where async code will need to be run from inside non async code. As developers we won't always have full control over the entire pipeline of a process. Sometimes we will be constrained by the underlying environment (in our case Xamarin service methods), or we may be updating existing non async code with new async functionality. Either way, there will be times when we need to run async code from non async code.
Below is the OnStartCommand from the Xamarin service. We have no control over this method and cannot therefore make it async.
public override StartCommandResult OnStartCommand(Intent intent, StartCommandFlags flags, int startId)
{
} The Task object contains a method for doing exactly this. The Run() method contains several overloads to allow for the execution of tasks, including async tasks. The Task.Run() method queues a task to run on the threadpool and returns a handle to the task.
Here's the Xamarin OnStartCommand method updated to execute our RESTful services for initialising and recording a journey.
public override StartCommandResult OnStartCommand(Intent intent, StartCommandFlags flags, int startId)
{
base.OnStartCommand(intent, flags, startId);
Task.Run(async () => { await this.InitialiseJourney(); });
Task.Run(async () => { await this.StartJourneyTracking(); });
return StartCommandResult.NotSticky;
} This is just one simple example of using the Task.Run() method. It can be used in many different ways to solve many similar problems. By giving the developer the ability to queue and execute tasks, this is a very powerful method that is worth understanding. If you're writing async code you should definitely check out this method.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
To get the most out of this article, you will need a good understanding of creating Expressions with the .NET Framework. If you don't already, then check out this article[^] before going any further.
During the development of our current ASP.NET Core web application, I had a particular set of forms where the client needed to dynamically query one particular dataset. The data being returned related to permission data. The data could be queried in a number of different ways e.g. the permissions for a particular user or the permissions for a particular service. Although the same data was being queried and returned in both cases, they would need to be implemented as two completely separate GET Requests.
All our queries are RESTful GET commands which invoke our ASP.NET Web API backend services. Each new query would involve creating a new controller, service-layer code, data-layer code etc. Much of this code would be very similar as it was effectively querying the same data and returning the same data structures (models).
This got me thinking. Instead of implementing each of these queries separately, that I instead create a dynamically queryable service instead. The client application passes a query to the service. The service executes this query against the data and returns the result back to the client. This would give the client the flexibility to query the data in any way as required.
I wasn't even sure if this was possible. After much investigation, I came across some posts on Stackoverflow confirming that it was indeed possible. The client application would create an Expression tree. This would be serialised and sent to the ASP.NET Web API service where it would be de-serialised and executed.
The first problem would be serialising the Expression. It turns out that .NET Expression trees cannot be serialised / de-serialised. An Expression is not based on a static structure in the same way as a class, and therefore does not contain any definition for its structure. A class can be serialised because it contains type and structure meta data that can used by the serialiser. An Expression tree contains none of this meta data.
It turns out that there is a nuget package called Remote.Linq[^] that is able to handle the serialisation of Expressions. This handles all the serialisation / de-serialisation allowing your Expression to be passed to your backend service from the client application.
The first step is to add two package references to your project in Visual Studio.
1. Remote.Linq
2. Remote.Linq.Newtonsoft.Json
These will add the necessary extension methods and functionality needed to serialise / de-serialise your Expression trees.
You may need to create some helper functions similar to the ones below. These encapsulate the logic involved with serialising / de-serialising your Expression trees.
public Remote.Linq.Expressions.Expression DeserialiseRemoteExpression<TExpression>(string json) where TExpression : Remote.Linq.Expressions.Expression
{
JsonSerializerSettings serializerSettings = new JsonSerializerSettings().ConfigureRemoteLinq();
Remote.Linq.Expressions.Expression result = JsonConvert.DeserializeObject<TExpression>(json, serializerSettings);
return result;
}
public string SerialiseRemoteExpression<TExpression>(TExpression expression) where TExpression : Remote.Linq.Expressions.Expression
{
JsonSerializerSettings serializerSettings = new JsonSerializerSettings().ConfigureRemoteLinq();
string json = JsonConvert.SerializeObject(expression, serializerSettings);
return json;
}
public System.Linq.Expressions.Expression<Func<T, TResult>> ToLinqExpression<T, TResult>(Remote.Linq.Expressions.LambdaExpression expression)
{
var exp = expression.ToLinqExpression();
var lambdaExpression = System.Linq.Expressions.Expression.Lambda<Func<T, TResult>>(exp.Body, exp.Parameters);
return lambdaExpression;
} With those created, the first thing you will need to do is create your Expression and serialise it.
The examples I will use all relate to the scenario I described at the beginning of the article i.e. the ability to dynamically query permissions data.
Here's a simple Expression that returns a list of permissions for the specified permission ID (in reality there would only ever be one permission returned for any given permission ID but for the purposes of this example let's assume that one or more permissions will be returned by the Expression).
const int permissionid = 1;
Expression<Func<PermissionEntities, List<PermissionEntity>>> expr1 = m => m.Permissions.FindAll(q => q.PermissionId == permissionid); Next we need to convert the Expression into a Remote.Linq expression and serialise it.
var serialised = SerializerManager().SerialiseRemoteExpression(expr1.ToRemoteLinqExpression()); The extension method ToRemoteLinqExpression() is provided by Remote.Linq and converts a .NET Expression into a Remote.Linq expression.
With our Expression now serialised into a string, we can pass it into a function to execute against our permission data. The function will need to perform the following actions.
1. De-serialise the Remote.Linq expression
2. Convert the Remote.Linq Expression into a .NET Expression
3. Invoke and execute the Expression against permissions data
Here's an example of a function that accepts a serialised Remote.Linq expression and executes it against a permissions dataset.
public PermissionModels GetPermissionsDynamic(string payload)
{
if (string.IsNullOrEmpty(payload)) return null;
PermissionModels result = new PermissionModels();
Remote.Linq.Expressions.LambdaExpression expression =
SerializerManager().DeserialiseRemoteExpression<Remote.Linq.Expressions.LambdaExpression>(payload) as LambdaExpression;
var localexpression = SerializerManager().ToLinqExpression<PermissionEntities, List<PermissionEntity>>(expression);
PermissionEntities permissions = this.Data.GetPermissions();
var compiled = localexpression.Compile();
var matches = compiled.Invoke(permissions);
if (matches == null || !matches.Any()) return result;
return matches
} Putting all the pieces together, here's a simple unit test that demonstrates how to create an Expression and pass this to the above function to execute against actual data.
[TestMethod]
public void GetPermissionsDynamicTests()
{
const int permissionid = 1;
PermissionsService service = new PermissionsService();
Expression<Func<PermissionEntities, List<PermissionEntity>>> expr1 = m => m.Permissions.FindAll(q => q.PermissionId == permissionid);
var serialised = SerializerManager().SerialiseRemoteExpression(expr1.ToRemoteLinqExpression());
var results = service.GetPermissionsDynamic(serialised);
Assert.IsNotNull(results);
Assert.IsNotNull(results.Permissions);
Assert.IsTrue(results.Permissions.Any());
Assert.IsNotNull(results.Permissions.Find(q => q.PermissionId == permissionid));
} To complete this we would need to write a controller method that invoked the function GetPermissionsDynamic(). It should be noted that although we are creating dynamic queries over HTTP, they will need to be implemented as POST rather than GET. The reason for this (as I found out) is because a GET querystring is limited in length. A serialised Expression will almost certainly break that limit. Therefore place the serialised Expression in the POST body of your Request. It is also more secure to pass your Expressions this way as they are less visible to prying eyes. You may want to consider encoding / encrypting the Expressions you pass from your client to your service for added security.
I wouldn't use this pattern for every query. I would still create a single GET query for each Request for the majority of the queries I implement. However, when you have several queries all acting on the same data and returning the same data structures (models), then this pattern allows you to simply and easily implement those as dynamic queries. Like all patterns, it can solve a very specific problem if used as intended and its usage is clearly understood by the developer.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
I recently had a need to start / stop our Azure web application from our deployment pipeline. As part of our deployment process I have written a script that uploads the latest changes to our Azure web hosting via FTP (using the excellent WinSCP). In testing this however I was finding that sometimes not all the files were getting uploaded, in particular the application's main assembly would often fail to upload as it was in use. I therfore decided that I would stop / start the web application as part of the deployment.
- stop the web application
- upload the latest changes via FTP
- start the web application
After some research I discovered the following Powershell (PS) cmdlets that I would require.
Start-AzWebApp
Stop-AzWebApp
Restart-AzWebApp (not needed for this particular requirement but might be useful elsewhere in the deployment pipeline).
So I began implementing a PS script that would invoke the aforementioned cmdlets to stop / start our web application. However, the script kept failing with the following error message.
Quote: No subscription found in the context. Please ensure that the credentials you provided are authorized to access the Azure subscription then run Connect-AzAccount to login. Okay, so I need to programatically login to our Azure account before I can invoke the methods. This seemed reasonable, as I wouldn't want just anyone interacting with our Azure resources. So I began investigating how to login to Azure using the Connect-AzAccount cmdlet. This is where I got a bit stuck. The parameters I needed to pass to the cmdlet I didn't have and couldn't find them anywhere. I looked all through the Azure dashboard and in particular the settings relating to the web application.
After some further investigation it seemed that the correct approach (at least as far as my particular requirements were concerned) was to create an Azure Service Principal and to use this to interact with the Azure resources from the PS script. You give the Azure Service Principal only the minimum level of privilege to accomplish the task(s) you require. In this case, I needed the Azure Service Principal to have the ability to stop / start a web application. The privileges you can assign are extremely granular and you can choose the exact level of privilege you need to accomplish your task(s).
In order to create the Azure Service Principal I followed the instructions in this guide[^].
This was simple and I had the Azure Service Principal created in no time. As part of the configuration of the App Registration step, you will also need to create a new Client Secret (Home -> {Azure Active Directory} -> App Registrations -> Certificates & secrets). I called the App Registration "PowershellAutomation" and named the Client Secret "PowershellAutomationSecret". All very self explanatory should someone else have to maintain this later on. Be sure to make a note of the value for your generated Client Secret as you will need to use it in your PS script for logging into your Azure account.
Then all you need to do is plug all the values into your PS script and you're done. You can now manage your Azure resources directly from your PS scripts.
Here's my PS script for stopping the Azure web application.
cls
$ResourceGroupName = "MyWebAppGroup"
$Name = "MyWebApp"
"Stopping web application " + $Name
"ResourceGroupName: " + $ResourceGroupName
"Name: " + $Name
$SubscriptionId = "xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"
"SubscriptionId is: " + $SubscriptionId
# from the Azure AD app registration
$clientSecret = "xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"
$azureAplicationId = "xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"
$azureTenantId= "xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"
$azurePassword = ConvertTo-SecureString $clientSecret -AsPlainText -Force
$psCred = New-Object System.Management.Automation.PSCredential($azureAplicationId , $azurePassword)
# login to Azure and set the current context
Connect-AzAccount -Credential $psCred -TenantId $azureTenantId -ServicePrincipal
Set-AzContext -Subscription $SubscriptionId
Stop-AzWebApp -ResourceGroupName $ResourceGroupName -Name $Name
"Finished" I am now able to stop / start the web application from our deployment pipeline using Powershell. This means I am still able to automate the entire deployment pipeline without it needing any manual input from me. So once we are in a position to deploy the web application to our STAGING or PRODUCTION environments, we can accomplish this entirely from our TFS deployment pipeline.
Now that I have managed to programatically login to our Azure account to accomplish these tasks, I am able to use the same process to carry out any number of other tasks where I want to automate the interaction of our Azure resources from a Powershell script. This gives me a whole new playing field of ideas where our build and deployment pipelines are concerned, or indeed where any of our Azure resources are concerned and could benefit from PS automation.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
modified 16-Mar-20 4:12am.
|
|
|
|
|
As part of my research into technologies to support the new web application that I am helping to build, I have been looking into Azure Cognitive Search (ACS). I came across this technology by accident while reading through an issue of Code magazine[^]. In that article, the author explains how the technology can be used to perform powerful queries across your data, even leveraging fuzzy logic and AI capabilities.
Quote: Azure Cognitive Search is the only cloud search service with built-in AI capabilities that enrich all types of information to easily identify and explore relevant content at scale. The article then goes on to give a step-by-step guide as to how to setup and configure a simple instance of ACS. This got me thinking how I could implement this in our own applications where we regularly mine large data sets using disparate ad hoc queries. I won't go into a detailed description of setting up or configuring ACS as there are already plenty of articles online that do this including this one[^]. In particular look at the 10-minute quickstart articles on that page.
The basic premise is to create a new instance of ACS, and as part of the configuration you connect to your data. Once you have connected to your data, you can then create Indexes. These are where the magic happens. By creating an Index onto your data, this allows you to query your data using the powerful capabilities of ACS. You can add AI capabilities if you so wish by enriching your data as part of the configuration.
For example, if your data includes images of vehicles, you can enrich your data to be able to search on the vehicle's registration number. ACS will use its pattern recognition to locate the vehicle registrations from within the images. You can then search through the image data for vehicles that match specified vehicle registrations. This is just one simple example of how you can enrich your data and leverage the powerful AI capabilities of ACS.
You are able to write simple queries using SQL-like syntax including the following commands[^]
search=
searchMode=
searchFields=
queryType=
$skip=
$top= Using the power of OData expression syntax, you can then filter and order the results of your data using the following commands[^]
$filter=
$orderby=
$select= If you need fuzzy logic search, you can implement Lucene query syntax very easily. This gives you the ability to search for data where the user may have mis-typed a name or email address for example. If the user meant to type "Fred" as the search term, but accidentally typed "Frod" then using Lucene fuzzy logic you are able to return all such close matches (and you can also define how close a close match is too). See this article[^] for more details.
You can implement your queries using the REST API[^] or the Azure Cognitive Search .NET library[^]
However you decide to implement ACS in your application, it gives you a powerful, yet simple view onto your data allowing you to return everything from a simple SQL-like query to full blown fuzzy logic / pattern matching / AI capabilities. All from the same, consistent APIs.
I haven't even scratched the surface of this technogy. My intention was merely to pique your interest and hopefully get other developers to explore ACS for their own applications. I'll keep exploring this fantastic technology and hopefully write future articles on how I have used it within the applications I am developing. In the meantime, if you haven't already done so, I can't recommend this technology highly enough.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
As part of our build process, I needed to version the assembly with the latest build number. This is the first step in the build pipeline. Initially, I investigated doing this using the dotnet command as below.
dotnet build MyProject.csproj --configuration Release /p:Version=%1 The %1 parameter is the latest build number and is passed into the script via a build step. This command will build the project using the arguments that have been specified and create the build articles in the bin folder. The built assembly that the command has created in the bin folder will correctly have the version number set as per the command. So if %1 has been set to 1.0.0.0 then right-clicking on the assembly (or EXE) in the bin folder will show a version number of 1.0.0.0. All of this works exactly as it should.
The problem I was having however, is that as part of our release pipeline, I deploy the web application to our Azure hosting. To deploy to Azure using the Azure deploy task you need to create a .zip file. I create the .zip file using an MSBUILD command. I don't create the .zip file from the previous dotnet build command. The MSBUILD command uses the current project files and creates the .zip file from them. Therefore the .csproj file needs to have the correct version number before I run the MSBUILD command which then creates the .zip file.
I therefore needed to find a way to update the version number in the .csproj file before I ran the MSBUILD command. That way the version number of the assembly that gets deployed to our Azure hosting will have the correct version number.
Here is part of a .csproj file showing the version number.
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<Version>1.0.0.0</Version>
</PropertyGroup>
</Project> I began to investigate how to update the version number within the .csproj file directly. I thought I could implement a simple PowerShell (PS) script that would do this. Also, by updating the version number directly in the .csproj file (and it being the first step in the build pipeline) then every subsequent step that referenced the version number would be referencing the correct version number.
After some Googling and trial-and-error I came up with the following PS script.
cls
Write-Host "Versioning started"
"Sources directory " + $Env:BUILD_SOURCESDIRECTORY
"Build number " + $Env:BUILD_BUILDNUMBER
$csprojfilename = $Env:BUILD_SOURCESDIRECTORY+"\MyProject.csproj"
"Project file to update " + $csprojfilename
[xml]$csprojcontents = Get-Content -Path $csprojfilename;
"Current version number is" + $csprojcontents.Project.PropertyGroup.Version
$oldversionNumber = $csprojcontents.Project.PropertyGroup.Version
$csprojcontents.Project.PropertyGroup.Version = $Env:BUILD_BUILDNUMBER
$csprojcontents.Save($csprojfilename)
"Version number has been udated from " + $oldversionNumber + " to " + $Env:BUILD_BUILDNUMBER
Write-Host "Finished" The environment variables $Env:BUILD_SOURCESDIRECTORY and $Env:BUILD_BUILDNUMBER are both provided by the build process as part of the Microsoft build environment. You don't need to do or set anything to have access to these. They are provided by the build environment straight out the box for free. In fact, there are many more such variables that are also available that you may find useful in your other build scripts and processes.
The PS script fetches the latest build number and the folder path of where the source files are located on the build server. It then reads the contents of the .csproj file as an XML document. The version number is set to the latest build number (as provided by the build environment) and then closes and saves the updated .csproj file. That's it. That's all you need to do to update the version number in your .csproj file.
I initially thought that this would involve some horrible string search and replace to update the .csproj file. Thankfully, PowerShell contains native support for manipulating XML files, and it was in fact much easier than I thought. So if you need to update your .NET project's .csproj version number, feel free to use this script.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
For anyone who uses Azure Powershell scripts, it will come as no surprise that as of November last year, they deprecated all their existing Powershell functions that related to Azure. These have instead been replaced with new Azure Resource Manager (ARM) equivalent functions. These new functions supercede the previous suite of Azure Powershell functions. Microsoft began initially deprecating these as far back as 2018, but due to pressure from the development community, Microsoft delayed this for a further year to give developers more time to find workarounds and / or solutions.
I was blissfully unaware of this fact, as I do not make any direct use of their Azure Powershell functions anywhere in our development ecosystem. I only became aware of the fact that these Azure Powershell functions were deprecated when our Team Foundation Services (TFS) deployments to Azure started failing. Unknown to me, the TFS Azure Web App Deployment task uses an Azure Powershell script to push deployments to Azure. This Powershell script implemented several of the now deprecated Azure Powershell functions. Hence it began failing as the functions it was attempting to invoke no longer existed.
This left me in the rather awkward predicament of not being to deploy any of our ASP.NET Web API services to Azure. No new features could therefore be deployed for any of the apps that consumed these services. All of our web and mobile apps consume these services, so until I found a solution we would be unable to release any new features and / or maintenance fixes for the apps that consumed these services.
I began scrambling to look for a solution. I came across several articles that outlined the replacement functions and how to implement them. This was all well and good, but I didn't particularly want to have to rewrite the entire Microsoft Azure Powershell script that was responsible for deployments using the new ARM functions. I tried looking for a new version of the original Azure Powershell script, but could only find the existing version on Github. There didn't seem to be an updated version of this script anywhere.
I could either rewrite the original Azure Powershell script myself (which for obvious reasons I wasn't particularly keen on doing), or try to find another solution. After some further head scratching, I had an epiphany. Why didn't I write an FTP script that would deploy to Azure? From the Azure portal for a web app, you are able to download the publishing profile. The publishing profile (among other things) contains the FTP details required to connect to your Azure web app.
So I looked at writing a simple FTP script. I needed an FTP client that could be invoked from the command line (and therefore be scripted) as I didn't want this to be a manual task. I wanted this to work exactly as it already did i.e. to be a TFS task (and also because after each deployment to a particular endpoint I run a suite of unit tests to ensure that the code all works correctly from the Azure hosting).
I had previous experience of using WinSCP[^] so knew that this would do the job. I had only ever previously used WinSCP for uploading single files, never for uploading the hundreds of files our ASP.NET Web API services contained. After some further research I discovered that WinSCP contained a command called synchronize. This command would synchronize a local directory with a remote one, including all sub-directories. This was exactly what I needed.
I eventually wrote the following scripts that deploy our services to Azure. They are invoked from our TFS build so there is no change to our deployment process whatsoever.
Here is the batch script that is invoked from the TFS deployment task.
@echo off
cls
"C:\Program Files (x86)\WinSCP\WinSCP.com" /script=ftpDeployServices.ftp Here is the FTP script containing the WinSCP FTP commands (ftpDeployServices.ftp).
option batch abort
option confirm off
option transfer binary
open ftp:
cd /site/wwwroot
synchronize remote -delete <mylocalfolder> /site/wwwroot
close
exit The synchronize command does all the heavy lifting of checking the timestamps and uploading only those files that have changed. The -delete parameter removes any files from the remote site that don't exist on the local site. If no files have been changed since the last deployment, then no files are uploaded. Only those files that have been changed are uploaded. This results in much faster deployments as only the incrementally changed files are ever uploaded.
By hand rolling my own FTP script means that I am no longer at the mercy of any future changes to any Microsoft related Powershell script(s) or infrastructure. This means I have one less headache to worry about. In fact, the updating of the previous Azure Powershell scripts was a blessing in disguise, as my workaround is faster and reduces our dependency on Microsoft infrastructure. A good result given the severity of the impact that this update had on our deployment process.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
When I first began writing the ASP.NET WebAPI services for our vehicle telemetry tracking, it was a small project with only a few controllers. The log data that was generated was small in volume and I would query the logs with relative ease. Fast forward to today (over 3 years later), and the ASP.NET WebAPI project now contains services that are consumed by our mobile app, web apps, licence and MOT checking apps etc. Basically, the project has grown substantially since I first created it.
At the time that I first wrote the ASP.NET WebAPI services, I investigated many of the logging frameworks that were targeted for the the .NET Framework at the time. I looked at Log4Net, NLog and Elmah. They all looked feature rich and customisable, but in the end I rolled my own logging framework. My requirements at the time were quite simple as were the sizes of the log data that would be generated. So I rolled a simple logging framework that would meet my requirements.
The log data that is generated today can get very large, and querying it can be time consuming. A single build can load 10k rows of log data. We have console apps, mobile apps and web apps all running on a daily basis that also contribute to the volume of log data that is generated. It is not uncommon to have 100s of thousands of log entries created each day. Trying to diagnose a problem with so much data to query is not always a simple task. This is where structured logging comes into play.
What is structured logging?
Log data is often comprised of log messages. These log messages are unstructured strings of text. Diagnosing an error entails searching through these text strings looking for the information that we need.
Typical log messages might look something like this.
INFORMATION - 2019-12-24 - This is a log message
ERROR - 2019-12-24 - Something went wrong
INFORMATION - 2019-12-24 - This is another log message Such unstructured data makes it difficult to query for useful information. As a developer it would be really helpful to be able to query and filter your log data. For example to query log data by a specific date, customer, log entry type etc. The goal behind structured logging is to solve this problem. The aim of structured logging is to bring structure to unstructured data to allow it be usefully queried and filtered to find the information you need in a timely manner. For log data to be queried and filtered requires the data to be mapped into a format that allows this e.g. XML and JSON would be ideal candidates.
What was clear was that my initial logging framework was no longer fit for purpose. The volumes of log data being created each day were getting increasingly larger, and I was no longer able to query the data in a sensible and timely manner. That's when I came across the notion of structured logging. After much reading around and looking at other frameworks that supported this, I decided that I would update my own logging framework rather than employ one of the .NET logging frameworks (again). Although the frameworks mentioned earlier all now provide support for structured logging, SeriLog seems to be the preferred choice due it having provided support from the very beginning (whereas the others have provided support after the fact). The reason for updating my own logging framework rather than using one of the 3rd party ones was that I had already invested a lot of time and effort into my own framework, and it would probably be more straight forward to update what was already there than to start again with a new logging framework.
How have I implemented structured logging?
I have added a new column to my log table. Previously my log table consisted of a log message column that would contain text strings (as in the examples given previously). All the key information would be contained within the log message as in the following example.
INFORMATION - 2019-12-24 - User Joe has logged onto the system The user name 'Joe' is what I would be looking for in the log message. As can be seen it is part of the log message itself. With structured logging all such information is now stored in JSON (other formats are available). My log table is composed of the following columns - with the Properties column storing the structured logging data in JSON format.
LogType
INFORMATION
Created
2019-12-24 10:45
Message
User has logged onto the system
Properties
{ "username": "Joe" } N.B. one added benefit of rolling my own logging framework is that I can implement as many log types (the column LogTypes in the above example) as I need (I currently have ERROR, INFORMATION, EXCEPTION, DEBUG). I have a .config setting that allows me to switch the DEBUG log data on or off. Currently I have this set to ON for my unit test projects and OFF for production.
The biggest challenge was not so much updating my logging framework to implement structured logging, but to update all the areas of the code where I am sending output to the log (which is everywhere). I am still currently in the process of doing this.
The signature for my log message method looks like this.
protected void LogMessage(string message, LogLevelTypes loglevel, Dictionary<string, string> properties = null)
{
} The LogLevelTypes is an enum containing the values ERROR, EXCEPTION, INFORMATION and DEBUG. The structured log data is passed to my method as a Dictionary of key-value pairs. The Dictionary is then serialised before being sent to the log table.
With your log data now stored in a structured format, you are able to utilise one of the many different tools that are available for viewing structured logging data e.g. Prefix from Stackify[^]
I haven't yet got round to investigating any of the tools available for viewing structured data, but when I do I will be sure to write an article about it on here. Until then, have a Merry Christmas and a Happy New Year.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
I recently came across an issue with several of our ASP.NET WebAPI services which were consuming a third-party set of APIs. These third-party APIs were configured to disable any requests from clients that were using TLS 1.0/1.1. Unfortunately, this included our own APIs. All requests to the third-party API were returning empty responses. After some discussion with one of the developers of the third-party APIs, he suggested the issue may be related to TLS 1.2 not being supported as he had seen the issue before.
Firstly, what is TLS? Here's a definition from a Microsoft article that describes TLS best practices with regards to the .NET Framework.
Quote: The Transport Layer Security (TLS) protocol is an industry standard designed to help protect the privacy of information communicated over the Internet. TLS 1.2 is a standard that provides security improvements over previous versions. TLS 1.2 will eventually be replaced by the newest released standard TLS 1.3 which is faster and has improved security - Transport Layer Security (TLS) best practices with the .NET Framework | Microsoft Docs
I was able to run the third-party APIs from our local test environment, but not when I ran them from our staging / production environments which were hosted on Azure. I had to make several changes, including code changes to the ASP.NET WebAPI services and changes to our Azure hosting environments.
As many current servers are moving towards TLS 1.2/1.3 and removing support for TLS 1.0 /1.1, connectivity issues between newer servers and older (legacy) .NET applications are becoming more common. Installing a newer version of the .NET Framework onto your development environment is not the answer. The solution is down to the version of the .NET Framework used for compiling your project. This is what actually matters when it comes to selecting the supported TLS version during the TLS handshake.
In this article I will describe the changes I have made to our Azure hosting (where our ASP.NET WebAPIs are hosted) and the code changes which enabled TLS 1.2 support.
Upgrading our Azure hosting to support TLS 1.2
More accurately the changes I have made to our Azure hosting have removed support for earlier versions of TLS i.e. TLS 1.0/1.1. Although this change was not strictly necessary to fix the problem I was experiencing, it was appropriate in terms of tightening up the security of our ASP.NET WebAPIs and to ensure that our own APIs can only be accessed by clients that support TLS 1.2. This is quite simply achieved by opening the Azure portal and navigating to the App Service hosting. From there the TLS/SSL Settings blade can be selected.
I have set this to TLS 1.2 for both our staging and production environments. This sets the minimum TLS version. Therefore our hosting environments will no longer accept requests from earlier versions of TLS.
Code changes to support TLS 1.2
Depending on what version of .NET Framework your project uses will dictate the possible solutions available to you. If your project compiles against .NET Framework >= 4.7 then you are already good to go. Applications developed in .NET Framework 4.7 or greater automatically default to whatever the operating system they run on considers safe (which currently is TLS 1.2 and will later include TLS 1.3).
If your application has been developed in a version of the .NET Framework prior to 4.7 then you have two options.
- Recompile your application using .NET Framework 4.7 or greater
- If recompiling your application is not something you can do then you can update your .config file by adding the following.
<configuration>
<runtime>
<AppContextSwitchOverrides value="Switch.System.Net.DontEnableSystemDefaultTlsVersions=false"/>
</runtime>
</configuration> Also make sure you have the following set in your .config file.
<system.web>
<compilation targetFramework="x.y.z" />
<httpRuntime targetFramework="x.y.z" /> <-- this is the important one!
</system.web> It is obviously preferable if x.y.x are the same i.e. that the application is compiled against and runs against the same .NET Framework version. So in the code sample x.y.z could be 4.6.1 or some other version of .NET Framework prior to 4.7.
In the cases where recompilation is not an option and you need to update your .config file instead (as described above), this should be viewed as a temporary workaround. The preferred (and best practice) solution is to recompile your application as soon as possible.
Microsoft have put together a useful document describing the best practices relating to TLS 1.2. The advice in this document should be carefully read and understood in order to fully secure your application.
So if your application doesn't already support TLS 1.2 then it's a good idea to put aside some time to make sure it does. Ensuring your application is up-to-date in terms of security can only be a good thing.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
modified 12-Dec-19 5:22am.
|
|
|
|
|
The latest version of our app is nearly ready to drop into the app stores. It's been a challenging project with more than a few hurdles along the way. The project remit initially was to add the ability for our drivers (the user's of the app) to track their journeys to allow them to make mileage expeneses claims. By logging their journeys, drivers would have evidence of their claimed mileages. At the outset this seemed like a really useful feature that should be straight-forward. Tracking a user's movements using an app is certainly nothing new, so we certainly didn't foresee the issues we would later encounter.
The current app is developed using Xamarin Forms. All functionality and business logic is supplied to the app by way of ASP.NET WebAPI RESTful services all of which are hosted on Azure. All of the services are processed by an Azure Service Bus to ensure we can scale the app and to add resiliency. The majority of the code within the current app is shared code (Xamarin Forms apps allows for the code that is the same across the different platforms to be contained in one project, whilst the Android and iOS specific code are contained in separate projects respecively).
We began the project with the aim of keeping as much of the journey logging code in the shared project to keep the platform specific code to an absolute minimum. This ambition was quickly forgotten when we got down to the details of the project. We soon realised that it wasn't possible to fully realise the journey logging functionality without writing a lot of platform specific code as so much of it was tied to the specific hardware on the devices. Although the cross-platform geolocator service we used ran on both platforms (courtesy of James Montemagno), we wanted the ability to run the tracking service as a background process.
Current devices have very strict constraints as to how they will execute long running processes (and quite rightly too). We needed to run the journey logging in the background, as modern platforms just don't support executing long running processes in the foreground. These constraints were different across the different platforms. Android and iOS handle long running processes differently, and their constraints and solutions are unsurprisingly different too.
We next wanted to enable local push notifications to keep the driver informed that the tracking service was still recording in the background. Especially if the user brought another app to the foreground, made a phone call, or in some way forced our app to the background. Local push notifications are handled completely differently on the different platforms, leading to further platform specific code. Implementing the journey logging service as a background process, and implementing local push notifications all entailed having to write vast swathes of platform specific code.
All of these platform specific deviations brought up brand new problems, and showed the many discrepencies between Android and iOS. Although Xamarin Forms does a magnificent job of hiding as much of these deviations as possible, there were many times on this project when we were fully exposed to the inner workings of the platforms and needed a deep understanding of the native APIs. iOS particularly threw up many problems. In particular, it was incredibly difficult trying to submit large journeys as a background service. On Android, it was relatively straight forward getting large uploads to run in the background. It was far more technically challenging on iOS due it's vastly more restrictive environment and permissions.
The services that support and provide all the functionality to the app are all ASP.NET WebAPI RESTful services. The services needed to support the journey logging functionality were initially thought to be straight forward. All we would need were services that would allow the journeys to be created, updated and deleted from the device. During initial testing with the app we came across several issues when trying to submit journeys from the device to our cloud hosted Azure SQL DB. Initially we ran into issues when submitting journeys from our development environment. After much head scratching and investigation I eventually pinned this down to a rule on our firewall that was set to truncate any outgoing traffic that exceeded 1MB in size. After resolving this issue, we ran into a further similar problem when we attempted to submit journeys from our staging (Azure) environment. We were getting SocketException errors. After further diagnosis we found that we could send smaller packets of data successfully. The error only appeared when attempting to send large journeys in one go. So I had to write a chunking algorithm to decompose larger journeys in multiple smaller journeys. This required making substantial changes to the underlying WebAPI service, as well as changes to the app code itself.
Another new feature of the app is the ability to create the main menu dynamically. Against each company we store a list of the menu options that will be available to them when they open the app. This gives us the ability to turn app features on and off for a company dynamically. Each time the app is launched we check the menu options dynamically at run-time. This allows us to update a driver's list of menu options without them even having to log-off or restart the app. It's all done while the app is running.
We're now in the final stages of testing the app and are hoping to have it in the stores very soon. It has been a real trial-by-fire. We encountered many problems along the way, and with much grit and determination, have managed to overcome all of them. The project has been challenging to say the least, but ultimately successful thanks to the sheer determination of the development team.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
As part of the development of our new app feature, we are adding the ability to allow users to track their journeys. They can Start / Stop the journey tracking and allow the app to record their distance, time taken etc. This is primarily to be used to allow users to support mileage claims.
A journey takes the form of a Model containing properties for storing the user, start date, end date, mileage etc. The journey also contains a list of waypoints. These are the longitude / latitude points that are generated by the user's position. A waypoint is taken every 5 seconds and added to the list of waypoints for the journey. From these waypoints we can then generate a map of their journey and display this to them.
During initial testing everything was fine as we tested the feature on smaller journeys containing a few hundred waypoints. As we began stress testing the feature, we noticed we were getting timouts as we started to exceed 800 or so waypoints. By 1000 waypoints were getting regular timeouts. We discovered the reason was due to the volume of waypoints we were posting back to our service. As the number of waypoints grew, the time taken to POST the data over our RESTful service grew too. And this was causing our timeout problem.
I investigated several options, but the cleanest and most simple was to chunk the waypoints into smaller discrete lists which we would POST. So instead of POSTing all of the waypoints in one large payload, we would instead send multiple smaller payloads.
So how do you chunk your list into a list of smaller lists? There are many ways of achieving this, and I'm sure those of you reading this article will be able to suggest your own versions of the algorithm I have used here. Firstly, instead of using a hard-coded version that only works with journey waypoint lists, I have implemented an extension method that can work with any type of list. This allows me to chunk any type of list data going forwards (I have already got a few ideas in mind of how I will reuse this extension method).
public static IEnumerable<IEnumerable<T>> GetChunk<T>(this IEnumerable<T> source, int chunksize)
{
if (chunksize <= 0 || source == null) yield return null;
var pos = 0;
while (source.Skip(pos).Any())
{
yield return source.Skip(pos).Take(chunksize);
pos += chunksize;
}
} So we can see that what is returned is a list of lists of type T. The extension method is applied to the source list (which is to be chunked into smaller lists). The parameter to the extension method is the number of items to appear in each chunked list. The implementation uses the LINQ methods of Skip() and Take() to iterate over the list. The Skip() method will ignore the first n items in the list. The Take() method will then take the next n elements from the list. These methods therefore when used in conjunction easily iterate over our list. The use of yield return helps to iterate over the list in an efficient manner by processing the next item without having to process the entire list (lazy evaluation).
In our specific case for chunking our journey waypoints, we have set the chunking value to 500. Although the problem didn't appear until at least 800 items, I wanted to keep the value to a safe, low limit just to err on the side of caution.
Here is the code from one of the unit tests I've written that exercises the chunking extension method.
[TestMethod]
public void GetChunk1000Tests()
{
const int waypointcount = 1000;
var journey = ListExtensionsTests.GetTaskTrackedJourneyForUnitTest(waypointcount);
Assert.IsNotNull(journey);
Assert.IsNotNull(journey.Waypoints);
Assert.IsNotNull(journey.Waypoints.Waypoints);
Assert.IsTrue(journey.Waypoints.Waypoints.Any());
Console.WriteLine($"ChunkCount: {ListExtensionsTests.ChunkCount}");
Console.WriteLine($"Number of waypoints: {journey.Waypoints.Waypoints.Count}");
Assert.IsTrue(journey.Waypoints.Waypoints.Count == waypointcount);
var result = journey.Waypoints.Waypoints.GetChunk(500);
Assert.IsNotNull(result);
var enumerable = result.ToList();
Console.WriteLine($"Number of chunks: {enumerable.Count()}");
int incrementalwaypointcount = 0;
foreach (var item in enumerable)
{
Console.WriteLine($"Number of chunks in list {item.Count()}");
incrementalwaypointcount += item.Count();
}
Assert.AreEqual(waypointcount, incrementalwaypointcount);
} So in summary, if you are dealing with large lists of items and need to break them down in smaller, more manageable lists, then chunking them is a simple and very effective solution. This works great in our mobile app where we are sending large lists of data to a backend service using the resource hungry environment of the smart phone where memory and processing power are in short supply. Processing numerous smaller lists is more efficient (and less error prone) than trying to process one large list. It also uses less resources (memory, CPU) to do so.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
In lambda calculus, a predicate is an expression that evaluates to either true of false. If you have written any LINQ or a SQL query you have probably written these types of expressions already. If you have written a SQL query that contains a WHERE clause for example, this is a type of predicate. If you've ever used LINQ to filter the contents of a list, this too is an example of a predicate.
Whether you realise it or not, you have probably already used predicates in your code. Whenever you have a need to filter the items in a dataset and / or list, then it is common to use predicates to do this. The notion of a predicate is widely used and understood, even if you weren't necessarily aware of them.
Within the .NET Framework the notion of a predicate is formally identified by
Predicate<T> This is a functional construct providing a convenient way of testing the truthy or falsity of a given expression relating to an instance of type T. If you're familiar with delegates then Predicate<T> is equivalent to
Func<T, bool> For example suppose we have a Car class that represents T. Each instance of T (Car) contains the properties Colour (red, green, black etc) and EngineSize (1000, 1200, 1600 cc etc).
public class Car
{
public string Colour { get; set; }
public int EngineSize { get; set; }
} Let's assume that we have a SQL query that returns a list of all the cars registered for a particular year.
var data = new DataService();
List<Car> = data.GetAllRegisteredCars(new DateTime(2019, 01, 01); The above query will return all cars registered during the year of 2019.
Suppose we want to filter that list of cars to just those that meet certain criteria e.g. those cars with an engine size of 1600cc or are blue in colour. To filter the data we would use predicates as follows.
var matches1 = cars.FindAll(p => p.EngineSize == 1600);
var matches2 = cars.FindAll(p => p.Colour == "Blue"); We could hardcode the predicates and leave them in the code as in the above examples. However, a benefit of using Predicate<T> in your code is that it gives you the ability to separate the data from the expressions used to filter it. Instead of hardcoding filters in your code, you can define these elsewhere and bring them into your code when needed.
Let's assume we have a completely separate class that defines our predicates called PredicateFilters.cs
public static class PredicateFilters
{
public static Predicate<Car> FindBlueCars = (Car p) => p => p.Colour == "Blue";
public static Predicate<Car> Find1600Cars = (Car p) => p => p.EngineSize == 1600;
} In our data code we would now write the following code to filter the cars.
var matches1 = cars.FindAll(PredicateFilters.Find1600Cars);
var matches2 = cars.FindAll(PredicateFilters.FindBlueCars); We can see even from this simple example that separating our queries from our code is straight-forward. We no longer need to pollute our code with hardcoded filters. We also have the ability to reuse those filters elsewhere. For example, we may have more than one function that needs to know which cars are blue. We write the filter once and use it everywhere we need it. If in the future it turns out that it's red cars we need instead of blue, we can change the filter in one place without having to change any of our data code.
Our filters may return a single item or may return a list of items. Alternatively, we may also want to know the number of items returned by our filter. We would probably want to do this for different types of data e.g. cars, drivers, orders etc. This is where we need to get a bit smarter with how we design our filters to allow them to work with different types of data.
Let's start by implementing an interface that defines the filters we want to execute on our data.
public interface IPredicateValue<T>
{
T GetValue(List<T> list, Predicate<T> filter);
List<T> GetValues(List<T> list, Predicate<T> filter);
int GetCount(List<T> list, Predicate<T> filter);
} Here we have defined an interface that takes a type of T. The functions will provide the following functionality.
- T GetValue(List<T> list, Predicate<T> filter) - return a single instance of T for the filter
- List<T> GetValues(List<T> list, Predicate<T> filter) - returns a list of T for the filter
- int GetCount(List<T> list, Predicate<T> filter) - returns the count of items of T that match the filter
For each type of data that we want to filter, we should implement this interface. This will provide a consistent set of methods that we can use to filter our data.
public class CarPredicate : IPredicateValue<Car>
{
public Car GetValue(List<Car> list, Predicate<Car> filter)
{
if (list == null || !list.Any() || filter == null) return null;
return list.Find(filter);
}
public List<Car> GetValues(List<Car> list, Predicate<Car> filter)
{
if (list == null || !list.Any() || filter == null) return null;
return list.FindAll(filter);
}
public int GetCount(List<Car> list, Predicate<Car> filter)
{
if (list == null || !list.Any() || filter == null) return 0;
return list.FindAll(filter).Count;
}
} We can now filter our data as follows.
var data = new DataService();
List<Car> = data.GetAllRegisteredCars(new DateTime(2019, 01, 01);
var predicatevalue = new CarPredicate();
var blueCars = predicatevalue.GetValue(data, PredicateFilters.FindBlueCars); Keeping your code and your predicates separate gives you far more flexibility, as well as giving you a single point of change should one of the expressions used to query your data need to change. You can implement filters for any / all types of data with the added benefit that it allows you to filter your data in a consistent manner.
If you want to get serious about how you filter data, then give predicates a try.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
|
I actually really enjoy writing stuff like this. I wrote an article about writing flexible RESTful services that was very similar to GraphQL (having since checked out GraphQL I can see the similarity). GraphQL was developed by the multi-billion dollar Facebook empire, my version was developed by little ol me
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
Following on from a previous article[^] I wrote as an introduction to writing asynchronous code with .NET, I want to describe a common problem I see developers making when they begin writing asynchronous code beyond the basics. A common mistake developers make when they first start writing asynchronous code using the .NET Framework, is to write blocking asynchronous code. I've seen this problem on Stackoverflow and with developers I have worked with directly (both junior and senior).
Rather than try to explain the problem, I'll give some example code that should hopefully highlight the problem. Here's an ASP.NET Web API RESTful service being invoked from a client application.
The back-end service code is taken from one of our RESTful services that returns vehicle telemetry to a client application. For the purposes of clarity, I have omitted all logging, error checking and authentication code.
public async Task<string> Get(string subscriber, string trackertype)
{
var response = await this.GetData(subscriber, trackertype);
return response;
} And here is a client that invokes the RESTful service. In this example the client is a unit test.
[TestMethod]
public async Task GetVehicleTests()
{
TrackerController controller = new TrackerController();
string subscriber = "testsubscriber";
string vehicle = "testvehicle";
var response = controller.Get(subscriber, vehicle);
Assert.IsNotNull(response);
Assert.IsNotNull(response.Result.ToString());
} The above unit test code will deadlock. Remember, that after you await a Task, when the method continues it will continue in a context.
1. The unit test calls the Get() RESTful service (within the ASP.NET Web API context).
2. The Get() method in turn calls the GetData() method.
3. The GetData() method returns an incomplete Task indicating that the Get() method has not yet completed (with the same context).
4. The Get() method awaits the Task returned by the GetData() method (the context is saved and can be re-instated later).
5. The unit test synchronously blocks on the Task returned by the Get() method which in turn blocks the context thread.
6. Eventually the Get() method will complete. This in turn completes the Task that was returned by the GetData() method.
7. The continuation for Get() is now ready to run, and it waits for the context to be available to allow it to execute in the context.
8. Deadlock. The unit test is blocking the context thread, waiting for the Get() method to complete, and GetData() is waiting for the context to be available so it can complete.
How can this situation be prevented? Simple. Don't block on Tasks.
1. Use async all the way down
2. Make (careful) use of ConfigureAwait(false)
For the first suggestion, awaitable code should always be executed asynchronously. So given the example code here, the unit test was not correctly awaiting the result from the RESTful service. The unit test code should be modified as follows.
[TestMethod]
public async Task GetVehicleTests()
{
TrackerController controller = new TrackerController();
string subscriber = "testsubscriber";
string vehicle = "testvehicle";
var response = await controller.Get(subscriber, vehicle);
Assert.IsNotNull(response);
Assert.IsNotNull(response.ToString());
} Like a handshake, whenever you have an await at one end of a service call, you should have async at the other.
The use of ConfigureAwait(false) is slightly more complicated. When an incomplete Task is awaited, the current context is captured to allow the method to be resumed when the task eventually completes e.g. after the await keyword. The context is null if invoked from a thread that is NOT the UI thread. Otherwise it returns the UI specific context depending on the specific platform you are using e.g. ASP.NET, WinForm etc). It is this constant context switching between UI thread context and worker thread context that can cause performance issues. These issues may lead to a less responsive application, especially as the amount of async code grows (due to the increased volume of context-switching). Yet this is exactly what we are trying to solve by using asynchronous code in the first place.
There are a few rules to bear in mind when using ConfigureAwait(false)
- The UI should always be updated on the UI thread i.e. you should not use ConfigureAwait(false) when the code immediately after the await updates the UI
- Each async method has its own context which means that the calling methods are not affected by ConfigureAwait()
- ConfigureAwait can return back on the original thread if the awaited task completes immediately or is already completed.
A good rule of thumb would be to separate out the context-dependent code from the context-free code. The goal is to reduce the amount of context-dependent code (which can typically include event handlers).
We can modify the Get() RESTful service as follows.
public async Task<string> Get(string subscriber, string trackertype)
{
var response = await this.GetData(subscriber, trackertype).ConfigureAwait(false);
return response;
} Deadlocks such as this arise from not fully understanding asynchronous code, and the developer ends up with code that is partly synchronous and partly asynchronous.
By following the suggestions in this article, you should see performance gains in your own code, as well as better understanding how asynchronous code works under the hood.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
Following on from a couple of my previous articles, I would like to both reinforce the ideas I laid out in them, as well as consolidate those ideas. In an article[^] from October 2017 I described a pattern I use for designing and implementing RESTful APIs, specifically with regards to implementing RESTful GET APIs. In an article[^] from July 2018 I described the principle of reducing the client surface area, and how this leads to cleaner, simpler and less complex code, particularly with regards to implementing RESTful APIs.
Where I currently work, we have a library of RESTful ASP.NET Web APIs that our web and mobile applications consume. These cover many different types of query as they are used in many different ways by the particular applications. For example the mobile app (which is aimed at fleet drivers) fetches data for the currently signed-in user, their latest mileage updates, their account manager, journeys they have made etc. The web application fetches data relating to users, roles, permissions, documents etc.
These are all GET methods that perform a variety of different queries against different data types. When designing the client API surface required for all these APIs I wanted to make them all consistent, irrespective of what data was being returned, or what query filters were being specified.
To clarify the problem a little further, I wanted to use the same client API for all data types e.g. mileage, user, company, journey etc. Further to this, I wanted the way in which the data was queried to be consistent. Example queries are listed below.
- Fetch me the mileage data for this user
- Fetch me the mileage data for this date range
- Fetch me the journey data for this date
- Fetch me the journey data for this user
- Fetch me the permissions for this user
- Fetch me the documents for this user
These are all queries that work on different data (mileage data, journey data, permissions data, documents data) and interrogate the data in different ways (by user, by date). Crucially, I wanted all of these queries to map onto a single GET API for consistency, and to reduce the complexity of the client (by reducing the client facing API to one API instead of multiple APIs). Reducing the client facing API is the principle of reducing the surface area of the client.
I finally came up with the following API design.
- I have a single controller with a GET method that accepts two parameters.
- The first parameter is a string that designates the type of query e.g. "getmileagebyuser", "getjourneybydate" etc
- The second parameter is a serialised query object that contains the values needed to query (or filter) the data e.g. the user ID, the date or whatever filters are required to satisfy the request.
- All queries must return their data as a serialised string (which the client can de-serialise back into an object).
For the purposes of clarity the code examples used here have omitted error checking, logging, authentication etc to keep the code as simple as possible. In my own library of RESTful APIs I have separated out the requests made by the mobile app from those made by the web app. I therefore have two controllers, each with a single GET method that does all the heavy lifing of fulfilling the many different query requests. I have created a different controller for each type of client so as to prevent the controllers from bloating. You can separate out the requests any way you want. If you don't have many queries in your application, then you could simply place all of these query requests in a single GET method in a single controller. That is obviously a design decision only the developer can make.
The controllers are called MobileTasksController and WebTasksController. For the purposes of this article I will focus on the latter controller only, although they both employ the same design pattern that I am about to describe.
First let's define our basic controller structure.
public class WebTasksController : BaseController
{
public WebTasksController()
{
}
public string WebGetData(string queryname, string queryterms)
{
}
} You will need to decorate the WebGetData() method for CORS to allow the clients to make requests from your GET method.
[HttpGet]
[EnableCors(origins: "*", headers: "*", methods: "*")]
public string WebGetData(string queryname, string queryterms)
{
} Enable CORS with the appropriate settings for your own particular application.
As we can see, the WebGetData() method has two parameters.
- queryname is a string that designates the type of query e.g. "getmileagebyuser", "getjourneybydate" etc
- queryterms is a serialised query object that contains the values needed to query (filter) the data e.g. the user ID, the date or whatever filters are required to satisfy the query request
Here's the class that I use for passing in the query filters.
[DataContract]
public class WebQueryTasks
{
[DataMember]
public Dictionary<string, object> QuerySearchTerms { get; set; }
public WebQueryTasks()
{
this.QuerySearchTerms = new Dictionary<string, object>();
}
} At its core it comprises a dictionary of named objects. By implementing a dictionary of objects this allows us to pass in filters for any type of data e.g. dates, ints, strings etc. We can also pass in as many filters as we need. We can therefore pass in multiple filters e.g. fetch all the journeys for a specific user for a specific date. In this example, we pass in two filters.
- The user ID
- The date
Once the query is serialised we have the following string which is then passed as the second parameter to the RESTful GET method.
{"QuerySearchTerms":{"email":"test@mycompany.co.uk"}} By implementing our queries in this way makes for very flexible code that allows us to query our data in any way we want.
var user = GetUser(emailaddress);
WebQueryTasks query = new WebQueryTasks();
query.QuerySearchTerms.Add("userid", user.Id);
query.QuerySearchTerms.Add("journeydate", Datetime.Now);
string queryterms = ManagerHelper.SerializerManager().SerializeObject(query); The WebGetData() method then needs to deserialise this object and extract the filters from within. Once we have extracted the filters we can then use them to fetch the data as required by the request.
WebQueryTasks query = ManagerHelper.SerializerManager().DeserializeObject<WebQueryTasks>(queryterms);
if (query == null || !query.QuerySearchTerms.Any())
{
throw new HttpResponseException(Request.CreateResponse(HttpStatusCode.BadRequest, new HttpError("Unable to deserialise search terms.")));
}
The core of the WebGetData() method is a switch statement that takes the queryname as its input. Then, depending on the type of query, the method will extract the necessary filters from the WebQueryTasks parameter.
The names of the queries are stored as constants but could equally be implemented an an enum if preferred. We don't want to have to hard-code the names of our queries into the method, so any approach that separates these is fine.
In the example below there are two queries. One returns company data for a specified user. The second returns company data for a specified company ID. In each case the code follows the same pattern.
- select the appropriate case statement in the switch
- extract the filters from the query
- invoke the appropriate backend service to fetch the date using the extracted filters (after firstly checking that the filter(s) are not empty)
- serialise the data and return it to the client
object temp;
string webResults;
switch (queryname.ToLower())
{
case WebTasksTypeConstants.GetCompanyByName:
webResults = this._userService.GetQuerySearchTerm("name", query);
temp = this._companiesService.Find(webResults);
break;
case WebTasksTypeConstants.GetCompanyById:
webResults = this._userService.GetQuerySearchTerm("companyid", query);
int companyId = Convert.ToInt32(webResults);
temp = this._companiesService.Find(companyId);
break;
default:
throw new HttpResponseException(Request.CreateResponse(HttpStatusCode.BadRequest, new HttpError($"Unknown query type {queryname}.")));
} We then need to serialise the results and return these to the client.
var result = ManagerHelper.SerializerManager().SerializeObject(temp);
return result; In the production version of this controller, I have implemented many more queries in the switch statement, but for clarity I have only implemented two for the purposes of this article.
Here is the full code listing.
[HttpGet]
[EnableCors(origins: "*", headers: "*", methods: "*")]
public string WebGetData(string queryname, string queryterms)
{
WebQueryTasks query = ManagerHelper.SerializerManager().DeserializeObject<WebQueryTasks>(queryterms);
if (query == null || !query.QuerySearchTerms.Any())
{
throw new HttpResponseException(Request.CreateResponse(HttpStatusCode.BadRequest, new HttpError("Unable to deserialise search terms.")));
}
object temp;
string webResults;
switch (queryname.ToLower())
{
case WebTasksTypeConstants.GetCompanyByName:
webResults = this._userService.GetQuerySearchTerm("name", query);
temp = this._companiesService.Find(webResults);
break;
case WebTasksTypeConstants.GetCompanyById:
webResults = this._userService.GetQuerySearchTerm("companyid", query);
int companyId = Convert.ToInt32(webResults);
temp = this._companiesService.Find(companyId);
break;
default:
throw new HttpResponseException(Request.CreateResponse(HttpStatusCode.BadRequest, new HttpError($"Unknown query type {queryname}.")));
}
var result = ManagerHelper.SerializerManager().SerializeObject(temp);
return result;
} Just to repeat, for the purposes of this article, the method above has had all error checking, logging, authentication etc removed for the sake of clarity.
I have implemented this pattern in all my GET APIs to great success. It is very flexible and allows me to query the data in multiple ways as neccesary. It also allows the client code to be simpler too, by reducing the client area (the client only needs to interact with a single endpoint / controller), and enforces consistency by ensuring that all queries are similar to one another (they must all pass in two parameters - the first designating the query type, the second containing the query filters).
This pattern of API design achieves all the following benefits
- Simpler server side code by producing substantially less code due to the generic nature of the pattern
- Simpler client side code by only having a single endpoint to interact with
- High degree of flexibility by allowing the APIs to filter the data any way the application requires
- Consistency by ensuring that all requests to the RESTful API are the same
I have been using this pattern in my own RESTful APIs for several years, including several production mobile apps (that are available in the stores) and line-of-buiness web apps. With the pattern in place, I can quickly and easily add new RESTful APIs. This makes adding new services to the apps more timely, and makes the process of adding value to the apps much quicker and simpler.
Feel free to take this idea and modify it as neccessary in your own RESTful APIs.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
I've been writing asychronous code with the .NET Framework for several years now, and find that the .NET Framework makes a good job of hiding the underlying conceptual details. The concepts are pretty straight forward if you undedrstand how asynchronicity works. As I've found over the years though, these concepts are not always well understood or applied by less experienced developers. By less experienced, I don't always mean junior developers. I've come across senior developers who have struggled with asynchronous code too.
I've helped several developers fix issues with their code that have been due to mis-understandings with asynchronicity, and I've found issues with their code during code reviews that have highlighted basic mis-understandings with implementing asynchronous code using the .NET Framework.
In this article I want to go through the basics of writing asynchronous code using the .NET Framework. I'll use C# to illustrate all examples, but conceptually the code will work the same when transposed to VB.NET or any other .NET language. I'll use examples from our ASP.NET Web API services code base which makes extensive use of asynchronicity to give performant and responsive code. Our mobile apps all rely on these services for delivering functionality to the end user's device. It is therefore incumbent on our apps to be highly responsive and performant. I have therefore made all these services asynchronous to effect these requirements. I may follow this article up in the future with more advanced scenarios, but for now, I will stick to the basics.
What is Asynchronous Programming?Let's start with some basic understanding of asynchronous programming. Most code gets executed in a sequential manner i.e.
- execute line 1
- execute line 2
- execute line 3
We have 3 lines of code that each execute some command, and they each run one after the other i.e. "execute line 1" is executed first. When this has finished execution then "execute line 2" gets executed. When this has finished executing then "execute line 3" is executed. These commands are run sequentially, one after another. This can also be referred to as synchronous code. The next line of code can only be executed when the previous line of code has completed.
var myList = new List<string>();
myList.Add("item1");
myList.Add("item2");
myList.Add("item3");
myList.Remove("item1"); A trivial example could be the code above. The first line creates a string list called myList. When this has completed the next 3 lines then add items to the string list (item1, item2 and item3). Finally, we remove item1 from the list. These lines of code are executed one after the other in a sequential (synchronous) manner.
When code is executed sequentially like this, one command after the other, we say that it has been executed synchronously.
We need to write our code differently when we interact with any kind of I/O device such as a file, a network or database. The same applies when we execute any CPU bound operations such as rendering high-intensity graphics during a game. We cannot make any guarantees about how quickly the device or operation may respond to our request, so we need to factor in waiting time when making requests to I/O devices or CPU intensive requests.
An anology may be making a telephone call to book an appointment to have your car serviced. Immediately after making your booking you need to then write down the date and time of the booking. You may get straight to the front of the telephone queue if you're lucky. Alternatively, you may find you are further down the telephone queue and have to wait to get through to the garage. Either way, you cannot write down the date and time of the booking until you have gotten through to the garage.
In this scernario you don't know exactly when you can write down the date and time of the booking as you may have to wait to get through to the garage.
And this is exactly how asynchronous code works.
When your code accesses I/O devices such as accessing a file, network or database (or makes a request to a CPU intensive operation) you cannot guarantee when your request will be serviced. For example, if you are accessing a database, there may be latency on the network, it may be hosted on legacy hardware, the record you are accessing may be locked and so on. Any one of these will affect the timeliness (or otherwise) of the response to your request.
If your network or database is busy and under extreme load, any request sent over it will be slower than requests made during less busy times. So it should be obvious that executing a command that relies on an I/O device immediately after submitting a request to that I/O device is likely to fail, as you may not have received any response from the I/O device.
Example
- connect to database
- fetch records from database
- close database connection
If you were to execute the above code synchronously, you could easily run into the situation where you are trying to fetch the database records before you have fully connected to the database. This would fail resulting in an exception being thrown. What you instead need to do is attempt to connect to the database, and ONLY when that has succeeded should you attempt to fetch the records from the database. Once you have fetched the records from the database, then you can close the database connection.
This is exactly how asynchronous code works. We can rewrite the above pseudo-code asynchronously.
- connect to the database
- wait for connection to database to be established
- once connected to the database fetch records from database
- close database connection
The two sets of pseudo-code look very similar, with the key difference being that the latter waits for the connection to the database to be established BEFORE making any attempts to fetch records from the database.
Hopefully by this point the goals of asynchronous programming should be clear. The goal of asynchronous programming is to allow our code to wait for responses from I/O or CPU bound recources such as files, networks, databases etc.
Asynchronous programming with C#Now that we understand the principles and goals behind asynchronous programming, how do we write asynchronous code in C#?
Asynchronous programming is implemented in C# using Task and Task<t>. These model asynchronous operations, and are supported by the keywords async and await. Task and Task<t> are return values from asynchronous operations that can be awaited.
Here's a function that POSTs data to a RESTful endpoint, and does so asynchronously. For the purposes of simplicity I have removed all authentication etc from the code samples I will use.
public async Task<HttpResponseMessage> PostData(string url, HttpContent content)
{
using (var client = new HttpClient())
{
return await client.PostAsync(new Uri(url), content);
}
} Things to note.
- The method returns a Task of type HttpResponseMessage to the calling program i.e. the method is returning an instance of HttpResponseMessage (e.g. an HTTP 200 if the method was successful).
- The async keyword in the method signature is required because the method invokes the PostAsync() method in the method body i.e. the method needs to await the response from the RESTful API before the response can be handed back to the calling program.
To call this function we write the following code.
var response = await PostData(url, content); The calling code (above) needs to await the response from the PostData() method and does so using the await keyword. Whenever you invoke an asynchronous method such as PostAsync(), you need to await the response. The two keywords go hand in hand. Asynchronous methods need to be awaited when they are invoked.
Here's another RESTful API method that fetches some data from a RESTful endpoint. The RESTful endpoint returns data in the form of a serialised JSON string (which the calling program will then de-serialise back into an object).
public async Task<string> GetData(string url)
{
using (var client = new HttpClient())
{
using (var r = await client.GetAsync(new Uri(url)))
{
string result = await r.Content.ReadAsStringAsync();
return result;
}
}
} Things to note.
- The method returns a Task of type string to the calling program (the JSON serialised response from the RESTful endpoint).
- The async keyword in the method signature is required because the method invokes the GetAsync() and ReadAsStringAsync() methods in the method body.
To call this function we write the following code.
string response = await GetData(url);
if (!string.IsNullOrEmpty(response))
{
} The calling code (above) needs to await the response from the GetData() method and does so using the await keyword.
Key takeaways- Async code be can used for both I/O bound as well as CPU bound code
- Async code uses Task and Task<t> which represent asynchronous methods and are the return values from asynchronous methods (as we saw in the PostData() and GetData() methods)
- The async keyword turns a method into an async method which then allows you to use the await keyword in its method body (as we saw in the PostData() and GetData() methods).
- Invoking an asynchronous method using the await keyword suspends the calling program and yields control back to the calling program until the awaited task is complete
- The await keyword can only be used within an async method
This is the first in what will hopefully be a series of articles I intend to write on asynchronous programming. I will cover other areas of asynchronous programming in future articles (giving tips, advice, advanced scenarios and even Javascript). Hopefully for now, this has given a taster of how to implement the basics of asynchronous programming with C#. Watch this space for further articles.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
I recently came across some strange behaviour in our ASP.NET Core 2.2 web application. A colleague of mine who was working on some new functionality, had checked in several Javascript files. These were 3rd party Javascript files to add support for drag & drop. The majority of the files for this 3rd party library were already minified, with the exception of one.
For some reason this one particular Javascript file was not minified. So we added the file to bundleconfig.json in Visual Studio so that our build process would minify the file. The bundleconfig.json minifies several Javascript files and outputs the aggragated file as site.min.js. Whilst I was testing the latest version of the app I was getting all sorts of errors in the browser as many of the Javascript functions were not being found. This seemed strange, as everything had been working perfectly, and all we had done was check in a few Javascript files.
Looking at the site.min.js file that was on the build and test servers, it became apparent that the site.min.js file contained only the contents of the un-minified 3rd party Javascript file. All of the other files we were minifying had somehow been removed from the resultant site.min.js file.
After much investigation I narrowed down the issue to the following command in our build pipeline.
dotnet publish -c release This command was recreating the site.min.js file, but failing to include all the files specified in the bundleconfig.json with the exception of the un-minified 3rd party Javascript file. I excluded this step from the build process to check, and sure enough, the culprit was definitely this build command.
I managed to solve the problem by manually minifying the culprit Javascript file and adding it to the project in its minified form. I then excluded it from the bundleconfig.json minification process. This has now solved the problem, and everything works perfectly again.
So basically, if you're including 3rd party Javascript files, make sure you add them to your Visual Studio project in minified form (unless you're using a CDN of course). Don't attempt to minify 3rd party files in your build process. Only minify your own Javascript files in your build process. It took me a few hours to diagnose and fix the problem, so hopefully by reading this, I may save someone else the same pain I went through fixing the problem.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
Whenever I'm mentoring a more junior member of the software development team, there are always two primary traits that I encourage them to learn. These are traits that transcend programming language, methodology, technical stack or anything else that may be relevant to their role. These are structure and dilligence. Both of these should permeate through everything they do in their every day work. Being mindful of these will help them become better as software developers. I will explain why these traits are so important to the software developer.
Structure
Approaching your work with a structured mindset allows you to demarcate and separate out the various elements to the problem you are solving. From grouping the different areas of the requirements specification, to grouping the components and classes in the class hierarchy, to grouping the related unit tests....having structure allows you demarcate the boundaries between these different elements. Everything has a structure. The trick is to clearly define it and communicate this to the rest of the team. If you are documenting the requirements to a piece of functionality, clearly structure the document to demarcate these different areas e.g. functional requirements, non-functional requirements, UI considerations etc. If developing a new component, your class structure should clearly demarcate the different behaviours and areas of responsibility from the class structure and their interactions. Anyone reading through the code should be able to quickly determine what the different classes do and how they relate to each other from the structure you have implemented. Group similarly related elements together and enforce this in your coding standards document. Everything you do should be structured, logical, and consistent.
Dilligence
Approaching your work with due care and dilligence will help in eliminating mistakes and make you a better developer. Be conscientious and mindful of what you are doing at all times. Before checking in that code, make sure you do a diff, run a full rebuild and execute all dirty unit tests. This may take additional time, but it will always be quicker than the time it will take to fix a broken build. If writing a document such as a requirements specification, take the time to proof read it, check it over for spelling and grammar as well as accuracy. Work smart, not fast. Reducing the number of mistakes you make by being more dilligent will earn you a reputation as someone who is dependable, produces high quality and takes their role seriously. Don't be that person who is known to constantly make mistakes, breaks the build or submits code that doesn't work because they didn't test it sufficiently enough.
By applying structure and dilligence to everything you do will have positive benefits on your work. These can be applied irrespective of your particular role (developer, tester, designer) or what tools and / or technologies you use. I would prefer to work with a developer who took these traits seriously than a developer who thinks producing more lines of code than the next developer makes them more productive. I would always pick quality over quantity. A customer is far more likely to forgive a slippage of a dealine if they eventually get something that is of high quality, rather than something delivered on time that contains bugs.
Be structured and dilligent and apply these with rigour and I can guarantee that the quality of the code produced by yourself and your team will increase.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|
At what point should you consider rewriting a software application? Just because an application is old, doesn't mean it should be rewritten. So what factors ought to be considered when attempting to justify rewriting an application? There are many things to consider, and rewriting any application should never be taken lightly. In fact, doing a cost-benefit analysis is probably a good starting point.
In today's fast paced software development environment, today's latest fad can quickly become tomorrow's long forgotten hype. What does it even mean to be legacy? According to Wikipedia
Quote: a legacy system is an old method, technology, computer system, or application program "of, relating to, or being a previous or outdated computer system" This article is not intended to be a detailed discussion of the considerations to take into account when looking to rewrite a legacy applicaiton. That would be a considerably lengthy article. Rather it is to look at some examples from my own experiences involving legacy applications. Like most developers, I thrive on working with the latest shiny new tools, but there are also times when you need to work with that legacy application too. I have heard many developers berating these legacy applications. Sometimes for good reason too. But quite often, the legacy application has been working away for years, quietly, solidly and not caused a fuss.
I've worked with many legacy applications over the years. Some were surprisingly good, some just plain awful. Some of them, despite their age, were rock solid and were capable of running far into the future. Others spluttered and juddered their way along and needed a lot of man-handling to keep them running.
Just because an applications is legacy is not reason enough to justify a rewrite. I remember working for one particular company where the business critical back-end application was developed in COBOL. It was over twenty years old but rock solid. It rarely caused problems or generated errors. It just worked.
Another company I worked for many years ago also had a lot of legacy code (and according to sources at the company, much of the legacy code is still there to this day). The code was part of their core business logic and had been around for over a decade. This was accountancy and financials logic, and whilst the code had been updated with bug fixes over the years, it didn't require much man-handling to keep it up and running. In fact, when they decided to upgrade the application to use newer development environments and tooling, they kept much of the legacy code as they knew it worked. They didn't want to risk screwing up their core business logic by rewriting it.
Age alone is not a deciding factor when considering whether to rewrite an application. There are many legacy applications that run just fine with few problems. Alternatively, there are a great many applications developed with modern technology and tooling that are plain awful.
A few things to consider.
- Is the application code buggy and / or cause regular problems or errors?
- Does it require man-handling to keep it up and running?
- Does it meet non-functional requirements i.e. is secure, performant etc.
- Is it easy to extend and add new features?
- Does it require legacy hardware that may be insecure?
- What are the running costs of the application (development costs fixing bugs, server / hardware costs, third-party costs etc)
- Does it interact with third-party applications that may have updated their APIs?
- Has it been developed using outdated environments or tools?
Not all of these considerations will be applicable to every scenario, so don't take the list in its entirety. They are merely intended to be conversation starters to elicit further discussion. Deciding whether or not to rewrite an applicaiton is not a decision that should ever be taken lightly, but equally you need to take into account many different pieces of information and assess them in the context of the bigger picture. Age alone is not a compelling argument for a rewrite, but taken in the context of other factors, may form one of them.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C.A.R. Hoare
Home | LinkedIn | Google+ | Twitter
|
|
|
|
|