Research-Inspired Extensions Hit Visual Studio App Store

Introduction

There are TONS of extensions in the Visual Studio App Store (i.e., Visual Studio Gallery), most of which are small improvements on VS's default tooling and, in my opinion, not worth the hassle of installing.  However, there's a new class of extensions, inspired by years of software engineering research, that show real potential to enhance developers' performance. Today I'll present one commercial extension and one free open source extension, both inspired by research, and I'll discuss how developers can use these tools to dramatically improve their performance on common development tasks.

Continuous Testing Research   

 

While making changes to a mature code base it's easy to accidentally break existing features, causing a regression. In fact, it is so common that while at MIT in 2004 David Saff created an (Eclipse) plugin to automatically run tests in the background and let developers know as soon as they'd broken an existing test. The driving principle being that the sooner any regression is caught the less of a productivity drain it will be, echoing the concept that the earlier in the lifecycle a bug is found the easier it is to fix. Not surprisingly, when they tested this idea in a user study developers performed much better when supported by a continuous testing tool. As you can see above, almost three times as many developers completed the task when given continuous testing tool support (i.e., 78% completed with tool support, only 27% without), relatively strong evidence of its usefulness.

Continuous Testing in Practice 

Fast forward to 2012 and not much has changed in testing. While it was more likely that your team had a continuous integration server, which may notify you after checking in a regression, Saff's original vision of instant notification has largely gone unrealized. Fortunately, Remco Mulder and team have since released NCrunch for Visual Studio. This extension, as shown above, provides decorators on every line of executable code (green, red, or black dots) that indicate (a) whether that line is covered by at least one test and (b) whether all tests executing that line are passing. It continuously runs tests in the background each time there is an update to code, and thus developers are aware of regressions almost immediately after they happen. Based on Saff's original research developers should see a measurable gain in performance thanks to this early warning system. 

Software Search Research

Anyone who has programmed in Visual Studio has used FindInFiles (above) at one time or another. Like its predecessor 'grep', FindInFiles searches each file in a solution and matches the user query against each line using regular expressions. Unfortunately, that means it is slow to execute and good search queries are extremely hard to construct. So hard, in fact, that in a 2006 study of developers performing maintenance tasks 88% of user's queries failed to find anything relevant. Fortunately, since about 2004 many researchers have been working on this issue. Andrian Marcus was among the first to use information retrieval techniques (think Google and Lucene) to index and search source code, making it much easier to search local code bases. Adding to this progress, researchers (e.g., Emily Hill and Sonia Haiduc) have investigated how to help users create better queries, one of the biggest challenges in code search.

 Software Search in Practice 

Again, fast forward to 2012 and not much had changed in software search. Developers were still struggling to use FindInFiles, scrolling through screens of irrelevant results. That's when a university professor, an open source contributor, and I came together to create the free, open source Sando Code Search Tool for Visual Studio.  Sando uses Lucene.NET to index source code and thus can search code almost instantly, providing both a keyword based search (e.g., [add child] as shown above) as well as literal search via quotes (e.g., ["//add person as a parent"]). It uses some of the latest research to recommend relevant search terms as users type. Our data shows that developers get significantly better results and are happier with those results when they use these recommendations. Based on previously mentioned studies, where developers' searches failed 88% of the time, using an advanced search tool like Sando can dramatically improve developers' productivity, especially when working on a new or unfamiliar code base.

Note: Another great Lucene-based search tool for Visual Studio is Entrian Source Search.

 Which Extensions to Use? 

As a former extension junkie, who downloaded any Visual Studio Gallery extension that was even vaguely relevant, I'll never stop experimenting with new extensions. However, for my stable set of core extensions I now have a higher standard. I look for extensions that will save time or increase code quality. If I can't make a clear argument as to why I have an extension installed when asked by a colleague, it doesn't make the cut. While I may be a bit extreme, I'd encourage you to put a bit more thought into which extensions you install. Don't believe every snake oil salesman hawking their extension; look for those who built extensions based on lessons from the past.  

Downloads: NCrunch, Sando