The Life of a Project - Issue Tracking: Day 1

Read this first, please

This is the first in a series of articles that hopefully will be educational to novices and helpful to more advanced developers alike.  I hope to learn a lot from doing this also.  I am taking the approach of maintaining a journal of the day-to-day process of software development starting from a concept and going to a product.  I hope you enjoy this series of articles.

Introduction

I had surgery on Friday June 6 and as a result am stuck at home until at least the 16th.  I could be reading, watching TV or doing just about anything a person can do while sitting on his/her butt.  Being a programmer, I have decided to take this time to develop a software application.  I have no shortage of ideas for an application, but there is one that I have wanted to write for a long time.  One of the problems I have at work is keeping track of change requests, program changes, ideas, etc.  What I need is an issue tracking system.  Actually, I have an issue tracking package called DevTrack which is not bad and in fact has many great features, but for many reasons I do not use as I should.  The primary reason I do not use it is because I feel that it just adds work, instead of augmenting my work.  I have worked with a few other issue tracking systems and they all seem to have this same common problem.  For this reason, I have chosen to develop my own issue tracking system that I will be less of a burden and more of a benefit.  

What I want is a tool that is almost transparent which allows me to record issues, change requests, bugs, defects, ideas and to-do lists quickly or ideally automatically.  I don't really care if it is integrated with my development tool as much as I care that it is virtually transparent.  I want to be able to easily put data into this tool without jumping through hoops and I want to get it out just as easily.  I don't want a conventional MDI/SDI/whatever application, I want a service that I can call upon to take my requests and deal with them accordingly.  I want to be able to make changes to the system while making requests, instead of having to jump to some other application, do something, then come back and finish my request.  Basically, I want a system that can read my mind, do what I want, when I want and interfere with my life as little as possible while making my life simpler.  Sounds like what most users want, doesn't it.

I am creating a short series of articles entitled "The Life of a Project" to detail my journey from here to the end of this project.  I hope to use these articles as a means to teach some valuable programming techniques (and learn some for myself) as well as to deliver a product that others can use and benefit from.  This is the first in that series and so will probably be somewhat more verbose than most.  I hope you enjoy reading them.

My little diatribe

Somehow, when most applications are designed a process is followed that inevitably leads to one or more nice and neat application programs.  These may be Windows GUI applications, web apps, or whatever, but the scientist in us developers likes to see straight lines and nice and neat boundaries between systems with well defined lines of intersection, union or whatever.  Part of our job is to take what users (managers, bosses, etc) request and turn that into code.  Often this is very simple and we can fit all of the requests into a nice and neat package, but with more complex requests, we end up with more complex designs where the lines are not so neat and the boundaries not so clear.  The reason for this is obvious, computers don't talk human and humans don't talk computer.  Until this changes, there will always be a challenge in software development.  

Some people will argue that we are making real progress in this area because we are adding more and more layers of abstraction which make it easier for developers to make it appear that computers indeed do talk human.  They may be right, maybe abstraction is all that is needed, but we all know that all of this abstraction is adding risk to.  What happens when there is a serious flaw in an abstraction layer way below our applications?  The eternal constant (change) takes over and we are forced to either redesign our applications or pigeon-hole them to work with the underlying changes, or worse some abstraction layer below ours adjusts itself to accommodate the change which means either that later was pigeon-holed or redesigned, forcing more changes down the line.  Where does all of this end?  I guess you could say that we will reach a point where the abstractions will become so stable and standardized that change will not be an issue, but I doubt we are anywhere near that point now.

So where does that leave us?  Well, it leaves us right back where we started, we must take a set of requirements and design an application that leverages as many or as few of these shaky abstractions as we feel necessary to fulfill the requests while trying to balance the risks.  We can choose more control by using fewer abstractions, but this often forces us to either reduce the feature set, increase the amount of code we write, or invent our own abstractions to replace the ones we avoided earlier.  We could take the risks of increased abstractions and hopefully benefit from the productivity gains, code reduction and feature enhancements that these abstractions promise, but we must be careful to choose abstractions that we feel carry the least risk.

OR, maybe the pipe dream that is object oriented programming is just plain wrong.  Maybe it is a house of cards that will ultimately collapse upon its own weight.  OOP has been around a long time and has promised quite a bit, and in truth has delivered some of its promises, but I always have a nagging feeling in the back of my head that says "it just isn't right."  Maybe I misunderstand OOP and maybe all of these abstractions are just illusions any my concerns are misplaced, but I doubt this too.  How can we have a system dependent on so many chances of failure?  How can we have a system that relies on a lack of change when we know that change is not only inevitable, but a daily occurrence?

Then, a little voice in the back of my head says, "what about all those mainframe applications that have been running unchanged for 20 years?".  Hmmm...could it be that these systems prove that OOP is and always has been wrong?  Could it be that COBOL is indeed the language of kings?  Could it be that the evil empire (Microsoft) created OOP to drive sales of its software for the next 100 years?  I doubt it, I suspect that the only reason any of these applications continue to run is because they are batch-oriented systems with well defined inputs that for the most part don't change ever.  I suspect that the interactive portion of these applications has indeed changed quite a bit, that the portion that deals with communicating information (either receiving or sending) has been changed many times while only the core processing engine has remained unchanged.

So where does that leave us?  Well, it leaves us right back where we started, only more confused than before.  Should we use OOP techniques or should we use some other programming technique such as functional programming or structured programming or should we give up and find a nice easy career as an accountant? (Hmmm.  Maybe all of this is just the effect pain medicine has on a programmers mind?) 

I always hate reaching this point in the application design stage because its like walking on a treadmill, you can go a long way and never get anywhere.  The reality is that in today's world, to write an interactive application with any-kind of human usable interface, OOP is the only way to go.  We must accept a certain amount of risk, in-fact we must accept inevitable and uncontrolled change, and we must cope with at least some abstractions.  Our job here is to wisely choose the abstractions we rely upon.  

Getting started

Before we can choose what abstractions we want to use, we need to do a little requirements gathering, some specification writing and put on our creative hat.  We need to understand the project in detail, from the users perspective and from a business needs perspective.  Then we must create a design that takes into consideration all of these needs and solves all of the problems.  So, without further ado, lets take a look at our requirements.

Requirements 

The users of this system have been interrogated and here are their demands:

• Want to record user change and feature requests
• Want to record defects and track their progress
• Want to record Ideas for existing projects, new projects, old projects, etc
• Want to record to-do lists
• Want to record change history for projects
• Want to report on outstanding changes, features, defects
• Want to update the contents of various items with ease
• Want to hide old data, but be able to access it when needed
• Want to report on past changes, defects, etc.
• Want to organize items for personal ease of navigation
• Will only use this app if it is dead-easy
• Will only use this app if they can customize everything as needed, when needed

As usual, the users don't tell us everything we need to know.  There are questions that must be asked and answered.  Here are some of them:

• Q: What information will be gathered for a change or feature request?
• A: I told you, I want to customize everything.
   
• Q: OK, What do you want the reports to look like?
• A: I told you, I want to customize everything.
   
• Q: That's great.  How do you see this information being entered?
• A: I don't know, but I know I don't like any of the tools I have used so far.
   
• Q: Define "dead-easy"?
• A: I don't want to do anything, I just want it to do everything for me.
   
• Q: OK, but obviously you will have to do something.  How much do you see yourself doing?
• A: I see myself clicking a button, entering the information and clicking OK.
   
• Q: Do you want to be notified by email when someone-else puts an issue in to the system?
• A: What?  I don't want anyone but me to use this system.  Anyone else will screw it up.

That's about all we are going to get out of the users for now.  This may seem funny, but is not too far from the truth.  Users sometimes know what they want, but not what they want.  Specifics are hard to come by and sometimes can't be found without a little (lot) more research.  That does not mean that talking to the users was a bad idea, only that we now realize that users aren't software developers, and typically don't think the way we do.  Trying to get a user to decide between a tree control, a list control, or a group bar is like trying to get a 2 year old to pick what career he wants to take when he/she grows up.  It's not that they aren't smart or don't understand their needs, but the question is beyond their scope of understanding.

Requirements - Take 2

We have spoken with the users, asked our follow-up questions and come to the conclusion that we need to do a little more research.  We know that our requirements list is a little light on details so we need to fill it out a little with some educated guesses combined with some well placed research.  

First, lets take a few educated guesses and get them out of the way.

• Lets assume that keeping track of when items are added/changed is important
• Lets assume that keeping track of who adds or changes items is important
• Lets assume that there is some kind of workflow process involved since most jobs have some workflow to them.  This workflow will probably start with opening and issue and end with closing the issue.
• Lets assume that users should not be allowed to permanently delete anything.
• Lets assume that a typical GUI application is not enough to satisfy this crowd.
• Lets assume that certain fields are common such as Title, Comments, Date encountered and Status.
• Lets assume that we can't assume what other fields the users will want.
• Lets assume that getting data out will be important so we need to make reporting as powerful as possible with printable as well as viewable reports
• Lets assume some form of logon is desired

Now, lets do a little research and see what we can come up with.  We are going to look at what the competition has to offer and try and glean some ideas from each of them.  Here is a list of the products we looked at.

• Fog Creek Software - FogBugz
• TechExcel - DevTrack
• SeaPine Software - Test Track Pro
• Elsinore Technologies - Visual Intercept
• Softwise company - PR Tracker
• Issue View - IssueView

We could have looked at many more, and it is clear that this is a market littered with applications.  There are several things we learned from doing this research.  Some of the things we learned are:

• All of these take an application approach to the problem.  What I mean is that all of these is an application which a user must launch (either as a standalone program or a web app) in order to perform usual tasks.
• Many are project based allowing for the creation of "projects" to help organize issues.
• All provide a pre-defined set of fields with pre-defined value lists.
• Most allow for a high level of field/value list customizations.
• All provide some form of email notification of new issues/changed statuses.
• All are clearly targeted at team development.
• Priority is a common field
• Attaching of files appears to be common
• Embedding images is often possible
• All provide some form of list-view based approach to listing issues.
• Common statuses include Open, Closed, Under Review, Fixed - Awaiting Review, Cancelled, On-Hold

Requirements - Take 3

We are now armed with some more information on the subject as well as some solid ideas of what features users will really want and use (based on what other applications provide).  Lets go back to our users and examine their workflows and behaviors a bit.  We want to learn how they do their job so that we can best tailor the application to them.   We are going to look over their shoulders for a while to passively observe and we are going to share what we have learned with them to get some feedback from them.  Since I am the only developer available at this time, I am going to examine my own workflow and process.  

Some basic details:

• I work in the R&D department of a market research firm
• Most of my work is done in Visual C++ 6.0, though Visual Studio .NET is on the horizon
• I support several web based (ASP) applications
• I support several GUI applications
• I support several Windows NT Services
• I typically leave Outlook, Developer Studio and IE open at all times.
• A large part of new code is done in C++.  The rest is ASP and HTML applications.
• I use Visual Source Safe

How projects/issues/change requests arrive at my desk:

• We sell a client something which we don't already posses and a sales person passes on that information to me through paper or email.
• We come up with an idea and choose to build it.  This usually involves discussions over lunches, and during breaks.
• We discover a problem in an existing application which usually occurs because I review a log and find an issue, or because someone else in the company encounters something and emails or calls me with the issue.
• A client asks for a change in an existing system which is passed down to me usually through email.

How I currently manage issues, change requests, ideas, to-do lists:

• I have a product (DevTrack) which I keep long-term issues in such as bugs I don't intend to fix soon, but I don't use it regularly.
• Typically ideas are passed between myself and my boss through email.
• When I finish a change, fix a bug, or whatever I put the change into production and notify appropriate people.

What can I tell you about the other issue tracking products that I looked at:

• The "application" approach works for managing issues, but not recording them which is where I struggle most
• Project based systems are fine, but I would like to be able to have issues that exist across projects and occasionally are assigned to No project
• Different types of items require different fields (ie. A Change requests will require fields that a Feature request will not, and idea will require only a few fields, etc)
• Priority is important, but difficult since everything is a high priority.  I would like a better way to assess the real priority of an item.
• Attaching files and images is a must.
• I will allow others to put issues in IF I can control what they can do and where it goes and WHEN I see it.

Vision

Having been through the requirements analysis up-to-this point, I have decided that I need an innovative solution to issue tracking if I (and other developers) are going to use it as an every day tool.  So, here is my vision for this tool.

I see a link on the Start menu (and a tray icon with a popup menu) with specific tasks I want to perform such as "Record change request", "Record bug", "Review open bug list", "Print bug list", "Close issue".

I see these same links available as toolbar icons in my favorite applications such as Developer Studio, Outlook and IE.

I see a dynamically generated form created for each type of task.  These forms should be customizable on-the-spot and I should be able to add new task-types on-the-spot.

I see a special email-recipient which I can forward emails to and have them automatically cataloged within this system.  I would like to have addresses like "issues@mycompany.com", "ideas@mycompany.com", etc" and I would like to be able to forward any email I get or send (or CC) to one or more of these addresses and have it inserted appropriately.

I want to be able to create logical groupings of issues which can overlap and I want to create my virtual email recipient addresses when defining these groupings.

I see an IE namespace extension where I can drag-drop files to that will automatically create entries and attach them to issues.

I see a journaling tool that runs in the background monitoring file changes to .cpp, .h and .asp files (and others) that records day-to-day changes and provides a way for me to comment these changes and to automatically extract comments from the source code itself.  These journaling entries should be part of my issues DB.

I see an application interface that allows for management of issues, but is not used much.

What's next?

Tomorrow, I will review where I am and begin to write a specification for this new application.  I may also create a simple prototype of some of the features, but I will wait and see how the specification goes first.