The Standalone Programmer: Real Time Software Development

Introduction

Have you noticed that development schedules are getting shorter while the requirements list is getting longer?  Does all your time seem to be crunch time?  Does your requirement list seem to be progressively including more sophisticated features?  Is your software being used by more and more people?  Does it seem that the level of respect for custom software is diminishing?  If so, you are not alone.  I believe this trend will not only continue but will accelerate.  Coping with these issues and learning to adapt and thrive within these constraints is going to become critical to long term success.  

Many of you have no doubt been reading on the subject of the "real time enterprise" which discuss the needs of businesses in the 21st century revolving around real-time data.  These articles discuss the growing availability of software applications to improve communications between business divisions, departments, partners, vendors and customers.  The demand for these applications is driven by competitive pressures, profit potential, client retention and acquisition requirements, and to an increasingly lesser degree, the FAD factor.  Some of these applications are quite simple while others involve terabyte size databases, disparate database technologies and platforms, etc.  Disconnected technologies like web services, SOAP and the internet are increasingly imperative to project success and the hype of these technologies only masks the complexity which is the reality we must face.  Obviously, these tools of communications are being developed by people like you and I. 

On the one hand, this new business environment is a good thing for software developers because it will generate jobs and hopefully increase salaries.  On the other hand, this new business environment will require some changes in the way software is developed and how software developers think about requirements, schedules and deliverables.

Is any of this really new?

No doubt many people have asked this question.  I believe that just like the term "new economy", the news media has hyped the term "real time enterprise" beyond all recognition.  At the core of this concept, though, there is a fundamental change in the way businesses are run.  The change is that there is a fundamental move from hardware being key to business success (ie. factories, sewing machines, etc) to software being key to success (online sales, instant stock quotes, CRM, ERP, etc).  The actual nature of business, though, I do not believe has changed, only the way it is performed.  Hardware stores will continue to sell hardware, automobile manufacturers will continue to sell cars, etc. 

Communications has always been important to businesses, but the demand for instant gratification from CEOs, vendors, accountants, sales people, the government and ultimately the consumer has advanced at a truly phenomenal rate.  Why has this happened?  Two words: the Internet.  I can't say where it started, but it got a major push when consumers were exposed to the internet as a medium for real consumer activities such as purchasing products, researching products and companies, processing returns, refunds, etc, the push was on for real-time systems to support this growing need.  And this was not the real beginning.  Some will say that this change began with the creation of the first personal computer. 

So, is any of this new?  I believe the answer is YES.  There is a crucial change in the way businesses communicate that is dependant on software which is indeed new.  It is this change that is creating the demand for the so-called real-time enterprise of the future.  It is this change that software developers must understand and come to grips with.  It is this change that will have a significant impact on the future of software development.

Business 101

It is increasingly clear to me that one thing that is needed is for software developers to have a clear and deep understanding of the needs of the businesses we serve, not at a technology level, but at the core business function level.  With this knowledge, we can put our talent to work on solving the most relevant problems which face the businesses we serve.  Setting aside for a moment the issues of corporate culture, nepotism, inept managers, etc. Lets discuss the mechanics and driving forces behind industry.  I realize that most of you will already know all of this (and much more), but I feel the investment in reading will be worth the effort.

Everyone knows that businesses exist to make money.  But, why does a particular business exist?  What is the need that drives it, what is the client demand that feeds the money machine?  How does a business acquire and retain customers?  What competitive pressures drive innovation within the industry?  Obviously, every business and industry is different, but there are significant similarities and there are very important reasons why understanding these forces is important to the standalone developer.  So, why is understanding these forces important?  Well, as you have no doubt already experienced, specifications are a form of vapor-ware that rarely even exists in the minds of those asking for development effort.  In most businesses, the true specification is defined by a business need.  Regardless of  internal politics, clueless managers, and poor direction from the higher-ups, most software is intended to solve some business need.  All too often, the software fails to solve this need, not because the software does not work or does not meet its specification, but because it solved the wrong problem or interfered with another critical business function or was created on a faulty set of specifications that were created ad-hoc.

For most industries, there is a well defined reason for its existence.  The reason may be a basic human need such as the reason for the agriculture industry, or it may be a demand for leisure time activities such as the reason for the tourism and entertainment industries.  Whatever the reason is, there is one and this reason results in demand for products and/or services.  Within an industry customers are king, because ultimately, they drive the money machine.  Customers can be individual consumers like you and me, other businesses or the government.  When it comes to customers, acquisition and retention are the two key issues which every business must cope with.  Often, acquisition is much more difficult and costly than retention.

Within a particular business, there are often unique circumstances that exist which also influence the need for and interest in new software development.  Issues related to the companies business plan (if one exists), cash flow problems, geographic constraints, etc.  These unique characteristics are equally important to understand because often they will affect the finer details of how software needs to function.

It is important for software developers to gain an in-depth understanding of not just the individual business, but the industry as a whole so that we can make better informed decisions.  Increasingly, we are being asked to develop systems on short notice, with short development cycles that are feature rich and specification poor.  Understanding the underlying demands that created the need for the software is crucial to being able to deliver the right solution at the right time.  This knowledge will influence everything from architectural design to user interface design and the more knowledgeable we are, the better fine-tuned these decisions will be to meet our clients actual needs.

Old habits die hard

Books, Books everywhere but not a word to read

For some reason software developers tend to have a hard time seeing past the end of our technology.  I realize that this tendency exists in all people, but as software developers we need to have a broad and deep understanding of many non-technical subjects in order to produce quality software.   Though I have not visited many developers offices, I strongly suspect that if you browse their bookshelves, you will not find many books on marketing, advertising, sales, managing people, industry specific books, etc.  I certainly understand this habit, because it is a particularly hard one to overcome, but one that needs to be overcome for us to reach our fullest potential.

I would suggest that you think of software development as a B2B service.  As such, understanding your target market is important.  Unless the software you develop is targeted at other software developers, all those books on C++, .NET, design patterns, security, etc, while interesting and useful, are not enough to get past age-old problems of communicating with clients, understanding client needs and delivering truly great results.  I am not suggesting that you stop reading any of the books you read now, just that you spend some serious time investigating your employers business and the issues the nature of job requirements of your users. 

So, what kind of materials should you read?  I would suggest that you start by looking on the bookshelves of your manager and coworkers.  My boss has a bookshelf loaded with books on business, managing people, advertising, etc.  As long as I return them, I am welcome to borrow any of them.  I suspect that your coworkers would be equally willing to share their resources, if asked.  There are also, no doubt, many magazines targeted at your employers industry.  I find it easier to read a magazine than a book simply because of time constraints.   I work in for a market research firm, and my employer keeps a table with market research magazines near the main entrance.  Anyone is welcome to pick them up and read them.  I have also subscribed to numerous non-technical magazines including Business Weekly and Fortune.  I have found that keeping up-to-date with what is going on in the business world helps to understand not just business, but the direction software development is going in.  

I am constantly looking for insight into the direction software development is taking because software development does not exist to serve itself.  The need for and demand for software is driven  by businesses seeking to do what they do best: make money.   Having insight into where business is going, what clients are demanding, what competitors are doing, all helps to stay one step ahead of the demands that will inevitably land on our desks.

Nothing can be done in a day

How many times has someone asked you how long a particular task is going to take and you responded with a ridiculously small amount of time, like a day, 4 hours, or even a few minutes?  And, how many times have you missed the mark on these timelines?  I don't know about anyone else, but for me this has been a huge source of problems.  At least one of the reasons I continue to make this mistake is that I tend to think in terms of "programming time" (similar to "bullet time") which is a magical time system that does not exist in reality.  If the only thing I had to do was fulfill this one request, and the necessary tools, source code, database configuration, etc were already present in the necessary places, my time estimate might, just might have been accurate.

Most of the time, though, there is more to fulfilling a request than just the pure coding aspects.  Not only is there more than just coding, there are scheduling conflicts that inevitably interfere with these short-term requests.  Meetings come up, emails must be responded to, CP time, lunch, other requests, are all likely to occur on any given day of the week.  Failing to take these issues into consideration when making time estimates and promises is a sure fire way to loose credibility, sleep or both.  

Coping with requests for time estimates is important, because you will be asked to give such estimates many times and in general, you will be expected to deliver on those timelines.  A common mistake I make is to try and give a verbal estimate on-the-spot.  This is a mistake.  Giving verbal estimates on-the-spot belittles the actual effort required to make these estimates and will almost always produce inaccurate results.  Although I have not completely overcome this habit, I am working on it.  The strategy I am trying to take is to tell people that I will check my schedule and get back to them in a few minutes.  Then, I sit down with Outlook open and send them an email with the time estimate.  Instead of writing a number of minutes/hours/days first, I write a brief overview of what I have on my schedule already.  This includes ongoing projects and scheduled activities (ie. meetings, deadlines, vacation, etc). Next, I write a brief specification for the request I was given.  Once this is done, I estimate how much time it will take to do in "real time", and then I propose when I can do it, noting any schedule changes that will be necessary.  Finally, I reorder the text in the email so that time time estimate is first, the specifications second and the current schedule/ongoing activities is last.  Often, I find that there are questions I need to have answered before proceeding.  

By taking the 5 minutes it takes to write this email, I typically save myself the headache of having to revise the timeline, work overtime and loosing credibility.  What I have learned is that even the smallest of requests requires that all the typical steps in software development be taken.  This includes requirements gathering, specification documentation, and design phases.  Failing to respect these simple projects and garner credibility makes it much harder to have credibility when larger projects come along.

Why is this particularly important in the real-time enterprise?  I have seen an increasing tendency for seemingly small requests to turn into much larger problems.  Remember how those requirements lists are getting longer and applications are becoming more feature rich, well these small requests typically must be integrated into these more complicated applications.

Accepting and/or excusing defects

There is a tendency among software developers to accept defects as just being part of the process.  To some degree this is necessary, but we must make good decisions when deciding what defects to allow to survive a release.  Within the context of the real-time enterprise, communication is key, so any defect which adversely affects the communications provided by, supported by or driven by your software must be dealt with before release.  Understanding the business need behind the software we develop will make accessing the severity of defects incredibly simple.  If a defect can adversely affect the business need, it must be fixed.  It is as simple as that.  If a defect cannot be fixed within scheduling requirements, the feature which exposes the defect must be disabled or removed until it is corrected.

I cannot stress enough the importance of eliminating defects in the communications our software provides.  Delivering incorrect data, inconsistent data or even delivering accurate data that is not timely or not formatted correctly are critical problems that cannot be allowed to exist in the wild.  And data is not the only aspect of communications that we must take seriously.  Presentation of information, particularly in systems targeted at clients or end-users, is also critical because it provides an impression of the company they are doing business with, and ultimately reflects upon the software developer.  

Beyond just insuring that the software we develop delivers accurate, timely, consistent and reliable results, we must also take into consideration how our software will be used and extended and create systems and processes to insure that communications are not compromised somewhere down the line.  You are probably wondering what I am talking about here, so, let me explain.  Many of the systems I develop (and no doubt, you develop also) provide features that allow users to configure the systems and extend their behavior.  As an example, one of the systems I developed is an internet surveying package.  At the core of this system is a set of Windows NT services that deal with logically representing a survey, processing the flow of a survey, persisting respondent data and much more.  This core system has been tested quite thoroughly to insure that each question type (ie. radio button, check boxes, grids, etc) work as they are intended to, generate HTML as desired and perform within reasonable limits.  There are safeguards built into the code to insure that things work according to plan, and when things go wrong, the condition is detected, logged and someone is notified.

All of this is good and important, but is not enough.  Beyond this system, there is survey design software which allows a user to create new surveys and test them.  This software provides the user with a web based interface for adding questions, editing questions, creating scripts to control flow of the survey and much more.  This system has also been tested to insure that each feature works as expected and generates the desired results.  But what about the final survey someone creates with this software?  In essence, it is an entirely new application.  It runs within a runtime environment (the set of Windows NT services mentioned earlier), contains flow logic, can contain scripts and in general can do just about anything any web application can do.  How can I be certain that the surveys someone else creates actually work the way they intended?

Many developers would say that it is not our job to insure that what someone does with our software will do what they want, only that our software does what it was designed to do.  Not too long ago, I would have agreed with this, and in many situations, I still do agree with this, but within the context of a real-time enterprise, that is not good enough.  We must be concerned about how our software will be used and we must take steps to eliminate potential problems that others will introduce.  Why?  Because we are providing a B2B service that is not just a one time deliverable piece of software, but is a living mechanism for communications that are increasingly critical to our employers and clients.  

Software is not enough

As I said before, software developers have a tendency to not see beyond the technology aspects of our jobs.  We need to realize that the software we develop will be used by other people to achieve some business need.  We need to think about how our software will be used and plan for the questions and problems that will inevitably arise.  One aspect of this is realizing the importance of documentation and making documentation a part of our development process.  However, documentation is not enough.  We must also think in terms of process.  We need to ask ourselves how our users can use our software and know without doubt that what the software produces (with their specifications) is what they actually want.

Consider the example problem I gave above regarding surveys created by someone else using the survey designer software.  What can be done to allow these users to create surveys with total confidence?  Some of the technical steps I have taken include:

1. Providing a testing environment where users can create, test and preview their surveys without affecting production systems.
2. Providing script debugging features.
3. Providing a tool to automatically generate test data.
4. Providing tools for viewing the data recorded by a survey.  This includes reports such as frequencies by question, as well as access to the raw data collected.
5. etc.

These steps are important and greatly advance the goal of having users be confident in the surveys they program.  However, these technical steps are not enough.  There must be processes in place that are designed to find problems with the surveys that all of this technical testing cannot.  Some of the non-technical process we put into place include:

1. Requiring that someone other than the original survey creator review the final survey design for accuracy.
2. Requiring that test data be generated and the results of that data reviewed by at least two people.
3. Requiring that live data be checked after the first few respondents have been taken.
4. Providing an easy to use system for recording issues and notifying me of these issues.
5. Etc.

With these technical features and processes in place, I feel that survey creators can feel confident in the surveys they create, but how can I feel confident in the surveys they create?  For this, I have included extensive diagnostic features within the code that drives these systems.  This diagnostic code checks for known issues with survey designs, (like not using an up-to-date presentation template, failing to properly set statuses for completed surveys, etc), and also checks for unexpected conditions like scripts that never return (ie. infinite loops), questions that get asked repeatedly, SQL queries that fail, etc.  When any condition like this occurs, the condition is logged along with details regarding which survey had the problem, which respondent had the problem as well as other details needed to isolate and diagnose the problem.  In addition to being logged, I can have the system automatically notify me, through email, of any of these conditions.

So, how do you apply this to your applications and projects?  

1. I suggest that when you develop software that you work with actual users to determine how the software will actually be used and what they are used to.  Use this information to better design the software to accommodate routine tasks and automate multi-step processes where possible.  
2. In addition to the basic system features, include features that make it easy for users to test their changes without applying them to a production system.  For GUI and batch apps, this is usually not a problem, but for web based and service based apps, this typically requires special consideration.  
3. Whenever possible, implement safeguards into your software to prevent user configuration problems from impacting the overall operation of the software.  For instance, plan to provide a way to automatically terminate user scripts/macros which never return.  
4. Document the steps users should take to verify and validate their work.  Often, these steps will involve using some software feature, but other times it may require involving a second person to look over their configuration and results.
5. Provide a means for users to report common problems, issues and concerns.  
6. Make configuration systems as visible as possible.  For example, if the configuration of a particular piece of software requires setting 20 different settings on various dialog panels, provide a report that lists each and every option.
7. Provide as much visibility as you can to running systems including the data involved, the configuration, performance information, etc.  You may want to restrict who has access to various information, but having visibility on this information can be critical to quickly isolating and recovering from problems when they occur.

Keeping up

I have talked about what to expect, habits we need to break, but how can we truly cope with the ever increasing feature lists, shorter schedules and greater exposure of our applications?  The standalone developer needs to be prepared to cope with these changing demands and needs to be able to stay on top of their game.  Some of the things I recommend include mastering tools of communications such as XML, mastering tools of presentation such as HTML, keeping an eye out for new tools of communications that can benefit organizations.  One additional thing which I believe is critical is that the standalone developer must find or develop a platform on which his/her applications can be built.  I will discuss my reasons for each of these items below.

Mastering XML

It is almost hypocritical of me to say this here, because I have not personally mastered XML and the related technologies, but I do intend to.  I believe that it is important for the standalone developer to master XML because we will be seeing it used more and more in everything from our databases to our presentation layers.  XML is a powerful, though often misunderstood, medium for transferring information between systems, people, departments, etc.  The old foggie in me says that there is nothing special about XML that sets it apart from just about any other text-based information deliver mechanism.   From a pure technical perspective this may be true, but the reality is that XML is quickly becoming the standard for data communications and will become ubiquitous throughout the software development process.

What is it about XML that makes it so important to the real-time enterprise?  XML provides the ability for new software systems, reports, etc to quickly consume data from other software systems.  This capability when mastered will make software development more efficient by eliminating many common data transformation and data definition tasks which are typically necessary when interconnecting systems.  Some developers are no doubt saying that the type of work they do does not deal with data much.  I would argue that even user preferences, configuration options and corporate policies can and will be represented through XML and our applications will need to understand these.

Having mastered the ability to read and interpret XML, we must turn our attention to leveraging XML within the software we develop.  The software we develop is becoming increasingly feature rich and interconnected, XML provides a means for sharing common data and functionality.  When designing new software, we need to consider how we can incorporate XML support to not only facilitate the application itself, but to facilitate this application exposing its data and services to other applications.  Which brings up an important aspect of how software is used in a real-time enterprise.  Instead of applications, real-time enterprises are looking for services.

Mastering presentation technologies

I doubt I need to say much about this.  We all know how important the internet and HTML have become to software development.  One thing which I believe we need to stay focused on, though, is the advancements being made in how the web is being used to present information to users.  Web browsers have advanced quite a bit from the early days of text only browsing and these advancements continue.  We need to be prepared to learn to use the new techniques, methods, languages, etc that are developed to keep up with our competition and to, when possible, stay ahead of the curve.  We need to stay current with trends in how user interfaces are being delivered to users and know when to use various methods of content delivery.

As an example, there have been many studies done over the years on the affect scrolling has on web users.  In the early years of the internet, these studies indicated that users did not tend to scroll and when scrolling was necessitated by a web site, users would often abandon the site.  The most recent information I have seen indicates that though scrolling is still a concern, users are more familiar with it and tend to accept it (except for horizontal scrolling).  Other areas of interest include page download times.  The last I checked, 5 seconds was the maximum page download time developers should target for dial-up users.  Other areas include the use of tabs (such as Amazon), DHTML for popup menus, right-click menus, multiple open windows, frames, etc.  

Beyond the current web browser experience, technology is providing more and more ways of communicating with our clients/users.  PDAs, cell phones, email and IM are all being used to deliver critical business data to clients.  New technologies will no doubt be developed which will revolutionize our ability to communicate with users and clients. I believe that we need to keep up with these advancements so that we can bring these technologies to our employers and enhance their communication systems.

Find or build a platform

I am not talking about building your own OS, nor am I talking about J2EE or .NET.  What I am talking about is having a code base from which you build many, it not all of your applications.  This is probably going to be more than just a library of classes and functions, and is probably going to be some type of architecture supported by code that encapsulates common functionality and exposes interfaces from which services can be built.  I recommend that this platform include data representation, data persistence, data validation, workflow management, some form of security mechanism and where appropriate presentation capabilities. 

Why is such a platform important?  Simply because it is the only way that I have found for a standalone developer to produce software with ever increasing feature sets and still maintain high quality of the overall product.  Having a platform means that you can implement features which are beyond simple application feature sets, but enhance not only the specific application, but enhance your ability to continue to deliver high quality results.  Features that I consider beyond basic application feature sets (for most applications) include:

1. Diagnostic logging that is both automated and can be supplemented with application specific messages.
2. Instrumentation of data flow.
3. Tracing of database activity.
4. Enhanced data persistence services such as change detection, change history, transactional management.
5. Pooling services to enhance scalability.
6. Auditing of user activity.
7. Expression evaluation.
8. Etc.

Building on a platform has many advantages which should not be ignored.  In fact, these advantages should be capitalized on whenever possible.  Some of the advantages that come to my mind first include:

1. Quicker time-to-deliver final product.
2. Prototypes can go away and be replaced with "limited edition" applications.
3. Easier to implement across-the-board enhancements.
4. Testing of new applications and features is typically simpler because many core features are pre-existing.
5. Testing of complex applications is typically simpler thanks to diagnostics logging, instrumentation, tracing, etc.
6. Developing more scalable solutions is easier.
7. Core functionality is typically used quite heavily and therefore tested quite thoroughly.
8. Get to work on application-specifics quicker.

Before proceeding, I should also issue a word of warning regarding platforms.  While platforms can be great things, they can also be a prison that prevents you from achieving desired results.  Platforms require more thought process and will require that the code used within the platform be thoroughly tested and engineered for maximum performance, scalability and memory footprint which are often conflicting requirements.  Platforms must be engineered to provide maximum benefit to applications using the platform while placing the fewest requirements on the design of that application.  Consider the Microsoft Foundation Classes (MFC) library.  This is an excellent framework for developing small to medium scale Windows GUI applications, but using MFC in a non-GUI application is often unwise and risky.  Also, MFC places certain burdens on applications which use it such as thread creation processing which can complicate code and break existing systems.  So, building your own platform on top of MFC would probably be a bad idea unless you were certain the only type of application you are going to develop is a small to medium scale Windows GUI.

One open-source example of such a platform is found right here on CP.  Check out Marc Clifton's AAL for some great ideas and techniques for building your own platform, or use his AAL as your own.. 

Conclusion

As you can see, the real-time enterprise is forcing changes in the way standalone developers work.  We can benefit from this trend by providing valuable services to our employers and to our industry.  We need to gain a deep understanding of our industry and of our employer and leverage this knowledge in the software we develop by fine tuning the application features to meet the unique needs of our employer/client/industry.  We must establish credibility for ourselves within our company or industry so that we can not only thrive today, but progress tomorrow.

I believe that most software developers have a desire to learn more and continually increase our skills.  Many of the consequences of working in a real-time enterprise will drive this desire by providing a constant need for progress.  Standalone developers are uniquely ready for this challenge because we are often in a position to make critical decisions regarding what technologies, tools, techniques we use to deliver results.  We simply need to make better informed decisions so that instead of creating more work for ourselves, we are creating systems and processes (and platforms) that we can leverage throughout our development process.