Monday, June 26, 2006

Requirements - a False Sense of Security

The Challenge

In an industry with over 35 years of software engineering experience, it's amazing that we have failed to produce any significant breakthrough that improves the ability of executing successful software projects. The brightest minds have been humbled battling the tide of anarchy that persistently undermines the progress of their complex initiatives.

"Discipline and control", the gurus cry, will help curb our irresponsible creative surges. Envying their manufacturing counterparts, knowledge workers have reached out to embrace disciplines that have reduced the insanity while simultaneously eroding innovation and creativity beneath the weight of bureaucratic process and fragmented work distribution. In this protected environment, the "sigh of relief" eventually evaporates into a new level of panic, as emerging competitors run roughshod across existing markets with disruptive innovations.

Successful assembly and delivery of complex artifacts is ultimately driven by microeconomics. In the software industry, the emerging market value has become increasingly complex to analyze and the capability constructs similarly growing in complexity and interdependence. It is into this intersection of complexity and discipline that requirements engineering has been born, and to some degree helped extend the coping lifespan of adopting organizations.

Systems Architects have long used the practice of abstraction and information hiding to manage the complexity of large-scale implementations. Requirements engineering is an attempt to leverage these disciplines to articulate the market needs or ultimately value opportunities in a manageable, traceable fashion. It is a formal attempt to align opportunity value with construction cost. However, trying to reduce the interaction with the opportunity domain to a clear-cut API in the form of well constructed requirements is failing for several reasons:
  • Requirements create a dysfunctional buffer between the value context and the construction capabilities. This cripples the ability of creative problem solvers to contribute except at the highest levels of the process.
  • Requirements fragment the value context in an attempt to facilitate a work break-down and scope determination. The risk of value dilution increases often resulting in unnecessarily complex or irrelevant implementations.
  • Requirements are underpinned by a contract based process, built on a philosophy of control and protectionism. Companies are crippled through an inflated time to market and the denial of using early corrective value validation, placing too much faith on the ability of the requirements process to communicate value effectively.
The intent of the organization is to profit from the intersection of value opportunity and contribution capability. Requirements convert value opportunities into non-negotiable statements, which it presumes can be fulfilled by the rigorous application of predefined capabilities.

The Solution

The journey out of this dilemma, is both simple and difficult. It requires a new or transformed set of skills and a change of corporate mindset. First let's focus on the activities needed for success:
  • Clearly understand and communicate opportunity value to all contributing levels of the organization.
  • Effectively envision and validate capability early and often.
Organizations have used the following methods to break the stranglehold and generate a new capacity for accelerated value innovation:
  1. Powerfully push context as deep as possible into the design and construction domains.
  2. Rapidly and iteratively test design possibilities with the opportunity domain stakeholders.
Tools and practices for driving context deep into the organization include: stakeholder analysis; rapid ethnography and persona development; context diagrams; business process diagrams and storytelling.

Tools and practices for connecting design possibilities include: early design prototyping; iterative design checkpoints and feedback; agile construction practices and multi-disciplinary design teams.

Companies like IDEO and Google have demonstrated the success of these approaches. We too need to relinquish our safety nets and grasp the new paradigms of innovation for success in the emerging market. As Gary Hamel, considered by some as “the world’s leading expert on business strategy” states, "Innovation is the only cure for the debilitating hyper-competition that drives margins ever downward."


Navid Sadikali said...

I would add one critical piece, which is that many of the "requirements" only exist once the "thing" is built. For example, a crowded remote control with many features on it's face might once constructed, not be readily understandable. Therefore modifications are needed to improve usability, this may mean a requirement to "remove lesser used buttons." Further, once the person begins interacting, the interaction with the set of buttons created may lead to unexpected user behaviors or errors. One such error would be hitting the power on the VCR and TV, and then the "channel up and down button intermittently on the TV does nothing."

Therefore, in the world of interactive software the complexity of software states coupled with variable user behavior and psychology intensifies the need to let dedicated resources specify "what the product should do."

HTL said...

I agree. I think another way of saying this is that early effective design prototyping (in conjunction with proper user analysis and ethnographical study) can uncover value opportunities and highlight effective design decisions that are difficult or impossible to specify beforehand. This is particularly pertinent since most software is not created but transformed from previous implementations. Much can be learned about the users from observing their current usage of the existing implementation. In other words it is a matter of semantics to say that the design creates the requirement or that the design exposes the requirement (which existed as a possible, unarticulated requirement not yet apparent).