IT Systems Scorecard

Robert Kaplan and David Norton publicized the concept of a Balanced Scorecard for strategic management of a business in early 90s. The idea was to evaluate a business from multiple perspectives in addition to financial goals to have a holistic view of the organization’s ability to deliver on its vision. The Balanced Scorecard has since become a fertile area of management theory which continues to evolve in all sorts of specialized areas. The four perspectives of the balanced scorecard were financial, customer, internal processes and human resources.

Similar to the Balanced Scorecard as a concept which changed the approach towards performance management, John Zachman published a paper called Framework for Enterprise Architecture in 1989. It is commonly known as the Zachman Framework and it provides a two dimensional view into Information Technology. Over the last 2 decades the Zachman framework has also become a foundation for IT management theories and pieces of the framework have provided for numerous tools, methodologies and best practices. Today the term enterprise architecture refers to a holistic planning and management framework of all aspects of an information technology department. The Zachman Framework has become for IT what periodic table is for chemistry. Its visual representation follows a similar look and feel. A simplified view is presented below
Each cell is filled up with the relevant information at the intersection of a viewpoint with the information technology component. It will be important to note that this framework was designed in the mainframe era long before the multi-tiered web based systems became a norm in IT industry. The conceptual foundation of the framework remains intact even after 20 years but implementation details within each cell have been evolving with the innovation in technology.
The detailed explanation or overview of either the Balanced Scorecard or the Zachman Framework is beyond the scope of this article. which is concerned with developing a scorecard that allows the business owners and IT management to effectively assess the state of a system. Readers not familiar with these two concepts will not have any problems following the methodology being produced below to score or grade a system for effective management and planning of IT resources employed in building or maintaining an application system. This  scorecard inherits ideas both from the Balanced Scorecard and Zachman Framework.

Why a Scorecard
Let’s look at the problem first as to why an IT system would need a scorecard. A scorecard provides a point in time evaluation of the overall state of an IT application system. Similar to the need for a balanced scorecard where only the financial metrics were not enough and a holistic view was needed, the traditional IT systems metrics dealt with down time and maintenance issues. A more holistic view of the IT system which combines the development perspective, the business perspective and the infrastructure perspective was not available. The lack of such a holistic scorecard made it difficult for the application system to support the business in high priority areas of expansion, stability and compliance. It is therefore required that each system should go through a review and a scorecard developed. This will allow business to assess the readiness of the system to meet changing business needs as well as address cost pressures in tough economic times. The scorecard will allow management to track performance of the system and estimate its future stability from security, compliance, stability and ability to adapt to changing business needs. With frequent scorecards benchmarked, the IT management can continue to improve upon areas lacking an acceptable grade.

The scope of computing has broadened over the last 70 years. Initially, computing systems were used for heavy scientific calculations; the more famous ones being deciphers during the Second World War. Later on communication and record keeping became commercial drivers for computer systems. While there is no agreement on when exactly computing or scientific computing changed into Information Technology, it is safe to assume that mainframe systems when deployed in large corporations to run their business processes is when computing became IT. The score of the scorecard is limited to IT application systems which are built and deployed to automate business processes. All other systems related to scientific calculations, weather forecasting, risk modeling or archival applications are not covered through this scorecard.

Application System Scorecard
The scorecard is a snapshot of the state of the system along two dimensions, an IT dimension comprising Data, Function, Infrastructure and Development and a dimension of maturity comprising People, Process, Technology and Documentation. The cells at the intersection of the two dimensions are scored from 1 to 5. The following table depicts the scorecard template:

For each application system Data refers to its data structure including all the entities, relationships and attributes. Function refers to the business process that the system automates. Infrastructure is the environment including hardware and networking where the system is running while Development refers to the design and implementation code used to program the system. On the other dimension elements of quality are gauged on People, Processes, Technology and Documentation.

All the cells are explained below for their scoring schemes. The schemes explain the bottom (1) and the top end (5) of the score and leave the intermediate scales to the relative subjectivity of the evaluator.
Data – People: A score of 1 is where programmers built the structure as they moved along with coding with no database design specialist available. A score of 5 is where a trained Data Architect finished a holistic design of the entire system in the design stage of the development
Data – Process: A score of 1 is where no process exists to modify the data model. A score of 5 is where the process estimates the impact in terms of data migration, program code, performance and integration with other systems before a data model change is approved and version control is in place
Data – Technology: A score of 1 is where some obsolete database system is being used like Clipper, FoxPro or Dbase and/or no database security controls in place. A score of 5 is where a mainstream RDBMS with up-to-date version and patches is used with best practice security controls
Data – Documentation: A score of 1 is where no documentation is available and 5 is where a detailed data dictionary with logical and physical models is available
Function – People: A score of 1 is where no subject matter expertise is available for the business process and 5 is where a senior business analyst well versed in all aspects of the business process is available being able to address audit, security and integration requirements in addition to functional requirements
Function – Process: A score of 1 is where no change management process is in place while a 5 is where extensive pre and post analysis as well as testing takes place along with version control
Function – Technology: A score of 1 is where obsolete front-end technology is utilized and or web based GUI is not available. A score of 5 is where a robust thin client user front-end is available and changing the look and feel does not require complex code changes
Function – Documentation: A score of 1 is where no documentation on the business process is available while a score of 5 is where detailed process flows and data flows are available
Infrastructure – People: A score of 1 is where specialized roles and skills are not available for networking, system administration and DBA activities. A score of 5 is where certified staff in specialized skills is available
Infrastructure – Process: A score of 1 is where regular maintenance for back-up and patches is not performed and BCP and DR plans are not in place. A score of 5 is where a standard like ISO or ITIL is in place
Infrastructure – Technology: A score of 1 is where obsolete and vulnerable networking and storage systems run the application. A score of 5 is where modern hardware like SAN storage, blade servers and networking appliances are in place for the application system
Infrastructure – Documentation: A score of 1 is where no documentation of the equipment is available like models, vendors, specifications, etc. A score of 5 is when a hardware asset management system is in place along with BCP and DR plans and documentation on infrastructure design
Development – People: A score of 1 is where developers are not trained in any kind of SDLC and coding best practices. A score of 5 is where staff is certified in programming and design techniques
Development – Process: A score of 1 is where no change management or any SDLC is in place. A score of 5 is where the system development is governed by established processes like PRINCE or CMMI
Development – Technology: A score of 1 is where obsolete programming language or environment is being used. A score of 5 is where Object Oriented or multi-tiered development environment, reusable components and testing tools are available to developers
Development – Documentation: A score of 1 is where no formal documentation is in place while a score of 5 required detailed specification and testing plans available in a use-case like format

An enterprise architecture group outside of the development and infrastructure teams should be responsible for scoring the application systems. The reports of the score should be submitted to the upper management. A matrix filled up with numbers with detailed descriptions below is a simple report that can be used to assign targets for the next review cycle. These reports over time will also help address the build Vs buy decisions as well as outsourcing decisions. Hiring can also be influenced looking at the areas with weakness across multiple systems. Overall, a periodic health check using this scorecard provides an apple to apple comparison between teams, management and service providers to better manage an IT department with improved quality and lower costs. A history of periodic scorecards on application systems also raise flags early enough in the process when strategic shifts in business are going to force IT changes. Similarly, technology risk, compliance and readiness of certain business processes to take on a merger/acquisition or major business change can be pro-actively managed.