This book follows the usual pattern of the Wiley Series in Information Systems, with an introduction by the editors followed by four sections of contributions on major aspects of the debate. The contributions have been well chosen to give a breadth of appreciation of the topic of evaluating IT investments. The contributions are complementary with relatively little overlap or redundancy, and the editors have clearly made an effort to make links between chapters where appropriate.
The overall approach is strongly academic, which does not lead to easy reading but does provide an authoritative resource for the researcher in this area, including rigorous literature referencing. There is a lot of material, and I would recommend readers to take a chapter at a time to avoid overload!
The introduction by Willcocks and Lester provides an excellent overview to the themes of the book and puts the debate into perspective. They have used the “IT productivity paradox” as an introduction to the theory, practice and applications of IS evaluation. IT investment is going on not in a closed laboratory but in a complex and turbulent system of organizational activity. Further, the contribution of IT is not always tangible, and may follow investment with a time‐lag and non‐linearly. Finally, the important point is made that IT is unable to add value itself, but does so only in interaction with other factors.
Section 1 discusses the history of the IT productivity paradox at the macro level. Brynjolfsson and Hitt discuss how to measure productivity using economic models. Using these models they suggest that the productivity paradox disappeared after 1986, and that IT is now starting to deliver benefits. Their argument is not wholly convincing however, since it includes changes to method and data source as well as critical assumptions about economic indices and the diligence of respondents. Their most recent data is sourced from 1987‐1991, so the analysis is already nearly a decade out of date. They conclude that the main benefit from the study is a greater understanding of the relationship between IT and productivity, to which undoubtedly they have made a significant contribution. Willcocks and Lester critique the macro‐economic approach and suggest that results are gained and lessons are learned at the micro (firm) level. They suggest that productivity gains are earned from investing where IT can make the most difference in terms of business value. They put forward a simple conceptual model for the IT/IS evaluation and management cycle, and discuss a range of possible benefits and corresponding measures that management might choose to evaluate their investments. van Nievelt takes up the theme of benchmarking organisational and IT performance. He contributes an interesting set of functional representations that demonstrate the importance of choosing appropriate objectives for investment in IT and plotting paths toward these objectives using contour maps of the functions. He argues that customer satisfaction is a key measure of the return on IT investment, and demonstrates how investment in IT could be counter‐productive, for example because customers will not appreciate IT investments if they are not satisfied with current business performance.
Section 2 discusses more detailed techniques and approaches to IT investment appraisal. Ballantine, Galliers and Stray present a well‐written contextual and methodological approach to an investigation about management processes with respect to evaluation. Survey respondents indicate that they adopt a pragmatic approach to evaluation that depends on the relative size, value and risk of the project as well as on operational time constraints. They report a lack of formal evaluation procedures especially on larger projects. Powell neatly opens up the debate by identifying a number of reasons why organisations might rationally invest in IT despite no formal evaluation. He draws broad distinctions between objective and subjective methods, between ex ante vs ex‐post evaluation. He echoes the assertion (quoting Feeny) that “only the adoption of new business ideas can lead to business benefits”, IT alone being insufficient. He also exposes myths of evaluation practice, for example where bureaucracies manipulate figures to satisfy formal evaluation criteria that might not be felt appropriate by some actors. He recognises that formal evaluation is not a substitute for users’ final judgements, that the process of evaluation can often be more useful than the outcome and that evaluation is to a significant degree a learning process. “The important thing in business is not to make good forecasts, but to make them come true.”
Farbey, Land and Targett discuss the issue of a range of evaluation techniques and when to apply them, identifying a number of dimensions to the problem. They have reservations about the role of traditional evaluation, since some projects which have provided the best returns in term of competitive advantage would not have satisfied the prevailing ROI criteria. Like Powell, they explore rational reasons why projects are not evaluated, and offer a useful overview of the different methods’ roles and purposes. They develop a useful summary contingency matrix based on Earl and Hopwood for identifying suitable evaluation approaches; this could be particularly useful to practitioners. Venkatraman is concerned that current planning and resource allocation systems and criteria are no longer appropriate given recent changes in technology, business and markets. He proposes two dimensions on which to base the organization’s view of IT: first, the purpose of IT, whether to enhance business capability or increase operational efficiency and second the risk propensity of the firm. Together these dimensions determine an appropriate view IT between a cost centre, a service centre, an investment centre and a profit centre, a view which helps to determine an appropriate evaluation approach. The latter part of this chapter applies and reinforces this “value centre” concept.
Section 3 covers development, sourcing and infrastructure issues. Janne Ropponen contributes the (only) wholly practitioner chapter of the book and focuses on software project risk management as a means to increase leverage from IT investment. He uses Boehm’s top ten risk items, and finds that 75 per cent of project managers do not follow any detailed risk management approach, even they are engaged in risk management activities. He warns managers against merely following “gut feeling” and hoping for good luck. Graeser and Willcocks introduce a different research approach using a public sector case study from the California Franchise Tax Board, providing considerable detail. The story identifies an effective practice for inviting and evaluating bids for outsourced work, namely that vendors be invited to share the identified risks. This has the effect of aligning vendors’ objectives with the company, at a price that is argued to be well worth paying. Willcocks, Fitzgerald and Lacity introduce the theme of outsourcing into the debate. This contribution is a clear overlap with other books in the Wiley series, but on balance it is worth including, based on the argument that consideration of IT outsourcing can be a catalyst for forcing organizations to enhance (and first evaluate!) their evaluation practices. For example, hidden costs is a particular issue that is exposed by outsourcing and is relevant to IT evaluation. Further, the outsourcing decision cannot sensibly be made without first having an idea of current internal performance. Weill and Broadbent discuss four views of IT infrastructure. They argue that infrastructure, although perhaps difficult to justify through traditional evaluation, has the important objective of providing a stable base of reliable services and applications. They suggest that the view of infrastructure (“none”, “utility”, “dependent” and “enabling”) will determine how investments are evaluated, using a case to illustrate each view.
The final section “Towards interpretive approaches” emphasizes a social, political and stakeholder view of evaluation. Walsham argues that evaluation should be considered a dynamic, intersubjective and interpretive process of “sense‐making” about a project. He emphasizes the role of power, whereby different stakeholders may shape the evaluation process and whereby the whole evaluation may either sustain or question the existing power base. He reinforces the notion of evaluation as a learning process, and as a way of generating understanding, involvement and commitment to the project and also that sometimes the process is merely a ritual. He does not specify a set method for evaluation; rather he describes a broad outlook on the process, and suggests that interpretive evaluation should be experimented with more widely. Hirschheim and Smithson claim that an overly rational simplistic notion of evaluation can lead to unintended and possibly dysfunctional consequences. They express concern that evaluation has seen a concentration of the means to the detriment of the ends. They categorize the different purposes of evaluation as efficiency, effectiveness and understanding. Understanding of evaluation as a learning mechanism for an intersubjective appreciation of a situation is seen as the ultimate interest in evaluation:
With such a wide range of evaluation techniques and approaches available and with practical pressures demanding meaningful evaluations of operational systems, practitioners would clearly gain from advice on which techniques fit particular situations. There is no ‘one best method’ suitable for all situations.
As this quotation suggests, practitioners are likely be more satisfied with the contributions in the book that offer positive practical pointers to going about evaluation such as that provided by Farbey, Land and Targett. Several additional practical insights are present in the book however, and some have been noted above, but these hints will need to be carefully sifted and applied by the practitioner reader. As usual for academic works, the book does not provide the answers, only the questions, and several of the writers have acknowledged the important contribution of practitioners in testing out some of the new ideas. Apparently, some practitioners already have enough maturity to resist evaluation completely where it is an inappropriate investment of time and effort.
I feel that although the book does not dismiss the practical need for particular evaluation approaches, it diminishes the credibility of evaluation practice perhaps more than it intended to. By exposing the limitations of traditional evaluation practice from all sides it must make the reader question the whole emphasis and value of evaluation. However it is to be hoped that this will be a healthy challenge to prevailing practice and that it will lead to more informed and effective evaluation. As an authoritative and integrated resource for the teacher, student and researcher, the book’s influence will also be felt in the ongoing academic debate about evaluation. This dual role justifies its place on your reading list.
