|
By Steven Salterio & Alan Webb
Illustration: John Sapsford
RHETORIC VERSUS RESEARCH: A PROPERLY IMPLEMENTED
PROGRAM CAN HELP ARTICULATE AND COMMUNICATE STRATEGY
In the early 1990s, Harvard's Robert Kaplan and his consulting partner, David Norton, developed the balanced scorecard to broaden the focus of managers from traditional financial measures to a more diverse set of measures, including nonfinancial ones. Its appeal is so strong that some estimate 50% of Fortune 1000 firms are using this scorecard in some form or another. Canadian companies that have adopted the scorecard include AT&T Canada, Bank of Montreal, Best Foods Canada, BC Hydro, Nova Scotia Power and Nova Chemical.
Based on a firm's overall strategy, the scorecard typically contains a diverse set of 16 to 28 measures, commonly organized into four categories: financial performance, customer relations, internal business processes, and the organization's learning and growth activities. Kaplan and Norton provide guidance on measures to use in each scorecard category. (Examples are shown in figure 1 above.) Financial measures include such traditional general measures as return on sales or investment but should also relate to the business unit's strategic objectives. Customer measures should be based on the value propositions to be delivered to the targeted customers. Internal business process measures relate to the operational processes of the business unit. Learning and growth measures should be related to employee capabilities, information systems capabilities, and employee motivation and empowerment. A key feature of the scorecard is that the measures are intended to reflect an integrated set of cause-and-effect relations between outcomes (lag measures) and the critical drivers (lead measures) of those outcomes. The resources required to implement and maintain a scorecard can be significant, but Kaplan and Norton argue the approach establishes a frame-work that can be used in satisfying several managerial needs (see table 1 on following page).
Scorecard literature is replete with testimonials from satisfied users and consultants, suggesting many of these benefits are being realized. However, because of concerns regarding the objectivity of these claims, we will evaluate each of the four purported scorecard benefits based on what academic research has discovered. Before we consider the evidence, we evaluate whether there is any theoretical basis that would suggest the scorecard should work.
The concept of multiple-measure systems such as the scorecard is generally supported by accounting theoretical models that suggest measures should be added to a performance measurement system if they increase the likelihood that desired managerial goals, such as increased shareholder value, would be achieved. Other models support the use of nonfinancial performance measures since they may be more sensitive to changes in managerial effort than the traditional financial measures. psychology-based research funded by the Canadian Academic Accounting Association's research grants program has shown that the same set of performance measures, one set organized in the scorecard format (four categories, four to seven measures each) and the other set organized at random, or in alphabetical order, lead to systematic differences in managers' judgments. Thus, the mere process of categorizing the diverse performance measures appears to change managers' judgments; however, there are no criteria to distinguish which judgment is better.
While theoretical research generally supports the use of multiple performance measures organized into categories, it does not specify the number, the composition or the content of the categories. Now let's examine the academic evidence related to each claimed scorecard benefit.
Clarify, translate, communicate
According to Kaplan and Norton, the implementation and rollout of a scorecard can communicate and clarify to employees key strategic objectives and their critical drivers. This claim is important because research shows that effective communication of strategy can have an impact on the success of strategy implementation. Evidence regarding the purported strategy clarification and communication benefits of the scorecard is mixed. A recent institute of management Accountants survey on performance management indicates that the scorecard can be an effective strategy communication and clarification tool. Compared to non-scorecard users, survey respondents from companies using a scorecard rated their performance measurement system considerably higher as a means of communicating strategy. Consistent with the IMA data, a case study of a Fortune 500 company reports that the scorecard was an effective means of communicating its strategy. Contradictory evidence is reported in a case study of a financial services organization where adoption of the scorecard had a negligible impact on branch managers' understanding of strategic objectives and goals.
Link strategic objectives and measures
Kaplan and Norton say a critical component of establishing linkages between strategic objectives and the scorecard performance measures is the identification of the cause-and-effect relations between outcomes on lag measures, and the critical lead indicators of those outcomes. In the follow-up to their original book, they present numerous examples of successful scorecard implementations suggesting a correlation between outcomes on lead and lag performance measures.
Several academic studies have directly examined the significance of the correlation between scorecard lead and lag indicators of performance, but to date have focused on pairs of performance measures. One of the first studies shows that at Sears, employee attitudes have a positive impact on customer satisfaction (both lead indicators), which in turn has a significant effect on revenue growth (lag indicator). Similarly, another case study, using data from the branch locations of a financial services organization, reports that the customer satisfaction measure used in the scorecard has a positive association with future period revenue and profit margins. Finally, interview data reported in a recent case study of a Fortune 500 company indicates managers believe the cause-effect relations included in their scorecard have led to improved efficiency and profitability.
Research has begun to examine managers' ability to predict the future operating performance of a business unit using relations between lead and lag indicators. One finding of this research is that the accuracy of individuals' predictions of future profit (lag indicator) is greater when the quality performance (lead indicator) is expressed in nonfinancial (e.g., percent defects) instead of financial (e.g., cost of re-work) terms. This is relevant to the scorecard setting given that performance on many lead indicators can be measured in either financial or nonfinancial terms (e.g., training expense per employee versus training hours per employee).
The same study suggests that an individual's ability to assess the lagged impact of current period quality-related spending on future profit is greater when nonfinancial data (product quality, defect rates) reflecting the shorter-term consequences is also provided. The evidence also suggests that the provision of the nonfinancial data clarifies the lead-lag relation between quality spending and future profit. This finding is relevant to scorecard users and designers as it indicates that inclusion of nonfinancial performance measures, which clarify lead-lag relations between other key measures, can have beneficial effects on managerial decision-making.
Plan, set targets and align initiatives
The success of planning, target-setting and aligning performance measures to strategic initiatives often depends on whether the managerial performance evaluation system directs managers' attention to these areas. The frequently cited truism "what gets measured gets managed" needs to be amended to "what gets measured and used in evaluations gets managed." Both theoretical and empirical research has established that measures not attended to in performance evaluation are unlikely to receive serious managerial consideration. Kaplan and Norton's advice, which is consistent with theory, is that managers' performance compensation should be based on the scorecard. Implementation of this advice in practice has led to two main approaches, a subjective evaluation approach with no explicit weight on any category or measure and a formula-driven evaluation approach with explicit weights on each measure.
Researchers have extensively examined the subjective performance evaluation effects. A widely replicated finding from a Canadian Academic Accounting Association funded study is that managers evaluate their subordinates on measures that are common to all their scorecards but ignore the measures on those scorecards that uniquely reflect the critical drivers of the individual division's success. This has been called the common measures bias. Since the measures that are common across divisions are most frequently financial in nature, the bias may suggest an undue subjective weight is being placed on financial performance measures, which is what the scorecard was designed to avoid.
Despite attempts by several teams of researchers, only a study funded by the CICA's academic research fund has succeeded in reducing this bias by either having the manager account for his or her performance evaluations directly to his or her superior, or by providing the manager with an assurance report over the reliability of the measures on the unit's scorecard.
Formula-based scorecard evaluations also have their challenges. Case studies have shown that companies struggle to determine the right measures to reward, the explicit weight to put on any one measure or category of measures, and the potential dysfunctional outcomes that can result from basing rewards on outcomes of only a subset of the measures, not some minimal level on each. Hence, while formula-based evaluations are able to deal with some of the problems of subjective scorecard-based evaluations, they are not a panacea.
Enhance strategic feedback and learning
The provision of feedback as to whether the strategic objectives are being accomplished is one of the scorecard's most important benefits, according to Kaplan and Norton. By monitoring whether performance on the critical lead measures is having the expected consequences on key lag measures, managers are able to evaluate whether the cause-effect relations are valid and whether or not the strategic objectives are achievable.
The only academic study we know of that addresses this claim uses scorecard data from a convenience-store chain to evaluate which of two strategies simultaneously implemented and measured is more effective in increasing profit. The case study shows the critical drivers identified by management for one of the strategies did not in fact have any impact on financial performance. Management interpreted this finding as suggesting the cost of this strategy did not generate any benefits and hence this strategy should be dropped.
This evidence suggests that the scorecard can be used as a means of monitoring the success of an organization in achieving its strategic objectives. However, this study is rare given the fact that the scorecard's measures in practice are being continually refined, improved or dropped for some reason or another.
While we could close on the safe note of "more research is required" (which it is), we are a bit bolder in our conclusions (see table 2 on previous page for our report card on the scorecard). While based on limited research, our tentative conclusion is that a properly implemented and consistently employed scorecard may aid some organizations in better articulating and communicating their strategy, measuring the drivers of their performance and detecting the superiority of one strategy over another.
Academic research has also detected problems that should dampen some of the enthusiastic claims made by proponents. Concerns over linking the scorecard to performance evaluation and hence to compensation seem just as challenging as with any other performance-based compensation system.
Concerns also exist over the practice of adding and deleting scorecard measures as better and/or more strategically relevant measures become available or as existing measures are deemed to be soft and/or unreliable. The practice of frequently adding and deleting measures makes it difficult to detect the cause-effect relationships at the heart of the scorecard proponents' claims. Overall, like any other managerial tool, the balanced scorecard has good features and not so good features. It is up to accountants within organizations to ensure that if a scorecard is implemented, the benefits will be maximized and the limitations recognized when making decisions based on scorecard information.
Further reading:
R. Kaplan and D. Norton. 2001. "The Strategy-Focused Organization: How Balanced Scorecard Companies Thrive in the New Business Environment." Boston, MA: Harvard Business School Press.
T. Libby, S. Salterio and A. Webb. 2003. "The Balanced Scorecard: The Effects of Assurance and Process Accountability on Managerial Judgment." Working paper, University of Waterloo and Wilfrid Laurier University, available at SSRN http://papers.ssrn.com/sol3/papers.cfm?abstract_id=317486.
M. Lipe and S. Salterio. 2000. "The Balanced Scorecard: Judgmental Effects of Common and Unique Performance Measures." The Accounting Review 75 (3): 283-298.
M. Lipe and S. Salterio. 2002. "A Note on the Judgmental Effects of the Balanced Scorecard's Information Organization." Accounting, Organizations and Society 27(6): 531-540.
A.J. Rucci, S.P. Kirn and R.T. Quinn. 1998. "The Employee Customer-Profit Chain at Sears." Harvard Business Review: 83-97.
The Balanced Scorecard Collaborative Inc. at www.bscol.com
Steven Salterio, PhD, CA, is a professor of business at Queen's University School of Business. He was a visiting professor at the Richard Ivey School of Business when this article was written. He has carried out extensive research on the balanced scorecard, corporate governance and management-auditor negotiations over accounting issues
Alan Webb, PhD, CA, is an assistant professor of accounting at the University of Waterloo's School of Accountancy. He has conducted research on the balanced scorecard, budget accuracy issues and management-auditor negotiations
Technical Editor: Michel Magnan, PhD, FCA, Lawrence Bloomberg Chair of Accountancy, John Molson School of Business university associate, Raymond Chabot Grant Thornton LLP | 
