PART I. THEORETICAL FRAMEWORK DEVELOPMENT OF PERFORMANCE MANAGEMENT
SECTION 2. P ERFORMANCE MANAGEMENT SYSTEMS IN OECD COUNTRIES : V ARIABLE DESCRIPTION
1993; Pollitt and Bouckaert, 2004). As a result, it is possible to argue that it is the decision-making process – particularly in the PDCA cycle – instead of the political regime that influences the implementation of an effective performance management system, which is further confirmed by the OECD data.
Section 2. Performance management systems in OECD countries: Variable description
performance and accrual (cost) information. Constructs are used in instances when the concepts of interest cannot be directly observed. For example, as the OECD database does not provide information on cost, this research considered the budgeting basis, accounting basis, separation between operating and capital budget, internal charges (for capital assets and goods/services), multi-year estimate and budget flexibility as variables to construct cost information. In addition, these constructs are particularly useful when it comes to capturing and representing concepts that are multidimensional. Some variables are used to measure underlying factors or constructs not directly measured in the questionnaires. As a result, some variables are categorized as both dependent and independent variables.
The variables in the questionnaires are measured in nominal and ordinal levels of measurement using the Likert scale. Nominal measurement assigned numbers or other symbols to a set of categories for the purpose of naming, labeling or classifying the observations, while ordinal measurement assigned rank-ordered categories ranging from low to high (Nachmias and Guerrero, 2006). As will be discussed later in this chapter, these underlying factors were identified through factor analysis, and then used in a Chi-square test and multiple regression analysis to test the hypotheses and to develop a relational and causal linkage model of performance management system in distributed governance. Lambda is calculated to measure a symmetrical association between nominal variables, while gamma is calculated to measure a symmetrical association of ordinal variables or with dichotomous nominal variables (Nachmias and Guerrero, 2006).
Table 4.4 Description of independent variables
Construct measure: Performance measurement system Type Question Number
Performance target Nominal OECD (VI.75)
Output Nominal OECD (VI.73)
Outcome Nominal OECD (VI.73)
Evaluation report Nominal OECD (VI.71)
Benchmarking Nominal OECD (VI.71)
Construct measure: Cost information (Accrual concept)
Budgeting basis Nominal OECD (V.62)
Accounting basis Nominal OECD (V.62)
Categorization of operating and capital expenditure Nominal OECD (II.27) Internal charge for capital spending* Ordinal OECD (IV.50) Internal charge for goods/services* Ordinal OECD (V.64)
Multi year estimate* Nominal OECD (II.16)
Flexible budget* Nominal OECD (IV.51, 52, 53)
Construct measure: the link between expenditure (cost) and performance information
The link between expenditure (cost) and performance information Ordinal OECD (VI.75) The link between expenditure (cost) and performance goals/objective Ordinal OECD (VI.76) Source: Developed by author based on OECD and OBI questionnaires.
Note: * The variable is categorized as both dependent and independent.
2.1. Independent variables
2.1.1. Performance measurement system
More than 70% of OECD countries have implemented a well-developed performance measurement system built on a framework of performance targets, measure and evaluation (Table 4.5.). The OECD survey results reveal there are more countries that implement performance evaluations than those that implement performance measures and targets. This indicates that the performance management systems in OECD countries are mainly driven by the feedback from performance evaluations. It is possible to argue that performance evaluation is more flexible because it is possible to define it in a more loosely form and link to the objective. It is relatively difficult to develop performance measures and targets because they consist of specific indicators, which should be linked to the objectives.
Table 4.5 Types of performance information in OECD countries
Types of performance information No. of country %
None 2 6.7%
Performance targets 21 70%
Performance measures 24 80%
Evaluation reports (e.g. programs, sectoral, efficiency, or cost effectiveness reviews) 25 83.3%
Benchmarking 9 30%
Other 3 10%
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
2.1.2. The accrual accounting system and cost information
The survey reveals diverse cost concepts in OECD countries. More specifically, 27% of OECD countries have implemented and utilized the full cost concept as they implemented accrual-based budgeting and accounting. While the rest of the countries still implement a combination of cash and accrual basis, 27% of OECD countries that implements cash-based budgeting and accounting have separate operational and capital budgets, indicating the availability of cost information.
Table 4.6 Reporting basis
Reporting basis Budget
Cash Accrual
No. of country % No. of country %
Accounting Cash 16 (8) 53% (27%*) 0 0%
Accrual 6 (0) 20% (0%*) 8 27%
Total 22 73% 8 27%
No. of country %
Single budget 14 46.7 %
Capital and operating budgets are separated (dual budget) 13 43.3 %
Other 3 10 %
Total 30 100%
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
Note: * 27% separate operational and capital expenditures.
Internal charges imposed on agencies for using capital assets and delivering goods and services indicate the availability of cost information. However, the OECD survey confirmed the limited practice of internal charges in the majority of OECD countries.
Table 4.7 Internal charges on capital assets
Is any charge imposed on agencies/executive organizations for using capital assets (e.g. use of a government owned building)
No. of country
%
No capital charge is used 15 50%
Yes, a general capital charge is used 6 20%
Yes, a capital charge is used in certain cases 9 30%
Total 30 100%
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
Table 4.8 Internal charges on goods and services
Is there a system to charge a price for goods and services provided by one government organization to another?
No. of country
%
No 6 20%
Yes, but it is used only to a minor extent 6 20%
Yes, it is used to some extent 11 37%
Yes, it is used to a great extent 6 20%
Other 1 3%
Total 30 100%
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
2.1.2. The link between expenditures (cost) and performance information
Most OECD countries prepare performance information and adopt some aspect of the accrual concept in the performance management to generate and link cost information to performance information in order to justify the performance level and to measure efficiency/effectiveness.
However, as the implementation level of the cost concept in OECD countries still varies, it is difficult to link the performance with cost information in most OECD countries. As a result, most OECD countries link performance information to (line item) expenditures instead of cost.
The OECD survey confirmed the diverse linkage level between (line item) expenditures and performance targets/goals/objectives, ranging from none to full implementation (Table 4.9).
Table 4.9 The link between expenditures and performance targets/goals/objectives
Level Linking expenditure with
Performance target performance goals/ objective No. of country % No. of country %
None 6 20% 7 23.3%
1-20% 2 6.7% 5 16.7%
21-40% 2 6.7% - -
41-60% 3 10% 1 3.3%
61-80% 1 3.3% 1 3.3%
81-100% 6 20% 7 23.3%
Other 10 33.3% 9 30%
Total 30 100% 30 100%
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
OECD survey showed a relatively low level of integration between performance reports with main budget documents (performance based budgeting) and financial statements (Table 4.12b).
Nearly 50% of OECD countries publish their performance reports for each line ministry as the responsible entity in performance target setting, achievement and evaluation. However, only 23.3% integrate the performance reports with the main budget or annual financial statement, which is similar to the number of countries that link expenditures with performance targets/goals/objectives. This observation indicates the countries that integrate performance reports to the main budget or annual financial statements link expenditures with performance targets/goals/objectives. On the other hand, the number of OECD countries that prepare performance reports as accompanying documents to the budget (20%) is similar to those that link the expenditures with performance targets/goals/objectives to some extent. The number of OECD countries that presents the performance reports in the government-wide report – not as an accompanying document to the budget – on an ad hoc basis and do not prepare any performance reports is approximately close to those countries that do not link expenditures with performance targets/goals/objectives. As a result, it is possible to conclude that the link between expenditures with performance targets/goals/objectives influences the integration of performance reports into the budgeting and accountability process.
Table 4.10 Are performance results made available to the public?
No. of country % Yes, government-wide performance report is published 7 23.3 % Yes, individual ministries publish performance report 13 43.3 %
Yes, as part of government-wide documents 5 16.7%
Yes, as part of ministry-specific document 9 30%
There is an internet site for this information 8 26.7
No 3 10 %
Other 5 16.7
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
Table 4.11 Are performance reports routinely presented to the legislature?
No. of country % Each ministry prepare performance reports accompanying the budget 6 20%
Integrated into the main budget documents 4 13.3%
Integrated into the annual financial documents 3 10%
Presented in government wide report not accompanying the budget 2 6.7%
Ad Hoc 3 10%
No 2 6.7%
Others 10 33.3%
Total 30 100%
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
In order for the new mechanism of policy making and service delivery under the distributed and networked governance structure to work properly, the strong role of line ministries should be regulated under the quasi-contractual relation (link between performance and accrual (cost) information) and balanced by stakeholder management and involvement.
2.2. Dependent variables 2.2.1. Quasi-contractual relation
The quasi-contractual relation is indicated by the link between expenditures (cost) and performance information in planning, budgeting and reporting. From the perspective of service delivery partners, the existence of a quasi-contractual relation is indicated by lump sum appropriation, as the autonomy should be regulated under the link between expenditures (cost) and performance targets/goals/objectives. Agencies in 30% of OECD countries receive an appropriation that specifies expenditures below the agencies’ level, thus indicating a hierarchical and non-contractual relationship between central governments and agencies. The agencies in another 30% of OECD countries have received lump sump appropriation, indicating the autonomy levels of agencies within contractual relationship with central governments. The rest could not be categorized in either category, which highlights the scattered level of quasi-contractual relationships in OECD countries. It is possible to argue that the ongoing process in OECD countries to define, implement, and utilize cost information and its link with performance information in the PDCA cycle has contributed to this outcome.
Table 4.12 Lump sum appropriation
Do your agencies/executive organizations receive lump sum appropriations? No. of country % No, each agency/executive agencies receives an appropriation that specifies
expenditure below the agency level
9 30%
Yes, each agency/executive organization receives a lump sum appropriation for operating expenditures only, without sub-limits
4 13.3%
Yes, each agency/executive organization receives a lump sum appropriation for operating expenditures only, with a sub-limit on wages
2 6.7%
Yes, each agency/executive organization receives a lump sum appropriation covering both operating and capital expenditures, without sub-limits
3 10%
Yes, each agency/executive organization receives a lump sum appropriation covering both operating and capital expenditures, with a sub-limit on wages
5 16.7%
Other 7 23.3%
Total 30 100%
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
In summary, it is possible to argue the relatively low level of quasi-contractual relationships between central governments and service delivery partners in OECD countries, which is caused by the ambiguity of cost information and its link with performance information. The weight given to performance and cost information will depend on the quality of the information, which is not only about the reliability but also the relevance of information to the users in decision-making processes.
2.2.2. Stakeholder management and involvement
The anticipated benefits of performance and accrual (cost) information must be considered under the context of the stakeholder role in decision-making processes. The stakeholders should
also be given the opportunity to evaluate performance. To enable the public to have confidence that performance measurements are properly undertaken, certain measures have been introduced.
The disclosure of performance and accounting information to stakeholders would invite stakeholder involvement in the decision-making process. Performance reports and technical notes, giving details on how actual performance against targets would be measured, in addition to accounting techniques to reconcile cash and accrual numbers to improve stakeholders’
understanding and invite their participation were introduced in some OECD countries, for example in the United Kingdom. Such measures could overcome increased concerns about the implications of indirect accountability and control and reduce the reliance on professional judgments of accrual-based management.
The detailed results of the Open Budget Initiative (OBI) of 2008 reveal uneven patterns of transparency at different stages of the budget process, although in some countries, governments and stakeholders have begun to find ways to make the budgeting process more consultative. At each stage of the budget process, the information made publicly available determines the ability of stakeholders to influence, monitor and assess the effectiveness of government policies.
Although not all the OECD countries are included in the survey, the results indicate that the OECD countries have largely implemented more advanced levels of budget transparency compared to the rest of the world. Only five of the 85 countries surveyed provide extensive budget information, but four of them are OECD countries. More than 50% of OECD countries publish the required reporting for budget transparency (Table 4.13), which encourages stakeholder involvement.
Table 4.13 Budget Transparency Scores
Number of countries providing Scant or no
information (OBI sub scores 0-20)
Minimal information (OBI sub scores 21-40)
Some information (OBI sub scores 41-60)
Significant information (OBI sub scores 61-80)
Extensive information (OBI sub scores 81-100)
Pre-budget statement 55 (3) 0 4 (1) 8 (3) 18 (5)
Executive’s Budget proposal 24 10 28 (4) 17 (3) 6 (5)
Enacted budget 4 11 0 30 (4) 40 (8)
Citizens budget 68 (5) 4 (1) 0 3 (1) 10 (5)
In-Year reports 21 5 (1) 10 (1) 22 (1) 27 (9)
Mid-Year review 63 (5) 5 4 (1) 3 (2) 10 (4)
Year-End report 37 (1) 18 14 (2) 11 (6) 5 (3)
Audit report 32 8 (2) 13 11 (2) 21 (8)
Source: Open Budget Initiative, 2008.
Notes: Twelve OECD countries are included in the survey: the Czech Republic (62), France (87), Germany (64), Mexico (55), New Zealand (86), Norway (80), Poland (67), South Korea (66), Sweden (78), Turkey (43), The United Kingdom (88) and the United States (82)). The number of OECD countries is indicated in brackets.
Most OECD countries included in the survey publish the annual budget approved by the legislature, allowing for some level of stakeholder involvement in the budget process. The
budget formulation stage, where most major policy and resource allocation decisions are made, is relatively open, when compared with non-OECD countries. The OBI 2008 clearly shows that most OECD countries provide sufficient information to allow stakeholders to engage meaningfully in the decisive stage of the budget process. This provides stakeholders with critical opportunities to offer input about the major assumptions underlying the budget, the key macro policy issues, and the setting of major priorities. It also encourages the potential stakeholders’ contribution to enhancing the priorities setting, resource allocations and value for money received from public spending. Although all OECD countries surveyed make their budgets publicly available once approved to allow stakeholders to engage in some level of budget execution monitoring, 50% of them do not publish citizen budgets (budget summaries for citizens), which is instrumental in generating greater stakeholder understanding and engagement in the budget process.
Table 4.14 Stakeholder Involvement Scores
Stakeholder involvement Number of countries providing Scant
(OBI sub scores 0)
Low (OBI sub scores 33)
Significant (OBI sub scores 67)
Extensive (OBI sub scores 100) Can citizens obtain disaggregated program financial
information?
38 (1) 17 14 (4) 16 (7)
Can citizens obtain disaggregated non-financial information?
42 (1) 21 (2) 14 (4) 8 (5) Public hearings on macroeconomic and fiscal
framework
34 33 (4) 15 (6) 3 (2) Public hearings on individual budget of central
government administrative units
37 22 (3) 17 (5) 9 (4) When is the audit report released? 36 12 (1) 22 (5) 15 (6) Source: Open Budget Initiative, 2008.
Notes: Twelve OECD countries are included in the survey. The number of OECD countries is indicated in brackets.
Having timely, accurate, accessible and useful information during budget execution is important, and can enable stakeholders to augment their government-monitoring capacity. Although OBI results indicate transparency at the budget evaluation and audit levels, information about budget execution is mixed and rather poor in overall with better in-year reporting than mid-year or year-end reporting. Most OECD countries surveyed publish comprehensive audit reports within the recommended timeframe, which provides opportunity for stakeholders to use audit information to advocate for performance improvement.
Table 4.15 Are the findings of the Supreme Audit Institution available to the public?
No. of country %
Yes, always 17 57%
Yes, in most cases but with some exception (e.g. audit of the military) 10 33%
Rarely 1 3%
Other 2 7%
Total 30 100%
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
Table 4.16 When are the audited financial statements publicly available?
No. of country %
1 – 3 months after the end of the fiscal year 4 13%
4 – 6 months after the end of the fiscal year 10 33%
7 – 9 months after the end of the fiscal year 8 27%
10 – 12 months after the end of the fiscal year 6 20%
More than 12 months after the end of the fiscal year 2 7%
Total 30 100%
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
Furthermore, the OBI survey results confirm that OECD countries provide more opportunities for stakeholder involvement compared to non-OECD countries. In the former, stakeholders can obtain disaggregated financial and non-financial information, and they can participate on public hearing about macroeconomic and fiscal frameworks and the individual budgets of central government units.
2.2.3. Strategic perspective: Multi-year estimates and budget flexibility
The majority of OECD countries prepare aggregate and line item level multi-year estimates, which is accompanied by limited budget flexibility in terms of the authority to change spending, to reallocate funds and to carry over unused funds. Thus, it could be argued that the multi-year estimates in OECD countries are only considered to be one component of planning and do not bind appropriation for future budget periods. The implementation of multi-year estimates is more effective when complemented with flexible budget practices (see table 4.18a and 4.18b.).
It is possible to improve the multi-year estimates in the future, as the link between expenditures (cost) and performance goals/objectives, and budget flexibility level improve.
Table 4.17 Multi-year estimates and expenditure targets
Level Multi year
Expenditure estimate Expenditure target (ceiling) No. of country % No. of country %
No 1 3.3% 10 33.3%
Aggregate level 10 33.3% 11 36.7%
Ministry level 5 16.7% 3 10%
Line item level 9 30% 1 3.3%
Other 5 16.7% 5 16.7
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
Notes: In this table, 43.3% of OECD countries prepare three years’ estimates, 40% of OECD countries prepare four years’ estimates and 10% of OECD countries prepare five years’ estimates. Moreover, 80%
of OECD countries update their multi-year estimates annually, while 6.7% of OECD countries update such estimates every two years.
Table 4.18a Budget flexibility
Government has authority to change spending once the budget has been approved by the Legislature
Ministers are allowed to reallocate funds between
LM within their responsibility Increase spending Cut/cancel/ rescind
spending
No. of country % No. of country % No. of country %
No 4 13.3% 6 20% 1 3.3%
Yes, without restriction 10 33.3% 10 33.3% - -
Yes, with restriction 16 53.3% 14 46.7% 29 96.7%
Does it require any approval?
Approval is not required 2 6.7% 8 26.7% -
It requires prior CBA (MoF) approval
12 40% 7 23.3% 10 33.3%
It requires prior legislative approval
9 30% 2 6.7% 8 26.7%
It requires subsequent legislative approval
6 20% 3 10% - -
Other 4 13.3% 7 23.3% - -
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
Table 4.18b Budget flexibility
Ministries are allowed to borrow against future appropriations
Ministers are allowed to carry-over unused funds or appropriations from one year to another Operating
expenditure
Investment Transfer/
subsidy
Operating expenditure
Investment Transfers /subsidy No. of
country
% No. of country
% No. of country
% No. of country
% No. of country
% No. of country
%
No 22 73.3% 21 70% 24 80% 5 16.7% 3 10% 9 30%
Yes, without restriction
4 13.3% 3 10% 3 10% 3 10% 5 16.7% 2 6.7%
Yes, with restriction - - - - - - 22 73.3% 22 73.3% 18 60%
Does it require any approval?
Approval is not required
- - - It requires prior
CBA (MoF) approval
2 6.7% 2 6.7% 10 33.3% 11 36.7% 8 26.7%
It requires prior legislative approval
- - 1 3.3% - - 6 20% 6 20% 4 13.3%
Other 2 6.7% 6.7% 2 6.7% - - -
Source: OECD (GOV), 2007 Budget Practices and Procedures Database.
2.2.4. Performance evaluation and utilization of performance information
The main focus of the performance management system in OECD countries is on the performance evaluations, which are mainly carried out by different actors, usually internal. The results of such evaluations are used as feedback to improve the planning and resource allocation process. The link between performance evaluation and the planning and budgeting process ties the PDCA cycle between fiscal years and encourages the adoption of the strategic perspective.
OECD countries have moved to the next level by implementing performance budgeting through loosely linking performance evaluations to planning and budgeting processes (table 4.19).
According to the OECD survey conducted in 2007, 20% of OECD countries confirmed that the
performance against targets and evaluation results are used to determine budget allocation, although there is no further explanation regarding the methods of resource allocation. More than 50% of OECD countries confirmed that there is a loose or indirect link between performance information and funding as performance measures and evaluations are most often used along with information about fiscal policies and policy priorities to inform, but not determine, budget allocations (performance-informed budgeting).
Table 4.19 Link between performance information and funding
Type Linkage between performance information and funding
Planned or actual performance Main purpose in the budget process Presentational No link Performance targets and/or
performance results
Accountability Performance-informed
budgeting
Loose/indirect link Performance targets and/or performance results
Planning and/or accountability Direct/formula
performance budgeting
Tight/direct link Performance results Resource allocation and accountability Source: OECD (2007), Performance Budgeting in OECD Countries, OECD Publishing, Paris, p.21.
Figure 4.3 Use of performance budgeting system at the central level of government (2007)
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
FINLAND AUSTRALIA KOREA NETHERLANDS NORWAY POLAND UNITED STATES CANADA UNITED KINGDOM SWEDEN TURKEY SLOVAK REPUBLIC JAPAN NEW ZEALAND AUSTRIA DENMARK IRELAND MEXICO OECD30 GREECE LUXEMBOURG FRANCE ITALY ICELAND SWITZERLAND PORTUGAL GERMANY HUNGARY SPAIN BELGIUM CEZCH REPUBLIC
Source: Government at a Glance 2009.
Note: Index comprised between 0 (no performance budgeting) and 1 (high performance budgeting level).
Cronbach's alpha : 0.861 (computed with SPSS). A Cronbach 's alpha close to 0.6 or 0.7 indicates a high degree of correlation among a set of variables.
In terms of sponsoring public sector reform, the Ministry of Finance has been a strong advocate of performance management reform, as shown by the OECD database. In most cases the Ministry of Finance or Treasury serves as the primary control point for spending and tax decisions. The Ministry of Finance or Treasury is responsible for the multi-year estimates of expenditures and revenues based on the review of government priorities. In relation to expenditures, the role of the Ministry of Finance or Treasury is to validate line ministries’
program costs to ensure their accuracy and reasonableness. Despite the political regime, the line ministries (including the Ministry of Finance) have significant authority in policy-making and implementation processes. Consequently, they have a strong role in the performance management system, either in the performance target setting, execution or evaluation. This is