PART I. THEORETICAL FRAMEWORK DEVELOPMENT OF PERFORMANCE MANAGEMENT
SECTION 3. H YPOTHESIS TESTING AND MODELING RELATIONAL AND CAUSAL LINKAGES
This section will test the hypotheses presented in Chapter 3 to identify the relationships between variables and to develop a relational and causal linkage of performance management systems based on OECD data, as OECD countries are considered to have implemented a well-developed performance management systems based on the discussion on the previous section. The main variables in the hypothesis testing are listed in Table 4.3 and 4.4.
3.1. Factor analysis for classification method and scale validation
This study’s main research questions seek to understand the utilization of performance and accrual (cost) information under a set of three different constructs as dependent variables: a quasi-contractual relationship, stakeholder management and involvement and a strategic perspective.
The next logical step is conducting exploratory factor analysis to determine the number of factors being assessed and the variables representing each of these factors. The questionnaire items (variables) and the corresponding factor loading for the dependent variables measuring the quasi-contractual relationship, stakeholder management and involvement, and a strategic perspective are presented in Tables 4.3 and 4.4. A factor loading above .30 can be considered high (Spector, 1992; Hatcher, 1994). The results of the various factor analyses are shown in the Tables 4.27, 4.28, 4.29, 4.30, and 4.31. The results of the testing for scale reliability are shown in Table 4.32. Having a high factor loading is part of the criteria used to determine which factors should be retained.
Exploratory factor analysis was used to analyze all the variables comprising the general constructs, which are performance measurement and accounting systems. The principal factor method was used to extract the factors, as the analysis uses the variability in an item that it has in common with the other items in order to detect structure (factor analysis as a classification method). This procedure was followed by variance maximizing (varimax) rotation, where goal is to obtain a clear pattern of loadings – that is, factors that are somehow clearly marked by high loadings for some variables and low loadings for others, by maximizing the variance (variability) of the “new” variable (factor), while minimizing the variance around the new variable. Varimax rotation obtains a pattern of loadings on each factor that is as diverse as possible, thus lending itself to easier interpretation.
Each factor is constructed based on the variables with the highest loading (above .30) that describes similar characteristics. Factor 1 is labeled as the link between cost and performance information. The analysis confirmed that internal charges of capital assets and internal charges
for goods and service are indicators of cost information becasue the charge requires the calculation of depreciation and full cost information. Factor 2 is labeled as the link between expenditures and performance information. Factor 3 is labeled as cost information and Factor 4 is labeled as benchmarking. The reliability test for this scale yielded a coefficient alpha that is below the recommended minimum of .70. Thus, the variable with a low loading value on the factor analysis might decrease the internal consistency of the scale.
Table 4.27 Results of exploratory factor analysis to link expenditures (cost) with performance information
Link expenditure (cost) with performance information Factor 1 Factor 2 Factor 3 Factor 4
Performance target .88 .12 .04 .18
Performance measure .88 -.14 .22 .11
Output .63 .25 .05 -.14
Outcome .53 .61 .11 .32
Benchmarking .29 .25 -.04 .78
Link expenditure with performance target .25 .86 .01 -.22 Link expenditure with performance goals/ objectives .01 .85 .01 .12
Internal charge for capital spending .39 .27 .03 -.66
Internal charge for goods/services .69 .37 .04 -.09
Budgeting basis .05 -.18 .93 -.23
Categorization of operating and capital expenditure -.16 -.51 .34 -.39
Accrual information .35 .29 .75 .23
The explanatory factor analysis test was also conducted on dependent variables because they are constructed based on various questionnaire items. It constructed the variables for the quasi-contractual relation: lump sum appropriation (Factor 1) and internal charges (Factor 2).
The two factors underline the quasi-contractual relation because the contractual relation should be developed upon autonomy and flexibility – as indicated by lump sum appropriation – under the link between performance and cost information, which is indicated by the internal charges.
However, the reliability test for the internal charge yielded a coefficient alpha that was below the recommended minimum of .70. Thus the variables with a low loading values on the factor analysis might show internal consistency.
Furthermore, the measure of stakeholder management and involvement is constructed based on the accountability process (Factor 1), and on stakeholder ability to obtain information and participate in decision-making processes (Factor 2). The reliability test for the accountability process yielded a coefficient alpha that was above the recommended minimum of .70. Thus, the variables with low loading values on the factor analysis do not decrease the internal consistency of the scale. On the other hand, the reliability test for stakeholder ability to obtain information and participate in decision-making processes yielded a coefficient alpha below the minimum recommended, causing internal inconsistency in the scale.
Table 4.28 Results of exploratory factor analysis for the quasi-contractual relation
Quasi contractual relation Factor 1 Factor 2
Lump sum appropriation -.003 .955
Internal charge for capital assets .820 .264
Internal charge for goods and service .805 -.288
Table 4.29 Results of exploratory factor analysis for stakeholder management and involvement
Stakeholder management/involvement Factor 1 Factor 2
Pre-budget statement .71 .-51
Executive’s Budget proposal .59 .23
Enacted budget .44 .62
Citizens budget .76 -.11
In-Year reports -.21 .00
Mid-Year review .06 .55
Year-End report .86 .29
Audit report .81 .55
Can citizen obtain disaggregate program financial information? .54 .63 Can citizen obtain non disaggregate financial information? .81 .26 Public hearing on macro economic and fiscal framework -.15 .78 Public hearing on individual budget of central government administrative unit .09 .63
When audit report is released? .37 .03
Table 4.30 Results of exploratory factor analysis for the strategic perspective
Strategic perspective Factor 1 Factor 2 Factor 3 Factor 4
Review of ongoing program by CBA -.05 .18 .63 .54
Review of ongoing program by ministry of finance -.04 .13 .88 -.06 Review of ongoing program by line ministries .69 .18 -.06 .67
Review of ongoing program by SAI .45 .51 -.13 .26
Review of ongoing program by Legislative .11 .96 .08 .10 Review efficiency and cost effectiveness by CBA .64 .12 .48 .03 Review efficiency and cost effectiveness by ministry of finance .26 -.13 .87 .28 Review efficiency and cost effectiveness by line ministries .92 -.16 .15 .28 Review efficiency and cost effectiveness by SAI .92 -16 .15 .28 Review efficiency and cost effectiveness by Legislative .48 .73 .32 -.23
Ex post review of program by CBA -.17 .83 -.30 .32
Ex post review of program by ministry of finance .52 .48 .15 -.08 Ex post review of program by line ministries .69 .18 -.06 .67
Ex post review of program by SAI .33 .44 -.23 .07
Ex post review of program by Legislative -.18 .88 .14 -.22
Review of new initiative by CBA -.07 .41 -.06 .61
Review of new initiative by ministry of finance .04 -.05 .57 -.13 Review of new initiative by line ministries .33 .30 .24 .80
Review of new initiative by SAI -.03 .58 -.51 .40
Review of new initiative by Legislative -.17 .83 -.30 .32 Budget flexibility to increase spending -.21 -60 .57 -.32 Budget flexibility to carry over unspent operating spending -.51 -.10 .51 .19 Budget flexibility to carry over unspent capital spending -69 -.10 .51 -.09
Multi year estimate .17 -.26 -.15 .78
Multi year expenditure target -.77 -.09 -.28 .12
Public hearing on macro economic and fiscal framework .73 -.04 -.22 .15 Public hearing on individual budget of central government
administrative unit
.28 .29 .39 -.18
Furthermore, the exploratory factor analysis of the strategic perspective confirmed that the measure is constructed by the performance evaluations of line ministries (Factor 1), performance evaluations of SAI and legislatures (Factor 2), performance evaluations by the
CBA or Ministry of Finance and budget flexibility (Factor 3), and multi-year estimates (Factor 4). The reliability test for performance evaluations by line ministries, SAI and legislatures yielded a coefficient alpha above the recommended minimum of .70. Thus the variables with low loading values on the factor analysis do not decrease the internal consistency of the scale.
However, the reliability test for performance evaluations by the CBA or Ministry of Finance, budget flexibility and multi-year estimates yielded a coefficient alpha below the recommended minimum of .70, leading to internal inconsistency of the scale.
Table 4.31 Results of exploratory factor analysis for the utilization of performance and accrual (cost) information
Utilization of performance and accrual information Factor 1 Factor 2 Utilization of performance information in budgeting by ministry of finance .83 .49 Utilization of performance information in budgeting by CBA .79 .51 Utilization of performance information in budgeting by line ministries .94 .21 Performance against target is used by CBA to determine budget allocation .02 .96 Performance against target is used by CBA to inform budget allocation .97 .11 Performance against target is not used by CBA to inform budget allocation -.64 -.66 Evaluation is used by CBA to determine budget allocation .15 .92 Evaluation is used by CBA to inform budget allocation .78 -.01 Evaluation is not used by CBA to inform budget allocation -.57 -.64
Lastly, the exploratory factor analysis of the utilization of performance and accrual (cost) information confirmed that the measure is constructed by: the utilization of performance information to inform budget allocation decisions (Factor 1), and the utilization of performance information to determine budget allocation decisions (Factor 2). The reliability test for utilization of performance information to inform and determine budget allocation decisions yielded a coefficient alpha that was above the recommended minimum of .70. Thus, the variables with low loading values on the factor analysis do not decrease the internal consistency of the scale.
Table 4.32 Cronbach Coefficient Alpha for independent factors
The link of expenditure (cost) and performance information Raw variable coefficient Link between cost and performance information .633
Link expenditure and performance information .589
Cost information .559
Benchmarking* -
Quasi contractual relation
Lump sum appropriation* -
Internal charge .255
Stakeholder management/involvement
Accountability process .825
Stakeholder ability to obtain information and participate in decision making .660 Strategic perspective
Performance evaluation by line ministries .832
Performance evaluation by SAI and legislative .740 Performance evaluation by CBA/MoF and budget flexibility .636
Multi year estimate .483
The utilization of performance and accrual (cost) information
Utilization of performance information to inform budget allocation .934 Utilization of performance information to determine budget allocation .933 Note: * The reliability test cannot be performed on this factor because it only consists of one variable.
3.2. Chi-square test: Hypothesis testing and modeling variables’ relationships
This section presents the findings of the Chi-square test conducted to test the hypotheses presented earlier and to develop model regarding the relationships between variables. The analyses consisted of running Chi-square tests on the variables and constructs identified in the previous section to develop an elaborated performance management model.
3.2.1. Chi-square test on the relationships between independent variables
The Chi-square test confirmed the link between performance measurement (performance target, measure, and evaluation) and accounting systems to generate and link cost information with the performance information. The statistical test confirmed that performance targets, measures, evaluation reports, benchmarking, and internal charges for capital assets are significantly related to accrual accounting to at a significant level of 5%. Furthermore, the test confirmed the relation between budgeting and accounting basis at a significant level of 5%.
The statistical test revealed independent relationships between variables to construct cost information–budgeting basis, accounting basis and the separation of operational and capital budgets–and confirmed the link between expenditures and performance targets/goals/objectives.
In other words, the implementation of the accrual (cost) concept in the budgeting and reporting does not influence the link between expenditures and performance targets/goals/objectives, and vice versa. One country could link (line item) expenditures with performance targets, despite of its budgeting and reporting basis. It is possible to argue that performance management systems in OECD countries are developed based on performance-informed systems (performance evaluation and benchmarking), in which performance targets/goals/objectives are linked to (line item) expenditures, instead of cost, as the OECD countries are still developing the cost concept.
Table 4.33a Chi square test of the relation between variables on performance and accounting information
Link between cost and performance information Performance target
Performance measure
Output Outcome Performance target - .001**(.333) .144(.083) .001**(.471) Performance measure .001**(.333) - .033**(.111) .019**(.143) Internal charge for capital assets .046**(.208) .068*(.190) .068*(.167) .516(.087) Internal charge for goods/services .172(.148) .186(.042) .185(.048) .130(.154) Link between expenditure and performance information
Link between expenditure with performance targets .460(.105) .916(.000) .985(.000) .116(.316) Link expenditure with performance goals/objectives .345(.150) .186(.111) .352(.125) .070*(.300) Cost information
Budgeting basis .207(.000) .099(.000) .271(.000) .546(.000) Accounting basis .011**(.217) .010**(.200) .626(.000) .075*(.182) Separation of operational and capital budget .785(.000) .303(.105) .957(.000) .453(.095) Benchmarking (and evaluation)
Evaluation report .109(.071) .000**(.455) .014**(.125) .036**(.077) Benchmarking .139(.000) .073*(.000) .232(.000) .433(.000) Note: * p <0.10; **p<0.05. H0 is rejected if less than p (the variables are not independent of each other).
Table 4.33b Chi square test of the relation between variables on performance and accounting information
Link between cost and performance information Internal charge for capital assets
Internal charge for goods/services
Performance target .046**(.208) .172(.148)
Performance measure .068*(.190) .186(.042)
Internal charge for capital assets - .249(.156)
Internal charge for goods/services .249(.156) -
Link between expenditure and performance information
Link between expenditure with performance targets .573(.091) .093*(.360) Link expenditure with performance goals/objectives .287(.227) .544(.259) Cost information
Budgeting basis .099(.174) .813(.000)
Accounting basis .028**(.379) .369(.065)
Separation operational and capital budget .568(.077) .276(.200) Benchmarking (and evaluation)
Evaluation report .142(.150) .504(.000)
Benchmarking .690(.042) .406(.000)
Note: * p <0.10; **p<0.05. H0 is rejected if less than p (the variables are not independent of each other).
Table 4.33c Chi square test of the relation between variables on performance and accounting information
Link between expenditure and performance information
Budgeting basis
Accounting basis
Separation operational and
capital budget
Evaluation report
Benchmarking
Link between expenditure with performance targets
.488(.100) .195(.318) .213(.409) .964(.000) .573(.143) Link expenditure with
performance goals/objectives
.380(.050) .464(.208) .296(.286) .352(.125) .096*(.286) Cost information
Budgeting basis - .000**(.455) .152(.150) .140(.000) .589(.000) Accounting basis .000**(.455) - .816(.000) .022**(.158) .025**(.217) Separation of operational and
capital budget
.152(.150) .816(.000) - .557(.056) .037**(.200) Benchmarking (and evaluation)
Evaluation report .140(.000) .022**(.158) .557(.056) - .109(.000) Benchmarking .589(.000) .025**(.217) .037**(.200) .109(.000) - Note: * p <0.10; **p<0.05. H0 is rejected if less than p (the variables are not independent of each other).
3.2.2. Hypothesis testing and modeling relational linkage on the quasi-contractual relationship and stakeholder management and involvement
The reform priority in the separation of policy making and implementation has been to shift the balance of power from the central government to service delivery partners and stakeholders in decision-making processes. The agencies, as one form of service delivery partners, are created to pursue a more independent decision-making process that has difficulty existing within the bureaucracy. The development of agencies highlights the transformation of the service delivery mechanism. Agencies are responsible to the line ministry, which is in turn accountable to the head of the executive branch or to parliament, and thus, ultimately, to the stakeholders. At the same time, agencies are also directly responsible for their stakeholders, to whom they deliver goods and services. It is difficult to articulate the role or power of the responsible minister (the central government) and stakeholders in their relationship with agencies, as these relationships
vary across countries. However, the responsible minister’s role in monitoring and oversight, which are critical to ensuring that the government’s interests are managed properly, should be regulated by the balance between a quasi-contractual relationship and stakeholder management and involvement.
3.2.2.1. Hypothesis testing and modeling relational linkages on the quasi-contractual relationship
The quasi-contractual relationship is indicated by the lump sum appropriation received by agencies, internal charges for capital assets, and internal charges for goods and assets. The quasi-contractual relation is significantly influenced by performance measurement and accrual accounting systems, and by the link between these two systems. The statistical test highlighted the significance of accrual accounting in the context of the quasi-contractual relation as accrual accounting positively influences lump sum appropriation and internal charges for capital assets.
It is also confirmed that internal charges for goods and service are related to the link between expenditures and performance targets. Performance targets and measures positively influence the internal charges for capital assets because performance information is evaluated by its link to full cost information. It is confirmed by statistical testing that lump sum appropriation is not directly related to the utilization of performance information in the budgeting process, but the utilization of cost information in terms of internal charges will positively influence the utilization of performance information in the budgeting process (by the Ministry of Finance).
The statistical test revealed that lump sum appropriation for agencies is significantly related to accrual accounting and benchmarking. Lump sum appropriation is evaluated based on benchmarking, which is significantly influenced by the link between cost and performance information. It is indicated by the Chi-square test of the relation between benchmarking with the main variables in the construct of cost information. These variables are performance measure, accrual accounting, the separation of operational and capital budgets, and the link between expenditures and performance goals/objectives. This confirmed the significance of cost information and its link to performance information in benchmarking. As a result, it is possible to argue that lump sum appropriation is related to the link between cost and performance information through the benchmarking process.
The quasi contractual relationship requires full cost information because it should be based on the performance contract that links performance with cost information. The performance contract sets the standardized performance for performance evaluation purposes. However, most OECD countries link performance information with (line item) expenditures instead of cost, highlighting the lack of full cost information. The statistical test indicates that the
implementation of accrual accounting has encouraged the quasi-contractual relation between central governments and service delivery partners, but the quasi-contractual relation is mainly based on benchmarking on the link between (accrual) expenditures instead of cost, with performance targets/goals/objectives due to the unavailability of full cost information. The statistical test confirmed the requirement to link cost and performance information in the quasi-contractual relation. It confirmed hypothesis 1a that the utilization of performance and cost information in decision-making process develops a quasi-contractual relation between the central government and service delivery partners.
In summary, it is possible to argue that the emphasis on evaluation and benchmarking without full cost information leads to relatively subjective evaluations in decision-making processes. As a result, a country that develops its evaluation and benchmarking practices based on a quasi-contractual relation without full cost information tends to develop a hierarchical and non-contractual relationship between the central government and service delivery partners.
Table 4.34 Hypothesis testing on the quasi-contractual relation as a dependent variable
Independent variables Dependent variable
Link between cost and performance information Lump sum appropriation
for agencies
Internal charge for capital assets
Internal charge for goods/
services Performance target .306(.100) .046**(.208) .172(.148) Performance measure .431(.000) .068*(.190) .186(.042) Internal charge for capital assets .462(.120) - .249(.156) Internal charge for goods/services .407(.143) .249(.156) - Link between expenditure and performance information
Link between expenditure with performance targets .701(.286) .573(.091) .093*(.360) Link expenditure with performance goals/objectives .759(.095) .287(.227) .544(.259) Cost information
Budgeting basis .575(.048) .099*(.174) .813(.000) Accounting basis .018**(.400) .028**(.379) .369(.065) Separation operational and capital budget .659(.091) .568(.077) .276(.200) Benchmarking (and evaluation)
Evaluation report .476(.000) .142(.150) .504(.000)
Benchmarking .042**(.227) .690(.042) .406(.000)
Strategic perspective
Multi year estimate .416(.208) .253(.231) .672(.071) Budget flexibility to increase spending .240(.115) .723(.069) .954(.000) Budget flexibility to carry over unspent operating
spending
.345(.048) .766(.043) .246(.115) Budget flexibility to carry over unspent capital spending .029**(.200) 140(.174.) .888(.000) Performance evaluation
Review of ongoing program by CBA .542(.160) .143(.241) .868(.031) Review of ongoing program by LM .847(.000) .031**(.227) .873(.000) Review efficiency and cost effectiveness by CBA .876(.000) .136(.185) .134(.167) Review efficiency and cost effectiveness by LM .889(.000) .005**(.333) .860(.000) Ex post review of program by CBA .313(.045) .020**(.280) .405(.000) Ex post review of program by LM .513(.053) .013**(.261) .906(.000) Review of new initiative by CBA .808(.080) .065*(.286) .148(.194) Review of new initiative by LM .935(.000) .013**(.261) .906(.000) Note: * p <0.10; **p<0.05. H0 is rejected if less than p (the variables are not independent to each other).
Figure 4.4. The link between the quasi-contractual relation (lump sum appropriation, and internal charges for capital assets and for goods/services), strategic perspective and utilization of performance information in decision making
Performance target Performance measure
(+) Utilization of performance information in budget negotiation
(+) Accrual accounting (+)
(+) Internal charge for capital assets (+)
(+) (-) Benchmarking (+) Lump sum appropriation for
agencies
(+)
(+) (-) (-)
Link between expenditure and performance information
(+) Internal charge for goods/service (+) Link between expenditure and performance information (+) (+) (+) Performance evaluation (+) Utilization of performance
information in budget negotiation
(+) Budget flexibility Source: Developed by author based on statistical test (see Table 4.34)
3.2.2.2. Hypothesis testing and modeling relational linkages on stakeholder management and involvement
The statistical test confirmed the significant relationship between the implementation of performance measurement and accrual (cost) concepts with stakeholder management, in the form of public availability of budget information and accountable budgeting practices.
The statistical test confirmed that the incorporation of performance information into the budget will positively influences the stakeholder management through the enacted budget. The performance measurement system – which generates performance targets, measures and evaluation – and the link between expenditures and performance information positively influence the public availability and timeliness of the enacted budget The assumptions used when developing the budget and allocating between sectors should be developed based on the link between expenditures (cost) and performance goals/objectives as main rule in the performance management system, according to the statistical test.
Furthermore, as the statistical test confirmed the relationship between accrual budgeting and accounting with the citizen budget, it is argued that the implementation of the accrual concept in budgeting and accounting (accrual budgeting and accounting) will enhance stakeholder management to the citizen level through the preparation of citizen budgets. The cost information would help governments to prepare citizens’ budgets because the cost information simplifies the budget and provides stakeholders with more objective standards to justify and evaluate performance. This test confirmed hypothesis 1a which holds that the utilization of performance and cost information in decision-making processes will develop stakeholder management and involvement.
The provision of information is not an end in itself. It is possible to argue that the provision of objective information in the public domain shifts the nature and quality of the public debate (Holzer and Yang, 2004). The debate moves beyond the subjective and biased evaluation of programs and self-serving assessment of interest groups toward the use of more objective criteria, by which to make rational decisions about policies and programs and resource allocation. The OBI survey provides strong evidence that the improved transparency has positively influenced stakeholder management and involvement. The statistical test confirmed that the implementation of performance measurement and evaluation provides stakeholders (citizens) with the ability to obtain disaggregate financial and non-financial information.
The statistical test confirmed hypothesis 1b regarding the relation between stakeholder management and stakeholder involvement. Stakeholder involvement is positively influenced by the ability of citizens to obtain the information about performance targets through the budget and evaluation reports. Citizens are able to obtain information mainly through the budgeting and accounting process: the pre-budget statement, enacted budget, in-year report and audit report.
The relationship between stakeholder management and involvement and the utilization of performance information in the budgeting process is also confirmed by the statistical test.
Stakeholder involvement through public hearings regarding the individual budget of central government units positively influence the utilization of performance information in budgeting by CBA.
Stakeholder management through the enacted budget is related to lump sum appropriation and a strategic perspective, which is reflected in the budget flexibility to carry over unspent operating spending and reviews of efficiency and cost effectiveness by line ministries. Among them, stakeholder management through the enacted budget is significantly influenced by budget flexibility in order to carry over unspent operating spending. The statistical test confirmed that the strategic perspective, in terms of budget flexibility and performance evaluation by the central government, is positively related to stakeholder management and involvement. The review of efficiency and cost effectiveness by line ministries positively influences the enacted budget as the decision making in resource allocation is based on the result of performance evaluations, conducted based on comparisons between budgeted and actual performance. The availability of in-year reports will positively influences the review of efficiency and cost effectiveness by the CBA because the performance evaluation is conducted periodically.
Furthermore, the ability of citizens to obtain disaggregate financial information is influenced by budget flexibility and ex post reviews by the CBA, as well as by the review of new initiatives by the legislature.
Stakeholder involvement through public hearings would likely help line ministries and the CBA