PARIS21 statistical capacity building indicators

A system with the entire statistical system in mind and using a simplified version of DQAF for any statistical process. An extract from the Task team report on Statistical Capacity Building Indicators (September 2002)

In a nutshell, what are the Statistical Capacity Building (SCB) indicators?

The SCB indicators measure the statistical conditions in a country through a prism that captures representative elements of these conditions:

• Sixteen quantitative indicators cover resources (domestically and externally funded annual budget, staff, and equipment), inputs (survey and administrative data sources), statistical products.
• Eighteen qualitative indicators focus on relevant aspects of environment (institutional and organizational), of core statistical processes, and of statistical products.

They are compiled using a questionnaire which can be self-administered by data-producing agencies and they can be used for international comparative purposes (applied at a set level of data-producing agencies and statistics) and for national uses (applied at a level customized to meet specific needs).

Qualitative indicators

The qualitative indicators embrace the broader view of factors in the statistical environment, the statistical process, and the characteristics of the statistical products in meeting users’ needs. Because the Data Quality Assessment Framework (DQAF), introduced by the IMF, encompasses these various aspects, its six-part structure was adopted very early in the process to derive and present the qualitative indicators.

In total, 18 qualitative indicators were identified, pertaining to Institutionnal Prerequisites, Integrity, Methodological soundness, Accuracy and reliability, Serviceability, Accessibility.

They cover:

• the legal and institutional environment, and resource conditions needed to perform statistical operations, obtain cooperation of respondents and administrative authorities, and manage statistical operations;
• the professional and cultural setting in which the statistical operations are conducted;
• the methodological expertise for establishing data sources and their links to the statistical products;
• the population to be covered, and the surveys, survey questionnaires, and administrative data sources;
• the skills and techniques to transform source data into statistical products;
• the assessment and validation of source data, the use of statistical techniques, the assessment nd validation of intermediate data, and statistical outputs;
• the relevance of the statistics to social and economic concerns, including the analytical capability to confirm certain issues and to identify those that need probing;
• the periodicity, timing, and internal/relational consistency of the statistics; and
• the methods and channels used to ensure wide and relevant dissemination of the statistical products.

The qualitative indicators serve more as measures of efficiency and effectiveness of statistical production. They help to show the following: (1) if the legal and institutional environment facilitates the production of the statistics, (2) if the resources are sufficient and activities meshed to promote productivity, (3) if the culture is amenable to quality work, (4) if the integrity and professionalism are protected and transparency measures are in place, (5) if the core statistical processes are performed according to methodological requirements and the source data available and techniques used are adequate, (6) if measures are in place to maintain the relevancy of the products, and (7) if the characteristics of the statistics produced fit users’ needs.

They can help to assess both the production of specific datasets and the health and well-functioning of data-producing agencies.

Each indicator is evaluated against a four-scale assessment level, to which are attached benchmark descriptions:

• level 4 applies to highly developed statistical activities;
• level 3 to moderately well-developed activities;
• level 2 to activities that are developing but still have many deficiencies; and
• level 1 to activities that are underdeveloped.

The ratings were designed with a view that ratings of 3 or 4 would refer to activities where no external support would be required.

While the benchmark descriptions reduce the subjectivity inherent in qualitative indicators, these descriptions may need to be further adjusted as experience is gained from their use. For instance, comparing the responses from self-assessment against independent expert views would help to confirm the validity of the benchmark descriptions. Further, if the recorded results concentrate at the 4 and 3 levels, rebalancing may be required to better delineate responses across levels 1 to 4

The 18 data-related indicators

Rating scale: 4: Highly developed; 3: Developed; 2: Largely Undeveloped; 1: Undeveloped

0. Prerequisites:

0.1 Collection of information and preservation of confidentiality guaranteed by law and effective

0.2 Effective coordination of statistics

0.3 Staff level and expertise adequacy

0.4 Buildings and equipment adequacy

0.5 Planning, monitoring and evaluation measures implemented

0.6 Organizational focus on quality

1. Integrity:

1.1 Independence of statistical operations

1.2 Culture of professional and ethical standards

2. Methodological soundness:

2.1 International/regional standards implemented

3. Accuracy and reliability:

3.1 Source data adequacy

3.2 Response monitoring

3.3 Validation of administrative data

3.4 Validation of intermediate and final outputs

4. Serviceability:

4.1 User consultation

4.2 Timeliness of statistical outputs

4.3 Periodicity of statistical outputs

5. Accessibility

5.1 Effectiveness of dissemination

5.2 Updated metadata

Example of a benchmark description

For indicator 0.1 Collection of information and preservation of confidentiality guaranteed by law and effective

Level 4

(i) There is effective access in practice to information (collection of basic information required and access to public sector administrative information) as provided for by the statistical legislation.

(ii) The legislation gives the data-producing agencies full responsibility to compile and disseminate a range of statistics.

(iii) The legislation provides that all individual source data must be used for statistical purposes only and remain confidential (unless the respondent consents to release).

(iv) There are prescribed penalties for breach of confidentiality that act as an effective deterrent to non-compliance. The current judicial system ensures that statistical legislation can be enforced.

Level 3

(i) There is limited effective access in practice to the information (collection of basic information required and access to public sector administrative information) even if such access is provided for by the statistical legislation.

(ii) The legislation gives the data-producing agencies responsibility to compile and disseminate a range of statistics

(iii) The legislation provides that all individual source data be used for statistical purposes only and remain confidential (unless the respondent consents to release).

(iv) The penalties for disclosure of confidential information are somewhat inadequate as a deterrent to non-compliance. The current judicial system is sufficiently developed to ensure broad enforcement of statistical legislation.

Level 2

(i) There is no effective access in practice to the information (collection of basic information required and access to public sector administrative information) even is such access is provided for by the statistical legislation.

(ii) The legislation does not specify the responsibility to compile and disseminate a range of statistics.

(iii) There is no clear statement about the confidentiality of individual data.

(iv) There are no penalties for disclosure of individual data. The current judicial system cannot adequately ensure enforcement of statistical legislation.

Level 1

(i) Statistical legislation is non-existent, gives no access to public sector administrative information.

(ii) No responsibility is specified by law to compile and disseminate the statistics.

(iii)There is no preservation of the confidentiality of individual data.

(iv) There are no penalties for disclosure of individual data. The current judicial system cannot ensure enforcement of statistical legislation.