South African statistical quality assessment framework

The challenge remains to build trust in official statistics. Central to this challenge is the improvement of the quality of the data produced in Stats SA and all organs of state. The higher objective is present to our country with very high integrity statistics – capable of informing the development agenda in government and useful in planning and decision-making in the private sector and non-governmental organisations

To achieve this objective of the delivery of data of high integrity we need to address the question of quality. SASQAF, with its framework of the eight quality dimensions, will play a vital role in achieving this objective. The eight dimensions of quality are:

• Relevance; • Accuracy; • Timeliness; • Accessibility; • Interpretability; • Comparability and Coherence; • Methodological soundness; and • Integrity.

Structure of the framework

SASQAF covers the various quality aspects of the entire statistical value chain (i.e. need, design, build, collection, processing, analysis and dissemination), and certifies national statistics on one of four levels. Level 4, certification (quality statistics), indicates optimal conditions for statistical production, while Level 1 (poor statistics) indicates the least favourable conditions. In outline, the four levels of certification are as follows:

Level Four: Quality Statistics – These are statistics that meet all the quality requirements as set out in SASQAF. They are designated as quality statistics to the extent that deductions can be made from them, and are ‘fit for use’ for the purpose for which they were designed. Level 4 applies to highly-developed statistical activities with respect to the corresponding indicator.

Level Three: Acceptable Statistics – These are statistics that meet most, but not all the quality requirements as stipulated in SASQAF. They are designated as acceptable to the extent that, despite their limitations, deductions can be made, and are ‘fit for use’ for the purpose for which they were designed. Level 3 refers to moderately well-developed activities with reference to a particular indicator.

Level Two: Questionable Statistics – These are statistics that meet few of the quality requirements as stipulated in SASQAF. They are designated as questionable to the extent that very limited deductions can be made, and they are therefore not ‘fit for use’ for the purpose for which they were designed. Level 2 refers to activities that are developing but still have many deficiencies.

Level One: Poor Statistics – These are statistics that meet almost none of the quality requirements as stipulated in SASQAF. They are designated as poor statistics to the extent that no deductions can be made from them, and are not ‘fit for use’ for the purpose for which they were designed. Level 1 refers to activities that are underdeveloped.

The dimensions of quality

Prerequisites of quality

Description: The prerequisites of quality refer to the institutional and organisational conditions that have an impact on data quality. It defines the minimum set of necessary conditions that have to be met in order to produce good quality statistics. It therefore serves as the foundation on which all other dimensions of data quality should be premised on.

Key components

• Legal and institutional environment (including Memoranda of Understanding (MoUs) or Service Level Agreements (SLAs)
• Privacy and confidentiality
• Commensurability of resources
• Quality as the cornerstone of statistical work

Relevance

Description: Relevance of statistical information reflects the degree to which the data meet the real needs of clients. It is concerned with whether the available information sheds light on the issues of most importance to users.

Key components

• Why do you need to conduct the survey or collect data?
• Who are the users of the statistics?
• What are their known needs?
• How well does the output meet these needs?
• Are user needs monitored and fed back into the design process?

Accuracy

Description: The accuracy of statistical information is the degree to which the output correctly describes the phenomena it was designed to measure. Source data available provide an adequate basis to compile statistics.

Key components

• Assessment of sampling errors where sampling was used.
• Assessment of coverage of data collection in comparison to the target population.
• Assessment of response rates and estimates of the impact of imputation.
• Assessment of non-sampling errors and any other serious accuracy or consistency problems with the survey results or register based statistics.
• Data capture, data coding and data processing errors.
• Source data available provide an adequate basis to compile statistics (e.g. administrative records).
• Source data reasonably approximate the definitions, scope, classifications, valuation, and time of recording required.
• Source data are timely.

Timeliness

Description: Timeliness of statistical information refers to the delay between the reference point to which the information pertains and the date on which the information becomes available. Timeliness also addresses aspects of periodicity and punctuality of production activities within the statistical value chain.

Key components

• Statistics production time
• Timely receipt of administrative records.
• Periodicity of statistical release.
• Punctuality of statistical release.

Accessibility

Description: The accessibility of statistical information and metadata refers to the ease with which it can be obtained from the agency. This includes the ease with which the existence of information can be ascertained, as well as the suitability of the form or medium through which the information can be accessed. The cost of the information may also be an aspect of accessibility for some users.

Key components

• Catalogue systems are available in the organ of state or statistical agency
• Delivery systems to access information
• Information and metadata coverage is adequate
• Measure of catalogue and delivery systems performance
• Means of sharing data between stakeholders

Interpretability

Description: Interpretability of statistical information is refers to the ease with which users understand statistical information through the provision of metadata.

Key components

• Concepts and definitions, and classifications that underlie the data;
• Metadata on the methodology used to collect and compile the data;
• Key findings, giving the summary of the results;
• Presentation of statistics in a meaningful way.

Comparability and Coherence

Description: Comparability of statistical information is the ability to compare statistics on the same characteristic between different points in time, geographical areas or statistical domains. The coherence of statistical information reflects the degree to which it can be successfully brought together with other similar statistical information from different sources within a broad analytic framework and over time. It is the extent to which differences between two sets of statistics are attributable to differences between the estimates and the true value of the statistics.

Key components

• The use of common concepts and definitions within and between series.
• The use of common variables and classifications within and between statistical series.
• The use of common methodology and systems for data collection and processing within series.
• The use of common methodology for various processing steps of a survey such as editing and imputations within series.

Methodological soundness

Description: It refers to the application of international, national, or peer-agreed standards, guidelines, and practices to produce statistical outputs. Application of such standards fosters national and international comparability.

Key components

• International norms and standards on methods.
• Data compilation methods employ acceptable procedures.
• Other statistical procedures employ sound statistical techniques.
• Transparent revision policy and studies of revisions are done and made public.

Integrity

Description: The integrity of statistical information refers to values and related practices that maintain users’ confidence in the agency producing statistics and ultimately in the statistical product. This includes, among others, the need for the statistical system to be based on the United Nations (UN) principles of official statistics and includes principles of objectivity in collection, compilation and dissemination of data to ensure unbiased statistics which are not subject to confidentiality breaches or premature releases.

Key components

• Professionalism and ethical standards which guide policies and practices.
• Assurances that statistics are produced on an impartial basis.
• Ethical standards are guided by policies and procedures.