The Quality of statistics
Quality is more and more understood as meaning 'fit for purpose' beyond the traditional Accuracy-Reliability dimension; quality could qualify all sort of entities : a data, a data set, a publication, a database, an employee, a process, a team, a, instrument, a system, an organisation, ... ; and not only customers or users may be involved as there are other stakeholders those requirements are accounted for as in Total Quality Management (example : taxpayers when use of public money is involved).
To set up a Quality management system requires to improve organisation, to increase knowledge, to train staff but relatively few additional financial or human resources; the major issue being leadership both at the setting up stage and for the running of the system.
In 1994, during a session of the United Nations Statistical Commission, the community of officials statisticians adopted a set of 10 fundamental principles for official statistics:
- Principle 1: Relevance, impartiality and equal access: Official statistics provide an indispensable element in the information system of a democratic society, serving the government, the economy and the public with data about the economic, demographic, social and environmental situation. To this end, official statistics that meet the test of practical utility are to be compiled and made available on an impartial basis by official statistical agencies to honour citizens' entitlement to public information.
- Principle 2: Professionalism: To retain trust in official statistics, the statistical agencies need to decide according to strictly professional considerations, including scientific principles and professional ethics, on the methods and procedures for the collection, processing, storage and presentation of statistical data.
- Principle 3: Accountability: To facilitate a correct interpretation of the data, the statistical agencies are to present information according to scientific standards on the sources, methods and procedures of the statistics.
- Principle 4: Prevention of misuse: The statistical agencies are entitled to comment on erroneous interpretation and misuse of statistics.
- Principle 5: Sources of official statistics: Data for statistical purposes may be drawn from all types of sources, be they statistical surveys or administrative records. Statistical agencies are to choose the source with regard to quality, timeliness, costs and the burden on respondents.
- Principle 6: Confidentiality: Individual data collected by statistical agencies for statistical compilation, whether they refer to natural or legal persons, are to be strictly confidential and used exclusively for statistical purposes.
- Principle 7: Legislation: The laws, regulations and measures under which the statistical systems operate are to be made public.
- Principle 8: National coordination: Coordination among statistical agencies within countries is essential to achieve consistency and efficiency in the statistical system.
- Principle 9: Use of international standards: The use by statistical agencies in each country of international concepts, classifications and methods promotes the consistency and efficiency of statistical systems at all official levels.
- Principle 10: International cooperation: Bilateral and multilateral cooperation in statistics contributes to the improvement of systems of official statistics in all countries.
Few years later, in 2005, the Chief Statisticians or coordinators of statistical activities of United Nations agencies and related organizations, members of the Committee for the Coordination of Statistical Activities (CCSA), agreed that implementation of the following principles will enhance the functioning of the international statistical system:
- High quality international statistics, accessible for all, are a fundamental element of global information systems
- To maintain the trust in international statistics, their production is to be impartial and strictly based on the highest professional standards
- The public has a right to be informed about the mandates for the statistical work of the organisations
- Concepts, definitions, classifications, sources, methods and procedures employed in the production of international statistics are chosen to meet professional scientific standards and are made transparent for the users
- Sources and methods for data collection are appropriately chosen to ensure timeliness and other aspects of quality, to be cost-efficient and to minimise the reporting burden for data providers
- Individual data collected about natural persons and legal entities, or about small aggregates that are subject to national confidentiality rules, are to be kept strictly confidential and are to be used exclusively for statistical purposes or for purposes mandated by legislation
- Erroneous interpretation and misuse of statistics are to be immediately appropriately addressed
- Standards for national and international statistics are to be developed on the basis of sound professional criteria, while also meeting the test of practical utility and feasibility
- Coordination of international statistical programmes is essential to strengthen the quality, coherence and governance of international statistics, and avoiding duplication of work
- Bilateral and multilateral cooperation in statistics contribute to the professional growth of the statisticians involved and to the improvement of statistics in the organisations and in countries
Quality management (ISO)
According to ISO, managing quality is based on the following principles :
- Principle 1: Customer focus
- Principle 2: Leadership
- Principle 3: Involvement of people
- Principle 4: Process approach
- Principle 5: System approach to management
- Principle 6: Continual improvement
- Principle 7: Factual approach to decision making
- Principle 8: Mutually beneficial supplier relationships
- a strong commitment from the general management, expressed often in a quality declaration
- a quality strategy: identifying priority users and stakeholders and their requirements, the corresponding outputs, the policies regarding quality, a vision for the quality system, etc
- a quality management system as a layer of the general management, with a quality manager overlooking the system and reporting regularly to general management
- a quality improvement plan
- a monitoring mechanism, based on systematic documentation and using internal and external audits to identify corrective measures
Regarding procedures, one may consider two major principles:
- systematic recording (write what will be done – do what is rewritten – write what is done – analyse differences - correct)
- demonstrate and prove : demonstrate you intention by quoting documentation – prove the actual implementation again by quoting records in the documentation.
Data quality management framework - DQAF (IMF)
Developed initially by IMF, the DQAF may be seen as a monitoring tool.
DQAFs were built on the general structure of a statistical production process tailored to different macro-economics statistical data production domains and identified six dimensions:
- 0 Pre-requisites of quality
- 1 Integrity
- 2 Methodological soundness
- 3 Accuracy and reliability
- 4 Serviceability
- 5 Accessibility
- dimensions 0 and 1 are about enablers of the production of official statistics.
- dimension 2 is about the characteristics of the planned outputs (data of the domain)
- dimension 3 is about the production process itself run to produce the planned outputs
- dimension 4 is about the outputs as out of the process
- dimension 5 is about delivery of outputs and relations with users.
The overall purpose is not to conduct a detail technical audit of the statistics production processes but to assess how close the statistical outputs could be to the international recommendations (as the best proxy to users needs) and whether the corresponding processes are properly controlled and managed for changes; recommendations for an improvement plan is then derived from the assessment. It is assumed that actions will be taken to make the necessary corrections and this will be demonstrate and prove if requested.
The PARIS21 Task team on statistical capacity building indicators, led by IMF, used the DQAF early structure to design qualitative indicators for datasets; see more about PARIS21 statistical capacity building indicators.
South African statistical quality assessment framework - SASQAF (South Africa)
SASQAF has been developed for this purpose: it provides the framework and criteria used for evaluating and certifying statistics produced by government departments and other organs of state and, in some circumstances, by non-governmental institutions and organisations. Within the NSS framework, SASQAF draws a distinction between ‘official’ and ‘national’ statistics. National statistics refer to those statistics used in the public domain but which the SG has not certified as being official. Official statistics those statistics that have been certified by the Statistician General (SG) as being official in terms Section 14(7)(a) of the Statistics Act. Certification of statistics produced by organs of state involves a standard assessment procedure undertaken by a Data Quality Assessment Team (DQAT), established by the SG.
The DQAT is required to report on the statistics, classifying them as one of the following:
- quality statistics;
- acceptable statistics;
- questionable statistics; or
- poor statistics.
The main purpose of SASQAF is to provide a flexible structure for the assessment of statistical products. SASQAF can be used for:
- self-assessment by producers of statistics;
- reviews performed by a DQAT in the context of the NSS work;
- assessment by data users (e.g. financial market participants) based on the producing agency’s quality declaration;
- assessment by international agencies (e.g. the International Monetary Fund) based on the quality declaration.
National quality assurance framework - NQAF (UN)
The development of the Template for a Generic National Quality Assurance Framework (NQAF) and the Guidelines to accompany the Template was undertaken by the Expert Group on NQAF in response to a request by the United Nations Statistical Commission at its forty-first session in 2010. The Template is intended to be used as a tool to provide the general structure within which countries that choose to do so can formulate and operationalize national quality frameworks of their own or further enhance existing ones.
While there are several general definitions of quality, one of the most commonly used and succinct definitions is fitness for use or fitness for purpose. The ISO 9000 Quality Management System’s definition, cited in the SDMX Metadata Common Vocabulary and in the NQAF expert group’s Glossary, is the degree to which a set of inherent characteristics fulfils requirements.
A quality assurance framework is clearly just one of a number of other frameworks, policies and strategies that typically are in place in statistical agencies. They all should be developed and implemented in an integrated manner to achieve the agency’s mission and vision statements. The formulation of a quality assurance framework requires an in-depth and thorough review of those mechanisms most directly related to quality since the framework’s main focus is on the management of the core statistical functions.
Statistical laws, regulations and acts, codes of practice, and statistical standards, policies and strategies will need to be explicitly considered, referenced and made readily available in the process of drawing up a quality framework.
- Template overall structure:
- 1. Quality context
- 2. Quality concepts and frameworks
- 3. Quality assurance guidelines
- 3.a Managing the statistical system
- 3b.Managing the institutional environment
- 3c.Managing statistical processes
- 3d.Managing statistical outputs
- 4. Quality assessment and reporting
- 5. Quality and other management frameworks
- UN: Fundamental Principles of Official Statistics
- United Nations Statistical Commission
- Committee for the Coordination of Statistical Activities (CCSA)
- Wikipedia : Official statistics, quality criteria
- ISO 9000: Quality management principles
- IMF: Data Quality Reference Site, Data Quality Assessment Framework
- PARIS21 Statistical Capacity Building Indicators Task team
- South African statistical quality assessment framework - SASQAF
- UN: National Quality Assurance Frameworks