The DQAF Manual
The UNESCO Institute for Statistics (UIS) is mandated to develop and implement evaluation methodologies that assess the quality of data produced by national statistical systems within the UNESCO domain of competence. The Institute has been active in using such standards and tools for assessing education data quality produced by member states.
The Education Data Quality Assessment Framework (DQAF) is an instrument initially developed by the International Monetary Fund in 2002 to assess the quality of economic data . In 2004, the World Bank and UIS modified it for use in the evaluation of education data .
For several years, UIS has engaged in diagnostic assessments of national education statistics systems using the DQAF methodology. Such ‘DQAFs’ were implemented in Latin American and in Sub-Saharan Africa (SSA) between 2005 and 2006.
Between 2008 and 2011, within the UNESCO Office in Dakar (BREDA) support to the African Union Second Decade of Education action plan , the SADC education programme and the ECCAS, a second round of seventeen diagnoses has been conducted in SSA with some modifications in the DQAF methodology itself and in its implementation.
When conducting these exercises, several inconsistencies in the structure of the instrument and in the relevance of some items as well as the need for standard scoring guide lines and other methodological issues have been identified.
At several occasions, the UIS together with other partners have conducted revisions of the methodology in order to improve it at the light of additional experiences (Read More... Main evolutions in the DQAF Methodology). The last revision is the result of the DQAF expert review meeting, Cape Town, South Africa, December 2010. One of the recommendations of this meeting was to elaborate a manual that could be used as a guide for conducting country diagnosis on the basis of the framework and that would be part of the UIS DQAF methodology.
Presentation of the UIS DQAF
More than an assessment of the quality of data itself (See: The Quality of statistics), the DQAF draws a picture of the environment and conditions through which data are produced.
After having been piloted in several countries and revised at the light of these experiences, the Framework is now stabilised and has become the UIS official Education DQAF.
The evaluation framework covers the different steps included in the statistical business process model at the national and sub-national levels, and assesses the strengths and weaknesses of the available structures, based on the six DQAF dimensions:
0 Pre-requisites of quality - 1 Integrity - 2 Methodological soundness - 3 Accuracy and reliability - 4 Serviceability - 5 Accessibility
The assessment provides a comprehensive evaluation of the quality of education data by comparing country’s practices with current international standards.
To do so, the Framework is organized in a cascading structure that progresses from the abstract/general to the more concrete/specific. The first-digit level defines the six dimensions presented above. The first-digit level is sub-divided by sub-dimensions (two-digit level) and indicators (three-digit level). At the next level, practices describe quality features that may be considered in assessing the indicators. The practices are meant to be suggestive, not exhaustive.
Using Accuracy and reliability as the example of a dimension of quality, the picture below shows how the framework identifies four sub-dimensions that point toward quality. Within Statistical techniques, one of the sub-dimensions , the framework next identifies two indicators. Specifically, for one of these, Data compilation, quality is assessed by considering four specific practices.
The essential part of the assessment consists in scoring each of these practices according to a scale from 1 to 4: Practice not observed (scored 1), Practice largely not observed (scored 2), Practice largely observed (scored 3), Practice observed (scored 4).
Read More... The UIS Education DQAF
The fact finding mission
The fact finding mission conducted to the country which education data quality is assessed repressents the most important part of the diagnosis. The success of the overall operation relies on it. It has to be prepared well in advance and needs to be conducted following a strict organisation.
The scope of the fact finding mission needs to be clearly identified. It will depend on the context of the intervention and on the national education system structure.
In the context of UNESCO support to 2nd Decade for education for instance and in most of the DQAF conducted by UIS so far, the whole education sector was investigated. In some cases however, the diagnosis may be giving priority to a specific sub-sector. In the case of Benin for instance, which DQAF will be conducted in the context of UNESCO CapEFA initiative, mainly the TVET sub-sector will be targeted.
This part of the manual is intended to guide the external evaluators in understanding what is expected from the mission, how to prepare it and how to conduct it. It provides some tools and instruments that can be downloaded and guide lines for scoring.
Read More... The fact finding mission
Writing the report
Writing the report is a heavy task that can be completed within a minimum of one month work by the expert team.
- Suggested report structure
- After a series of DQAF reports written and methodology review meetings conducted (See: Main evolutions in the DQAF Methodology), the UIS recommends that DQAF reports are developed according to the below structure. Examples of reports using this structure can be found at DQAF fact finding missions.
- 1. Background: Presents the context in which the DQAF has been conducted. The last DQAFs were conducted within UNESCO/UIS support to AU 2nd decade of Education / SADC EMIS development strategy.
- 2. Methodology used for the DQAF: A short presentation of the methodology: the six dimensions, scoring principles, data collection process (interviews, documents analysis, etc.), identification of the data sets / data producers, etc.
- 3. Objectives of the DQAF Assessment
- 4. Main findings: Examples of good practices, and opportunities to improve or strengthen current practices for optimal efficiency, are developed in this section.
- 5. Recommendations: It is recommended that the recommendations stemming from the analysis are not grouped according to the DQAF dimensions but in "general" categories: institutional, organizational, technical and capacity building for instance. However, reports writers will adapt these categories according to the context, bearing in mind that the objective is to make it easier the reading for decision makers.
- 6. Overview and functional structure of the education system: The education system is presented here identifying the main actors and responsibilities for each sub-sector.
- 7. Data collection process(es): This is a description of how data collection and processing is conducted under the responsibility of the identified data producers.
- 8. Data Quality Assessment Framework: Narrative descriptions are given of the state of the system as per each dimension. Significant information, gathered for each item (practice) is developed here.
- 9. Conclusion: A graph summarizing the scoring is provided and commented (See: guide lines for scoring).
Steps forwards for validation of the report and implementation of recommendations are also presented here.
- Reviewing the report
- A first draft of the report is sent to the country authorities requesting to circulate the report and provide comments by a proposed deadline; a draft is proposed below  .
- The main respondent should organise the review at national level and consolidate inputs from the different entities invited to react. The content of the report is discussed in details and relevant changes are incorporated until final agreement is met with technicians.
- Once validated, a final version of the report is prepared with last changes. The report can then be duplicated and disseminated. It can also be integrated in this Wiki for public upload.
- See IMF DQAF Factsheet
- See DQAF Education 2004
- See AU 2nd Decade AP
- See Cape Town Expert Meeting Report
- See Letter for submitting DQAF report
- UNESCO Office in Dakar or Bureau Régional de Dakar (BREDA)
- ECCAS Economic Community of Central African States Website or ECCAS on Wikipedia
- SADC Southern African Development Community Website or African Development Community on Wikipedia
- African Union Website or African Union on Wikipedia
- UNESCO Capacity Development for Education for All - CapEFA