< Back to Main Activity Page

Best Practice Guidelines

Activity

EDAP Best Practice Guidelines

The EDAP project has defined a set of 'best practice' guidelines that define a framework for the quality assessments it performs, aligned with the principles of QA4EO. These guidelines are intended to provide a description of the high-level principles and activities that are required in quality assessments for all types of Earth Observation missions, since these are largely common between different domains.

At the top level, these guidelines have been drafted to reflect a near 'ideal' scenario, which is hoped to serve as an aspiration to new space providers as well as space agencies. It is understood that many of the missions the project will assess will only partly comply with these requirements (to different degrees). This is acknowledged in the assessment grading system which is intended to primarily test whether aspects of a given mission are 'fit for purpose' within the context of the mission's stated performance and application.

The product evaluation activity they undertake is divided into six sections:

  • Details
  • Generation
  • Quality Flags
  • Uncertainty Characterisation
  • Validation
  • Inter-comparison

These sections are themselves divided into sub-sections, which constitute each of the different aspects of the data product that should be assessed and graded, either as Basic, Intermediate, Good or Excellent.

The information is gathered in a Quality Assurance Report (QAR) and summarised in a product quality evaluation matrix.

EDAP Product Quality Evaluation Matrix – example
EDAP Product Quality Evaluation Matrix – example

Assessment Criteria

Product Details

Certain basic descriptive information should be provided with any EO data product, some of which is absolutely necessary for the data to be at all meaningful. Additionally, some of this information, such as claimed measurement quality or resolution, can provide both the data product quality assessor and the reader of their report with a frame of reference from which to set expectations for a given product.

It is therefore required by the EDAP assessment that any EO product provides the following:

  • Product name
  • Sensor Name
  • Sensor Type
  • Product version number
  • Product ID
  • Processing level of product
  • Measured quantity name
  • Measured quantity units
  • Stated measurement quality
  • Spatial Resolution
  • Spatial Coverage
  • Temporal Resolution
  • Temporal Coverage
  • Mission coverage

Also recommended is the following (based on INSPIRE metadata):

  • Point of contact (Responsible organisation, including email address)
  • Product locator (e.g. URL, DOI if applicable)
  • Conditions for access and use
  • Limitation on public access
  • Product abstract (summary of resource)

 

Table 1 – Product Details – Assessment Criteria

Table 1 shows how a data product's provision of the above information relates to the grade it achieves for this sub-section of the quality assessment.
Grade Criteria
Not Assessed Assessment outside of the scope of study
Not Assessable Relevant information not made available
Basic Any required information missing
Intermediate  
Good All required information available, any recommended information missing
Excellent All required and recommended information available

 

Uncertainty Characterisation

Uncertainty values should be provided in EO data products per-pixel, in a manner that describes the pixel error-covariance. Since it is not practical to provide a full error-covariance matrix for an EO data product due to their data volume various approaches have been developed to approximate this. For example, the FIDUCEO project FCDRs contain three components of uncertainty – independent, structured and common – to describe the three typical scales of error correlation.

It is still typical however for uncertainty values to be provided, if at all, on a per-product or, more often, a per-mission basis – losing a great deal of information significant to users.

 

Table 2 – Uncertainty Characterisation  – Assessment Criteria

Table 2 shows the EDAP grades missions and the extent of uncertainty information they provide.
Grade Criteria
Not Assessed Assessment outside the scope of study
Not Assessable No uncertainty information provided
Basic Single uncertainty value provided for whole mission
Intermediate Single uncertainty value provided for subsets of data, e.g. per product
Good Total uncertainty per pixel is provided, with basic breakdown of key components no error-covariance
Excellent Uncertainties per pixel provided with error-covariance information for all appropriate components

Mission Quality Assessment Downloads

Download the Best Practice Guidelines with detailed focus on Optical Missions and a template for the Mission Quality Assessment Report:

For information purposes only, previous versions used by old assessment may also be downloaded:

 

Version 2.0 of the Best Practice Guidelines provides an updated framework, where the reporting divides the assessment results between two Cal/Val maturity matrices, as follows:

  • Summary Cal/Val Maturity Matrix
  • Validation Cal/Val Maturity Matrix

Quality Assurance Framework

Framework for EO Data Quality Assessment

A generalised framework for assessing the fitness of purpose of EO data is difficult to obtain as this would clearly depend on the users intent; however, there are key characteristics that give confidence to a user and allow them to decide if the data is fit for their individual purpose.

In particular, users wish to understand data coverage, data provenance, data characterisation, data availability and also data correctness. Each of these factors, given a different weighting by each user, can ultimately affect a user's opinion of how good the data is.

EDAP aims to address both data integrity and data credibility for the various missions identified, which may go on to become TPMs within ESA's Earthnet Programme.

Quality Assurance Framework for Earth Observation (QA4EO) logo
 

QA4EO

The Quality Assurance Framework for Earth Observation (QA4EO) was created and endorsed by CEOS explicitly for the purpose of providing guidance on EO quality assurance so that data users are able to trust the data / information they are provided with is fit for purpose and fully interoperable with other datasets from both satellite and in-situ providers.

As there is potential for some commercial missions to become TPM candidates, for the most efficient exploitation of EO data, assessment of data quality, calibration and validation are indispensable tasks and also form the basis for reliable scientific conclusions. It is for this reason that one of the core functions provided by EDAP is focused on data quality, with an emphasis on the adoption of QA4EO guidelines.