SAFE - Background

The European Space Agency has been managing the payload data operation of a number of Earth Observation satellites since 1975. The activity includes acquisition, archiving, processing and product distribution of data from ESA and Third Party missions, for which more than 1.5 PetaBytes of data are presently archived. The activity is performed via a distributed network of facilities located in Europe and in Canada (for ERS only) mostly belonging to National and private entities, operating on behalf of the Agency via contractual agreements. The management centre of this network of facilities is located in the ESRIN ESA centre of Frascati, near Rome, Italy.

The ESA EO activity is currently progressing with the operation of various ESA and Third Party satellites: ERS-2, Envisat, Landsat 5/7, NOAA 16/17, SeaWiFS, Spot 2/4, Terra, Aqua, Proba. Future missions are also planned to fly, which will be managed by the ESRIN EO network of centres: Cryosat, GOCE, SMOS, ADM, GMES, etc., which will substantially increase the size of the long term archive.

The normal mandate for ESA's EO missions are to maintain the archive for at least ten years after the end of the mission. The management of this vast amount of heterogeneous datasets poses problems for their long term preservation, and this is why the EO Ground Segment Department has recognized for many years the need to establish a clear strategy for the management of the long term archives.

In 2001, ESA's Programme Board on Earth Observation (PB-EO) endorsed a strategy for the "Management of Historical Archives" (ref. PB-EO(2001)4). In 2003 DOSTAG (the technical advisory board of the ESA's PB-EO) endorsed a document ESA/PB-EO/DOSTAG(2003)6 related to the Promotion of products and services across missions and exploitation of historical archives. Furthermore, in 2003 the Oxygen implementation plan (ESA/PB-EO(2003)51) has reinforced and enhanced in a wider context the overall issue of data archive and improved data sources user access and availability.

Finally, in 2004 the ad hoc nominated PB-EO Ground Segment Task Force concluded its report with a set of recommendations (ESA/PB-EO(2004)53) related to the facilities and archives infrastructure aiming at:

ESA's Earth Observation Department has since that time recognized that the main process to be undertaken is the standardization and harmonization of its ground segment architectures to reach scale economies during development, operations and maintenance. This includes the need to achieve the following goals:

archive maintenance in order to ensure data integrity
archive and data management rationalization
data conversion to new technologies in order to reduce cost of operations
enhancement of data access
standardization of the format in which the datasets are preserved

By achieving these goals, the road would be also paved for simplifying the exchange and interoperability of data between ESA and external operators.

It is well known that the data to be preserved for the long term require a special attention, which is reflected in costly operations for their exploitation and maintenance. Among these challenges:

the datasets have to be regularly converted into new media technology, to prevent the problems created by their obsolescence;
since the long term archive is normally based on datasets archived usually only up to a very low processing level, normally called L0, higher level products have to be generated by processing systems;
in addition, in the case of distribution of the data holdings directly in the archived format, it is normal practice that they are converted or reformatted into a format more oriented to the end-user utilization;
it is a common requirement to have to extract from the long term archive a portion of the single data file (subsetting) to create a "child" product, optionally to be pre-processed
finally, more and more data are distributed to end-users and exchanged among data holders in electronic format over network infrastructure (private Intranets, public Internet, academic networks, etc.)

One of the reasons that contribute to the high operations and maintenance costs of the long term archive is the excessive proliferation of diverse and heterogeneous data formats, caused by mainly three reasons:

The lack of an agreed standard in the Earth Observation community, reason for which the formats tended to be specific for the sensor(s) each missions carried on board.
Legacies from old ground segments architectures, which tended not to reuse elements previously developed.
The non-mature status, until recently, of the information technologies and standards used to describe and package the data, preventing the creation of a unique format able to satisfy at the same time the requirements for the long-term preservation of the data and their handling in the processing centres.

Taking all this into account, in early 2004 ESA set up a project called HARM (Historical Archives Rationalization and Management), which aimed mainly at converting its historical datasets into a new modern format, based on the latest technologies and standards and able to ensure the long-term preservation of its holdings.
The format developed by the HARM project was named Standard Archive Format for Europe (SAFE).