Minimize L2 (Level 2) Product Format
Contents


General Structure of Product

The structure of the products produced for delivery to the PDGS follows the requirements of [AD-12] as represented in Figure 1 below.

Figure 1: General Product Structure

Figure 1: General Product Structure

Each product is composed of two physical files:

  • XML Header file having extension .HDR
  • Product file having extension .DBL

The XML Header file is an ASCII file containing data information that users can easily access for identifying the product without needs to look inside the product file.

It consists of:

  • Fixed Header, a common header for all files in the Swarm Ground Segment
  • Variable Header, including
  • Main Product Header (MPH) containing general information.
  • Specific Product Header (SPH) containing product specific information. The Specific Product Header will also contain references to auxiliary and/or input files relevant for the specific product.

The product file is the real product containing the processing results.

Back to Top



XML Header File

The XML Header file contains information identifying the product.

The Fixed Header (hereafter called Standard Swarm Header) is the common header for all files in the Swarm Ground Segment, which means it is applied to all files flowing amongst the sub-systems composing the PDGS.

The format of the Standard Swarm Header is under ESA responsibility and is specified in [AD-12] and [AD-13].

The Variable Header (hereafter called Product Header) is the header with format and content depending on the file type and kind of product.

The next sub-paragraphs specify the content of these headers.

Level 2 Products Fixed Header (Standard Swarm Header)

The Standard Swarm Header is completely ASCII and based on XML syntax and conventions proposed in [AD-12].

Field # Description Units Format
1 File_Name Tag  
Product file name (without extension)
(see Section 4.4 in [AD-15])
  55*uc
2 File_Description Tag  
Description such as mid column in Table 5-1 - Table 5-3 in [AD-15] (For CAT-2 AUX products, allowed entries are described in [AD-16].)   x*uc
3 Notes Tag  
This field will be empty.    
4 Mission Tag  
Will be "Swarm".   5*uc
5 File_Class Tag  
Type of processing; character 9 - 18 in the file name (see Section 4.4 in [AD-15]), this can be:

OPER = Routine operations
RPRO = Reprocessing

  4*uc
6 File_Type Tag  
Product name, first column in Table 5-1- Table 5-3 in [AD-15]   10*uc
7 Validity_Period Tag  
7.1 Validity_Start Tag  
Start time of validity period

UTC=yyyy-mm-ddThh:mm:ss

  23*uc
7.2 Validity_Stop Tag  
Stop time of validity period

UTC=yyyy-mm-ddThh:mm:ss

  23*uc
8. File_Version Tag  
Version number of the product

Start with 0001 and increases by one if following product versions are generated.

   %04d
9. Source Tag  
9.1 System Tag  
Name of the Ground Segment component:

L2PS: for CAT_1 products
APDF: for CAT-2 products

  4*uc
9.2 Creator Tag  
Name of Level 2 processor , such as CI for comprehensive inversion, or IBI for bubble index, SwarmLevel2Create for CAT-1 AUX products, ADF Preprocessor for CAT-2 AUX products.   x*uc
9.3 Creator_Version Tag  
Version of Level 2 processor VV.rr   5*uc
9.4 Creation_Date Tag  
Date of file creation:

UTC=yyyy-mm-ddThh:mm:ss

  23*uc

Table 1: Level 2 Fixed Header

Back to Top

Level 2 Products Variable Header (Product Header)

The XML Variable Header (hereafter called Product Header) for the Level 2 products is composed by:

  • an XML Main Product Header (XML MPH), and
  • an XML Specific Product Header (XML SPH)

Each header is completely ASCII and based on XML syntax and conventions proposed in the [AD-12].

The following sections describe the MPH and SPH for Level 2, Cat-1 Auxiliary products, and Cat-2 Auxiliary products AUX_LIT_2F, AUX_COR_2F and AUX_IGR_2F. MPH and SPH for all other CAT-2 AUX products are described in [AD-16].

Back to Top

XML Main Product Header (XML MPH)

The Main Product Header (MPH) has the following format - very similar to the Level 0 and Level 1b Main Product Header ([AD-5], [AD-14]), see also [AD-13]:

Field # Description Units Format
1 MPH Tag  
Following fields 1.1 - 1.9 are valid for all L2, Cat-1 Auxiliary products, and Cat-2 AUX_LIT_2F, AUX_COR_2F and AUX_IGR_2F 'Auxiliary products' (not for other Cat-2 Auxiliary products)
1.1 Product Tag  
Product file name (without extension) (see Section 4.4 in [AD-15])   55*uc
1.2 Product_Format Tag  
Format of the Product file:

CDF, PDF or ASCII

  5*uc
1.3 Proc_Stage_Code Tag  
character 9 – 18 in the file name (see Section 4.4 in [AD-15]), this can be:

OPER = Routine operations
RPRO = Reprocessing

  4*uc
1.3 Ref_Doc Tag  
Reference Document describing the product, shall be

SW-DS-DTU-GS-0001

  17*uc
Data Processing Information
1.4 Proc_Center Tag  
Processing Center ID code:

BGS = British Geological Survey
DEOS = DEOS
DTU = DTU Space
ETH = Eidgenössische Technische Hochschule
GFZ = GeoForschungsZentrum Potsdam
IPGP = Institut de Physique du Globe de Paris
PDGS = Payload Data Ground Segment

  4*uc
1.5 Proc_Time Tag  
Processing Time, UTC

(Product Generation Time)
UTC=yyyy-mm-dd Thh:mm:ss.uuuuuu

  30*uc
1.6 Software_Version Tag  
Processor Name and software version number ProcessorName/VV.rr

(Processor name as tag 9.2 of Fixed header)

  x*uc
Product Confidence Data Information
1.7 Product_Err Tag  
Product Error Flag. Set to 0 if product is fully ok, non-zero if any problems or errors are present   uc
Product Size Information
1.8 Tot_Size Tag  
unit="bytes" Attribute  
Total size of product bytes  %+021d
1.9 CRC Tag  
Cyclic Redundancy Code computed as overall value of all records of the Measurement Data Set.

Will be set to "-0000000001" if not computed.

   %+011d

Table 2: Level 2 Main Product Header (MPH)

Back to Top

XML Specific Product Header (XML SPH)

The formats of the Specific Product Headers (SPHs) are described. The SPHs are separately described for Level-2 Products (Cat-1 and Cat-2) and for CAT-1 AUX products. The SPH for CAT-1 AUX products applies also to the CAT-2 AUX products AUX_IGR_2F, AUX_COR_2F and AUX_LIT_2F. The SPH's for all other CAT-2 AUX products are described in [AD-16].

Following fields 1.1 – 1.6 are valid for L2 Cat-1 products
Field # Description Units Format
1 SPH Tag  
1.1 SPH_Descriptor Tag  
Product Name, first column of Tables 5.1 - 5.3 in [AD-15]   10*uc
1.2 Original_Filename Tag  
Name of original files, e.g, "*.SP3" for GPS orbit files   x*uc
Information on Time of Data
1.3 Sensing_Time_Interval Tag  
1.3.1 Sensing_Start Tag  
Start time in UTC of sensing data UTC=yyyy-mm-ddThh:mm:ss.uuuuuu   30*uc
1.3.2 Sensing_Stop Tag  
Stop time in UTC of sensing data UTC=yyyy-mm-ddThh:mm:ss.uuuuuu   30*uc
Product Confidence Section - various information on the quality of the product, such as number of missing or erroneous ISPs (Instrument Source Packets) and number of rejected or suspicious samples.
1.4 Product_Confidence_Data Tag  
1.4.1 Quality_Indicator Tag  
General product quality indicator, 000 if best quality is expected, 001 if the quality is acceptable.    %03d
List of Input files
1.5 List_of_Input_File_Names Tag  
count="n" Attribute  
This part is repeated n times, one for each input file
1.5.i File_Name   *uc
Name of processor input file i    
List of Output files
1.6 List_of_Output_File_Names Tag  
  count="n" Attribute  
This part is repeated n times, one for each output file
1.6.i File_Name   *uc
Name of processor output file i  
Following fields 1.1 - 1.5 are valid for L2 Cat-2 products
Field # Description Units Format
1 SPH Tag  
1.1 SPH_Descriptor Tag  
Product Name, first column of Tables 5.1 - 5.3 in [AD-15]   10*uc
Information on Time of Data
1.2 Orbit_Information Tag  
1.2.1 Sensing_Start Tag  
Start time in UTC of sensing data

UTC=yyyy-mm-ddThh:mm:ss.uuuuuu

  30*uc
1.2.2 Sensing_Stop Tag  
Stop time in UTC of sensing data

UTC=yyyy-mm-ddThh:mm:ss.uuuuuu

  30*uc
Information on manoeuvre
1.3 Maneuver_Information Tag  
count="n" Attribute  
1.3.i Maneuver_Id

i=1, ..., n

Tag  
The ith, distinct maneuver identification code[RD-3]    %03d
Product Confidence Section - various information on the quality of the product, such as number of missing or erroneous ISPs (Instrument Source Packets) and number of rejected or suspicious samples.
1.4 Product_Confidence_Data Tag  
1.4.1 Quality_Indicator Tag  
General product quality indicator

FAC_TMS_F and FACx_TMS_2F:
000 no problems at all occurred
001 quality_indicators is 001 in the L1b header of one of the MAGx_LR_1B
010 Flags(1) or Flags(2) is/are larger than 0 (data gaps occurred)
100 if one or all of Flags(3...8) are larger than 0 (problems with auxiliary data for magnetospheric field calculation occurred);

And combinations of quality indicators

IBIxTMS_2F:
000 no problems at all occurred
001 quality_indicators is 001 in the L1b header of one of the MAGx_LR_1B or EFIx_PL_1B
010 all of the Bubble_Index within the output are -1 (unanalyzable)

TECxTMS_2F:
000 no problems at all occurred
001 L1b header of one of the MODx_SC_1B or GPSx_RO_1B quality_indicators is 001
010 daily average of Relative_STEC_RMS or daily average of DCB_Error is larger than 2 TECU

EFFxTMS_2F:
000: no problems at all occurred
001: L1b header of one of the MAGx_LR_1B quality indicators is 001
010: Flags(4) is larger than 0 (data gap prevented processing)

   %03d
List of input files
1.5 List_of_DSDs Tag  
count="n" Attribute  
This part is repeated n times, one for each input file
1.5.i DSD Tag  
Data Set i descriptor, i = 1,2,…,n    
1.5.i.1 Data_Set_Name Tag  
Product name (see Table 5-2 Final Level 2 products in [AD-15])   10*uc
1.5.i.2 Data_Set_Type Tag  
Type of Data Set:

M - measurement
R - reference

  uc
1.5.i.3 File_Name Tag  
Name of Input File (without extension)   55*uc
1.5.i.4 Data_Set_Offset Tag  
unit="bytes"

Offset (in bytes) of first byte of first DS record within Product File.
Only used if Data_Set_Type = "M", otherwise set to zeros.

Attribute  %+021d
1.5.i.5 Data_Set_Size Tag  
unit="bytes"

Total number of bytes in DS Only used if Data_Set_Type = "M", otherwise set to zeros.

Attribute  %+021d
1.5.i.6 Num_of_Records Tag  
Number of Data Set records. Only used if Data_Set_Type = "M", otherwise set to zeros.    %+011d
1.5.i.7 Record_Size Tag  
unit="bytes"

Size of Data Set records
If variable set to -0000000001
Only used if Data_Set_Type = "M", otherwise set to zeros.

Attribute  %+011d
1.5.i.8 Byte_Order Tag  
Byte ordering information.

3210 → Big-endian
0123 → Little-endian
Only used if Data_Set_Type = "M", otherwise set to "0000".

  4*uc

Table 3: Level 2 Specific Product Header (SPH) - For L2 Cat-1 and L2 Cat-2 products

 

Field # Description Units Format
1 SPH Tag  
1.1 SPH_Descriptor Tag  
Product Name, first column of Tables 5.1 – 5.3 in [AD-15]   10*uc
Information on Time of Data
1.2 Validity_Time_Interval Tag  
1.2.1 Validity_Start Tag  
Start time in UTC of sensing data

UTC=yyyy-mm-dd Thh:mm:ss.uuuuuu

  33*uc
1.2.2 Validity_Stop Tag  
Stop time in UTC of sensing data

UTC=yyyy-mm-dd Thh:mm:ss.uuuuuu

  33*uc
Product Confidence Section – various information on the quality of the product, such as number of missing or erroneous ISPs (Instrument Source Packets) and number of rejected or suspicious samples.
1.4 Product_Confidence_Data Tag  
1.4.1 Quality_Indicator Tag  
Generel product quality indicator, 000 if best quality is expected. A different number than 000, if problems occurred during processing.    %03d
Originator information
1.5 Source Tag  
1.5.1 Original_Filename Tag  
Name of the original file before reformatting

Examples:
AUX_KP__2_ : kpyyyy.wdc
AUX_DST_2_ : dstyyyymm.ig2
AUX_F10_2_ : DAILYPLT.OBS
AUX_IMF_2_ : omni2_h0_mrg1hr_yyyymmdd_v01.cdf
AUX_OBS_2_ : *yyyy*.min
(if only obsyyyymmddqmin.min files as input)

*yyyy.wdc (if only obsyyyy.wdc as input)
wdc_mixed (if obsyyyymmddqmin.min and obsyyyy.wdc as input)

AUX_COR_2_ : freely chosen by the chain
AUX_LIT_2_ : freely chosen by the chain
AUX_IGR_2_ : igrf12coeffs.txt AUX_COR_2F : as for AUX_COR_2_
AUX_LIT_2F : as for AUX_LIT_2_
AUX_IGR_2F : as for AUX_IGR_2_
with
yyyy=year, mm=month and dd=day

  x*uc
1.5.2 Creator_Date Tag  
Date of creation of the original file

UTC=yyyy-mm-ddThh:mm:ss

  23*uc
1.5.3 Creator Tag  
Name of the external facility creating the original file, e.g.,

AUX_KP__2_ : Niemegk observatory
AUX_DST_2_ : WDC Kyoto
AUX_F10_2_ : NGDC NOAA
AUX_IMF_2_ : NASA
AUX_OBS_2_ : INTERMAGNET and WDC Edinburgh
AUX_COR_2_ : GFZ
AUX_LIT_2_ : IPGP
AUX_IGR_2_ : IAGA
AUX_COR_2F : as for AUX_COR_2_
AUX_LIT_2F : as for AUX_LIT_2_
AUX_IGR_2F : as for AUX_IGR_2_

  x*uc

Table 4: Level 2 Specific Product Header (SPH) - For CAT-1 AUX products and for AUX_IGR_2F, AUX_COR_2F and AUX_LIT_2F

Back to Top

Input Files

Input files to the Level 2 Processor (Level 1b and auxiliary products) used in the generation of the products are specified in the specific product header (XML Specific Product Header (XML SPH)).

Back to Top



Product File Formats

The L2 and Auxiliary Product File consist of data sets with specific formats that have been agreed during the SCARF Technical Meeting 2 on February 22/23, 2010. Five different formats exist:

  • ASCII Listing (SHC format)
  • ASCII Listing (SP3 format)
  • ASCII Listing (XML)
  • ASCII Listing (others)
  • CDF-format
  • PDF-files

The following section describes the format categories in more detail. The format category for the individual products is provided in Table 5-1 – Table 5-3 in [AD-15].

ASCII Listing (SHC format)

All core and lithosphere magnetic field model products (final and intermediate) obey the SHC format. Missing data values are marked with NaN.

The Gauss coefficients g and h of spherical harmonic models are given as snapshots at different time instants. A static field (i.e. the high-degree lithospheric field) is given as one snapshot; the core field is provided as a series of snapshot models.

Each file contains one or more blocks. Each of these blocks describes a certain range of Gauss coefficients with the same description of time dependence (i.e. either static, linear time dependent, cubic splines, or order-6 splines, or others).

Each block contains a header part and a coefficient part of length K = N_max*(N_max+2) – (N_min-1)*(N_min+1) lines with the Gauss coefficients stored row-wise in natural order (g10, g11, h11, g20 g21 h21, …). The first column of this coefficient part contains spherical harmonic degree n, the second contains spherical harmonic order m. Non-negative values of the order refer to the Gauss coefficients gnm while negative orders indicate the Gauss coefficients hnm. Rows 3 to N_times+2 of this coefficient part contains the Gauss coefficients of the N_times snapshot models, provided at the time instants t (given in decimal years).

Such a series of snapshot models is sufficient to completely describe a model if its time dependence is constant, a Taylor expansion in time, or a piecewise linear varying spline (order 2 spline).

In the case of a time dependence described by splines of order larger than 2 (e.g. cubic splines, i.e. spline of order 4), additional information is needed to reconstruct the time dependence from the N_times snapshot models. Within SCARF, following format applies: providing spline_order snapshot models for each knot interval (i.e. an "oversampling" of the model), indicated by the value N_step (which is equal to spline_order-1 in this case). This is explained in Table 4-5 SHA format used in SCARF in [AD-15]. Every N_step value of time t (in decimal years) are the spline knots, and therefore N_step = spline_order - 1.. It is possible to read the whole block using space-delimited free format read instructions.

Back to Top