Minimize Data Quality for L1B and L2 Cat-2 data

Contents

1.      Definition of L1b and L2 Cat-2 Product Baseline and Dataset

2.      Current Dataset

2.1        Orbit and Attitude Products

2.1.1        Important Notes

2.1.2        Quality Information

2.2        Magnetic Products

2.2.1        Important Notes

2.2.2        Quality Information

2.3        Plasma Products

2.3.1        The Preliminary Plasma Dataset: Description and Quality Information

2.3.2        The Plasma "Current" Dataset: Quality Information

2.3.3        The Extended Langmuir Probe Dataset: Description and Quality Information

2.3.4        The 16 Hz electron density derived from Faceplate current: Description and Quality Information

2.3.5        The 2 Hz TII Cross-Track ion flow dataset: Description and Quality Information

2.4        Accelerometer Products

2.5        Level 2 - Cat2 Products

2.5.1        Important Notes

2.5.2        Quality Information

3.      Previous Datasets

3.1        Product Baseline: 03 (Validity: 25/11/2013 - 17/07/2015)

3.1.1        Magnetic Products

3.2        Product Baseline: 02 (Validity: 15/04/2014 - 12/05/2014)

3.3        Product Baseline: 01 (Validity: 25/11/2013 - 14/04/2014)

3.3.1        Level 2 - Cat 2 Products

 

1.    Definition of L1b and L2 Cat-2 Product Baseline and Dataset

CURRENT FOLDER: the user will find all the latest interoperable products in the 'Current' folder on the ESA FTP server. These data have been generated in a consistent way with the application of all significant data quality improvements.

PREVIOUS FOLDER: data no longer interoperable with the latest existing products are moved to the 'Previous' folder on the ESA FTP server. Such data is identified with an older Product Baseline number.

N.B. This strategy ensures that users do not mix old data with new products that have been improved in quality by significant upgrades. It might occur that not all of the old data is immediately regenerated or reprocessed into the Current Folder. Therefore if the user does not find the improved product in the Current Folder but s/he is nonetheless interested in accessing the data without the significant data quality improvements, s/he can find them in the Previous Folder.


Please refer to the following sections to check the Product Baselines details of the 'Current' and 'Previous' Level 1b and Level 2 Cat-2 Datasets.


PRODUCT BASELINE: is a number associated to a specific product type and satellite. The Product Baseline is identified by the first two of the four digits placed at the end of the file name, i.e. the first two digits of the File_Version field represent the Product Baseline, while the last two represent an incremental File Counter.

Example:
SW_OPER_STRCATT_1B_20140218T000000_20140218T235959_0301_MDR_SAT_AT.cdf Where 03 is the Product Baseline number, while 01 is only an incremental File Counter.

The Product Baseline is increased when upgrades lead to significant improvements in the data quality of the related products. This subsequently leads to a reprocessing of the affected product types. Data with previous Product Baseline are moved into the Previous Folder. Products with an older File Counter but same Product Baseline are erased, with few exceptions documented in the following Sections.

Please note that a Dataset may contain products with different Product Baselines, as shown in the example of Figure 1 below.

Figure 1: Schematics of a Swarm "Dataset"

Information on the Level 1B and Level 2 Category 2 Swarm products, can be found on Swarm Level 1b Product Definition and Swarm Level 2 Product Specification

For information on how to access and download Swarm data, please refer to the Data Access page.

 

2.    Current Dataset

The tables below present the "Current" Level 1b and Level 2 Cat-2 Datasets with their respective Product Baselines. Detailed explanations of the most relevant data quality features are given in the following sections.

Table 1: L1B Product Baselines included in Current Folder

  Sat Alpha Sat Bravo Sat Charlie
Level 1b Product Baseline Validity

Product Baseline

Validity

Product Baseline

Validity

Orbit and Attitude products (ORBATT)

03

From 25/11/2013 to present

03

From 25/11/2013 to present

03

From 25/11/2013 to present

1 Hz Magnetic products (MAGx_LR_1B)

04

From 25/11/2013 to present

04

From 28/11/2013 to present

04

From 04/12/2013 to present

50 Hz Magnetic products and other auxiliary magnetic products (MAGx_HR_1B, MAGx_CA_1B, VFMxAUX_1B, ASMxAUX_1B)

04

From 18/07/2015 to present

04

From 18/07/2015 to present

04

From 18/07/2015 to present

Plasma products (EFIx_PL_1B)

04

From 18/07/2015 to present

04

From 18/07/2015 to present

04

From 18/07/2015 to present


Table 2: L2 Cat-2 Product Baselines included in Current Folder

 

Sat Alpha

Sat Bravo

Sat Charlie

Level 2 - Cat2

Product Baseline

Validity

Product Baseline

Validity

Product Baseline

Validity

Field Aligned Currents – Single

02

From 26/11/2013 to Present

02

From 28/11/2013 to Present

02

From 04/12/2013 to Present

Field Aligned Currents - SW A and C combined

02

From 15/14/2014 to Present

02

N/A

02

From 15/04/2014 to Present

Ionospheric Bubble Index

02

From 09/12/2013 to present

02

From 09/12/2013 to present

02

From 11/12/2013 to present

Total Electron Content

02

From 25/11/2013 to present

02

From 25/11/2013 to present

02

From 25/11/2013 to present

Equatorial Electric Field

01

From 27/08/2015 to present

01

From 27/08/2015 to present

N/A

N/A

 

2.1          Orbit and attitude Products

2.1.1    Important Notes

  • From beginning of mission (25/11/2013) up to 20 October 2014 the following GPS and orbit products: GPSx_RO_1B, GPSx_RN_1B, GPSxNAV_1B, MODx_SC_1B, are included in the "Current" folder with two different file counters, 01 and 02.
    This is due to a partial regeneration of such products after the introduction of algorithmic modifications better detailed in Sect. 2.1.2. Contrarily to the usual conventions, the products with the old file counter 01 have not been removed, in order to guarantee the consistency with a set of magnetic products contained in the "Previous" folder as detailed in Sect. 3.1.1.1.
  • From 19 January 2017 the following changes have been introduced in the Swarm Orbit and Attitude data production:
    • The GPS related L1B products (GPSx_RO_1B and GPSx_RN_1B) are delivered in RINEX format as usual, but the extension changes from .DBL to ".rnx"
    • The Ephemerides (position) related Swarm L1B products (MODx_SC_1B and STRxATT_1B) are delivered in SP3c format as usual, but the extension changes from .DBL to ".sp3"
    All these changes are applicable from 19 Jan 2017 onwards. There are no changes to the previously disseminated files.
  • All products included in the "Current" folder (e.g. MAGNET, PLASMA, L2 Cat-2) are interoperable ONLY with orbit products with File Counter 02.

2.1.2    Quality Information

  • On 20 October 2014, some modifications have been introduced in the way GPS products are computed:
    • Deactivation of antenna and receiver corrections
    • Deactivation of the carrier smoothing algorithm that uses carrier range observations to smooth code range observations
  • Following such modifications, all GPSx and MODx_SC_1B products from the beginning of the mission have been regenerated with File Counter 02, as described in Sect. 2.1.1.
  • The Medium-accuracy Orbit (MOD) calculation has been improved. Within the products contained in Previous datasets, the rejection criteria for the GPS observations were too strict, which often caused a large number of observations to be discarded and, consequently, degraded accuracy of the orbit solution. Now, in the Current dataset products, the rejection criteria have been relaxed and the solution is more stable. Moreover, new steps have been introduced in the MOD, which make the processing of MOD products more robust.
  • Starting from 10 April 2016, a correction has been introduced when decoding the Instrument Measurement Time (IMT) from raw GPS observation data into Level 1B Rinex products (GPSx_RO_1B file types). The IMT scale was not read with the right precision, and this introduced a high level of noise in the pseudo-range gradients. As the impact on the final orbit determination is negligible, previous data will not be regenerated right now.
  • Due to an error in the quaternions computation, the q(:,4) element of the STRxATT_1B products was not correctly set to zero in case of data gap (Flags_q = 255). In the Current dataset products this issue has been fixed.
  • STRxATT_1B attitude products: The conditions for considering single cameras attitudes samples for combination have been relaxed such that the "Sequence" flag is no longer checked. This implies that attitudes computed on-board the spacecraft using the lost-in-space algorithm are used on equal terms as others. Moreover, attitudes with BBO ("Big, Bright Objects") flag set are also considered, though with additional conditions. The update of the algorithm is expected to decrease the occurrence of attitudes based on a single camera head unit (CHU), so improving overall the quality of the attitude determination.

 

2.2          Magnetic Products


2.2.1    Important Notes

  • The magnetic data MAGx_LR_1B are available with Product Baseline 04 from beginning of mission to present. The following magnetic data: MAGx_HR_1B (50 Hz vector field), MAGx_CA_1B, VFMxAUX_1B and ASMxAUX_1B are available with Product Baseline 04 only starting from 18/07/2015 on. Previous dates are available in the "Previous" folder of the ESA FTP server and, as described in Sect. 3.1.1, they are NOT CORRECTED from external disturbances effects and they are therefore NOT interoperable with Magnetic data with Product Baseline 04 contained in the "Current" folder. For the same reason, the MAGx_LR_1B (1 Hz scalar and vector field data) with Product Baseline 03 contained in the "Previous" folder, SHALL NOT be used together with the MAGx_LR_1B with Product Baseline 04 contained in the "Current" folder.
  • The ASM instrument on Swarm Charlie stopped providing data on 5 November 2014. The failure is likely permanent. This required a major review of the magnetic calibration for Swarm Charlie, as briefly described in Sect. 2.2.2, point 3.


2.2.2    Quality Information

The improvements in the new Product Baseline of magnetic data have been made possible with the joint efforts of a dedicated group of experts, who analysed instrument and housekeeping data and elaborated a model for correcting data from external disturbance effects, predominantly Sun-driven and manifested as scalar residuals between the ASM and VFM data. After the application of this correction model, we have reduced significantly the differences between ASM and VFM data. The overall rms of such differences were around 1 nT before the correction while we have obtained a rms around 200 pT after the correction, for all spacecraft. The expert group is still working to identify the exact root cause of the disturbance and possibly further improve the disturbance models.

The key points are the following:

  1. Early in the Swarm mission it was recognised that the measurements from the two magnetometer instruments onboard each spacecraft slightly differed. The effect has been analysed and interpreted as a magnetic disturbance of varying strength and direction. Such disturbances are highly correlated with the Sun incident angle with respect to the spacecraft and are manifested as scalar residuals between the ASM and VFM data. The experts endeavoured to model the disturbance parameterized in terms of a spherical harmonic model as a function of Sun incident angles, and this model has proven to provide excellent results.
  2. In the current processing strategy, the calibration parameters for the vector measurements estimated on a daily basis are replaced by the co-estimation of the adjustment of the VFM pre-flight calibration parameters and the magnetic vector disturbance model. This disturbance model (the so-called "dB_Sun)" is considered as a stray field vector at the VFM instrument location. A detailed description of this correction model can be found here.
    The applied dB_Sun for the correction is then stored in the following magnetic products: MAGx_HR_1B, MAGx_LR_1B and MAGx_CA_1B.
  3. In order to cope with the permanent failure of the ASM onboard Swarm Charlie, the magnetic scalar reference measurements for the disturbance model characterisation and VFM calibration described above have successfully been generated using the ASM measurements from Swarm Alpha. For more detailed information, please refer to the Scalar Mapping Technical Note.
  4. Due to an old bug in the L1b data processor, the ASM stray fields, dF_AOCS and dF_other, stored in the Swarm MAGx_LR_1B and MAGx_CA_1B data with Product Baseline 03 and 04 from the beginning of the mission to 21/02/2015 have not been computed correctly. Please note that this issue has no impact on the full correction of the ASM data from the total of the stray fields (dF_Total) which is indeed implemented and applied in the right way.


2.2.2.1  Original Residuals


Due to recent modifications in the disturbance model in the operational processor, it is not possible to recover the original BVFM values and, as a consequence, the original ASM-VFM residuals from the magnetic products (MAGx_LR_1B, MAGx_HR_1B, MAGx_CA_1B) available in the "Current" Dataset. Therefore in order to facilitate the data validation, ESA has made available the original non-corrected magnetic field vector data, in VFM frame (BVFM).

Such data can be found in the ESA FTP server at the following path:

~/Advanced/Magnetic_Data/ASM_VFM_Residuals/Sat_x

where x=A,B,C is the spacecraft identifier.

This dataset shall be used by all those scientists interested in analysing the non-corrected ASM-VFM scalar residuals.


2.2.2.2  Euler Angles Update

 

On 07/12/2015 an update of the rotation angles between the VFM instrument reference frame and the STR common reference frame (Euler angles) has been introduced. All the MAGx_LR_1B and MAGx_HR_1B products contained in the Current folder have been regenerated accordingly, leading to an increase of the File Counter. The only difference with the old File Counter (i.e. Product Datasets with Baseline 04 and File Counter less than 08) can be observed in the BNEC which is the parameter affected by the Euler angles transformation: the maximum difference is slightly above 1 nT, most pronounced on the BEast component. The products with File Counter less than 08 in the "Current" Dataset have been removed.


2.2.2.3 MAGx_LR_1B dataset updated


On 05 January 2017 an updated version of the Swarm MAGx_LR_1B products (Product Baseline and File Counter 0501) have been made available in the "Advanced" folder of the ESA FTP server. This dataset covers the time window from the beginning of the mission to the end of 2016 for all three spacecraft. The new dataset includes:

  1. the Star Tracker temperature correction
  2. an update of Star Tracker interpolation and
  3. an update of the vector data calibration and disturbance characterisation

Due to technical issues in the processing of these Baseline 05 data there are gaps not present in the Baseline 04 products. The gaps will be recovered in the next full mission data reprocessing.

 

2.3  Plasma Products

Currently four main datasets exist for Swarm Plasma products:

  • A Preliminary Plasma Dataset: it is the result of the efforts of the Swarm Electric Field Instrument (EFI) Principal Investigators (PIs) and their teams, and is meant to partly replace the official Ddtaset when it is not available. It is described in Sect. 2.3.1.
  • The Plasma "Current" Dataset: it consists of the official Swarm plasma products as from Swarm Level 1b Product Definition, available in the "Current" folder of the ESA FTP server. It is described in Sect. 2.3.2.
  • An "Extended" Langmuir Probe dataset: this is an experimental dataset where separate measurements from the single probes are given, in order to investigate open issues on data and for data validation purposes. More details are given in Sect. 2.3.3.
  • Electron density derived from Faceplate current: this is an experimental dataset which offers a different evaluation of the electron density, based on the measurements of the faceplate current at 16 Hz rate. More details are given in Sect. 2.3.4.


2.3.1    The Preliminary Plasma Dataset: Description and Quality Information

Temporarily, in addition to data contained in the 'Current' folder, a preliminary set of Swarm Plasma data for all three spacecraft has been made available to the Swarm science users thanks to the efforts of the Swarm Electric Field Instrument (EFI) Principal Investigators (PIs) and their teams.

This dataset is actually composed of two separate datasets, corresponding to the two instruments which compose the EFI: the Langmuir Probes derived data (plasma density, electron temperature and spacecraft potential) have been prepared by the Swedish Institute for Space Physics (IRF) at Uppsala; the Thermal Ion Imagers derived data (O+ bulk velocity and temperature, ionospheric convection electric field) have been prepared by the University of Calgary (Canada).

Due to thorough calibration activities on the instruments and the processing algorithms evolutions still ongoing to achieve the best possible data quality, this plasma dataset is a work in progress and can be considered as preliminary. The use of such data by the scientific community is encouraged by the EFI PIs, provided interested scientists and users carefully read the User Note concerning the state of the art of plasma data quality and the information about data formats and content. In case of doubts, the PIs will be pleased to answer your questions:

Please always include ESA in CC in your communications: swarm@eo-sppa.org

The dataset is available on the ESA FTP server accessible by all users, under the path: ~/Advanced/Plasma_Data/Provisional_Plasma_dataset

The dataset is structured as follows:

  • Thermal_Ion_Imagers_Data: Due to anomalies in the raw TII images and consequent difficulties in performing a continuous calibration of the instrument, the instrument experts have made available selected periods when data quality is certified and data can therefore be used for science studies. Such periods could be possibly extended in the future, depending on the progresses made in the instrument investigations and in the data calibration. The available data are in daily files, CDF format.
  • Langmuir_Probe_Data: This dataset is produced in daily files in ASCII format by the Langmuir Probes Instrument experts and is continuous from the instrument first light in early December 2013 up to 17 July 2015. From 18/07/2015, official Swarm plasma products, EFIx_PL_1B, are produced. The two datasets are generally interoperable, taking into account the following points:
    • Timestamps. The timestamps in the preliminary dataset correspond to the true Langmuir Probe measurement times, while the CDF epochs in the EFIx_PL_1B shall be incremented by a time lag "dt_LP" as described in Sect. 2.3.2.
    • S/C position and velocity are calculated differently in the two datasets: in the preliminary dataset the coordinates (latitude, longitude, height and radius) are calculated from Two-Line Elements predicted orbits and are geodetic, while in the EFIx_PL_1B the input data for calculating coordinates are the very Swarm GPS position measurements, and coordinates are geographic (NEC system). This has two consequences: 1) The difference in geolocation of the measurements between the two datasets is about 13-14 km; 2) the S/C velocity enters in the calculation of the electron density, therefore the difference in the absolute value of the density between the two datasets can be up to 103 m-3, which is about two orders of magnitude below the lowest "physical" value one can measure in the ionosphere.


2.3.2    The Plasma "Current" Dataset: Quality Information

Official Swarm Plasma products are available since 18/07/2015. The user is recommended to read carefully the following notes before using the data:

  1. EFIx_PL_1B only includes plasma parameters derived from the Langmuir Probes: electron density (N), electron temperature (T_elec) and S/C potential (U_SC). All the plasma parameters derived from the Thermal Ion Imagers (v_ion, v_ion_error, E, E_error, T_ion, T_ion_error, v_ion_H, v_ion_H _error, v_ion_V_error, rms_fit_H, rms_fit_V, var_x_H, var_y_H, var_x_V, var_y_V, dv_mtq_H, dv_mtq_V, Flags_TII) are set to zero. Selected TII data periods with good quality are distributed in the TII preliminary dataset (see Sect. 2.3.1).
  2. The correct Langmuir Probe epochs, ti, shall be computed as follows: ti = CDF_epoch + dt_LP, where dt_LP is a time lag parameter (given in seconds), also contained in EFIx_PL_1B products.
  3. TIIx_CA_1B products containing the daily determined TII detector centres are currently not distributed, because the occurrence of the TII image anomaly still does not allow an automated calibration process.
  4. Information on the quality of the Langmuir Probes derived parameters contained in the EFIx_PL_1B products can be found in the same technical note associated with the preliminary plasma dataset. Statistically, both the electron density and temperature behave as expected, and the general features of the ionosphere can be recognised. The electron temperature is still prone to possible overestimates in regions where the density is low, and high temperature shears and spiky events are often observed.
  5. The Langmuir Probes work in sweep mode for one second every 128 seconds. The sweep mode data are still under validation and are not included in the EFIx_PL_1B products. In order to guarantee data continuity, the sweep mode records are replaced with a replica of the preceding normal mode records. However the timestamps when the sweep mode occur are flagged with Flags_LP = 9.
  6. The Sweep mode itself seems to influence the following two-three measurements in normal mode, introducing small hicks up in the spacecraft potential and, to a lesser extent, in the electron temperature. Such small dips, where the magnitude is a few per cent of the absolute value of the affected parameters, should be filtered/interpolated away. The cause of such effect is still unknown; the most likely hypothesis points to a local charging of the probes induced by the voltage sweep, which requires a couple of seconds to decay towards the baseline level.
  7. The Langmuir Probe data calibration and validation requires a huge statistics of conjunctions with ground-based measurements (above all, incoherent scatter radars which measure the same parameters at roughly the same height as the Swarm S/C), so that, overall, it is not yet possible to associate reliable errors or quality flags to data. Therefore all the error parameters in the EFIx_PL_1B products are set to fixed values: N_error = 429496729.5, T_elec_error = 429496729.5 and U_SC_error = 32.767.
  8. The quality flag for the electron density (Flags_LP_N) is set to "20" for the whole dataset, which simply means data have to be used with caution and EOhelp should be contacted in case of doubts. The quality flags for the electron temperature and the S/C potential (Flags_LP_T_elec, Flags_LP_U_SC) can show values greater than 20: in such cases data are not usable at all for science purposes and shall be discarded.
  9. It has recently been noticed that the electron density contained in the EFIx_PL_1B products are at times affected by spike events, or overflows, of about 4.2 x 108 cm-3, whose duration usually does not exceed 1-2 seconds. This is due to a technical bug in the data processor which mostly affects the calculation in regions of the ionosphere characterised by very low electron densities. The lower spacecraft pair (Swarm A and C) is the most affected, while on Swarm B data such overflows occur very seldom. For the purpose of science analyses, such artefacts can be safely filtered away and remaining gaps can be linearly interpolated.


2.3.3    The Extended Langmuir Probe Dataset: Description and Quality Information

This dataset has been delivered by the Swedish Institute of Space Physics (IRF) for instrument validation purposes and advanced studies involving ionospheric plasma. It is intended to be an experimental dataset, and before using such data in papers or other scientific publications, please contact IRF (Dr. Stephan Buchert, scb@irfu.se).

The Swarm payload includes two Langmuir Probes, each carrying on independent measurements of ionospheric electron density, electron temperature and spacecraft potential. Then, the measurements from the two probes are "blended" together according to an algorithm which takes into account several environmental and instrument variables: different sensitivity of each probe to different plasma density regimes, saturation due to instrument factors, and more.

The final results are single values of electron density and temperature and spacecraft potential as a function of time, contained in the official Swarm plasma product type EFIx_PL_1B, and in the Langmuir Probe preliminary dataset described above (Sects. 2.3.1 and 2.3.2)

One of the major challenges the Swarm experts are presently facing is the validation and correct interpretation of the electron temperature and spacecraft potential measurements. Several features are being observed such as: unusually high temperature regions, trains of spikes apparently related with Sun illumination angle on the Swarm solar panels, small jumps in the spacecraft potential after the activation of the sweep mode.

Therefore, in order to help scientists in investigating these peculiar aspects, an extended dataset has been released, containing the electron temperature and spacecraft potential measured separately by the two probes. Data are in format of daily CDF files, spanning, for the time being, from the start of the mission to 11/06/2016 for all three spacecraft. Further updates are regularly expected.

Data are available in the ESA FTP server under the path:

~/Advanced/Plasma_Data/2_Hz_Langmuir_Probe_Extended_Dataset/Sat_x

Where X=A,B,C is the spacecraft identifier.

Before using this data, please read carefully the Release Note


2.3.4    The 16 Hz electron density derived from Faceplate current: Description and Quality Information

An extended dataset including ionospheric electron density at 16 Hz has been made available by the Swedish Institute of Space Physics (IRF Uppsala). The density is derived from the current measured at the EFI faceplate and offsets can be present with respect to the 2 Hz plasma density measured by the Swarm Langmuir probes.

Before using this data, please read carefully the Release Note, and, in case of doubt, please contact IRF for further clarifications (Dr. Stephan Buchert, scb@irfu.se).

Data are available on the ESA FTP server at the following path:

~/Advanced/Plasma_Data/16_Hz_Faceplate_plasma_density/Sat_x

where X=A,B,C is the spacecraft identifier.

Data cover from beginning October 2014 to end February 2016 for all spacecraft, in daily compressed (ZIP) CDF files. Please note the continuity of the coverage is not assured, as the settings of the faceplate bias are not always suitable for density measurements. The dataset is meant to be regularly updated with the most recent data.

The Faceplate dataset at 16 Hz has the following caveats:

  1. due to an initial misevaluation of the Faceplate area, an offset exists on the density. Users shall take into account that: Ne_true = 0.8 * Ne_data
  2. due to a bug in computation, coordinates in the dataset are not correct. Users interested in precise geolocation are suggested to use the coordinates stored in the "Experimental 2 Hz product" and interpolate to 16 Hz.

In the updated version of this dataset that will be available soon, the above mentioned issues will be fixed.


2.3.5    The 2 Hz TII Cross-Track ion flow dataset: Description and Quality Information

An experimental dataset of Cross-Track ion flow at 2 Hz has been made available by the University of Calgary (Canada).

This dataset includes MJD2000 timestamp, latitude, longitude, radius, quasi-dipole lat, magnetic local time, ion cross-track components, satellite velocity components, co-rotation signal, velocity flow angles for the two sensors, offset corrections for the two sensors (as explained in the Release Note), electric field components, magnetic field components in the same coordinate reference frame. Data are accompanied by quality flags (for the y and z component of ion velocity and for the electric field, separately).

Before using this data, please read carefully the Release Note where more details on data processing and data content are given. In case of doubt, please contact:

for further clarifications. Please always include ESA in CC in your communications: swarm@eo-sppa.org

Data are available on the ESA FTP server at the following path:

~/Advanced/Plasma_Data/2_Hz_Cross-Track_Dataset/Sat_x

where x=A,B,C is the Spacecraft identifier.

Data coverage ranges from beginning of January 2016 to end of January 2017 for S/C A and B, in daily compressed (ZIP) CDF files. Data coverage is not continuous. Starting in 2014 EFI TII operations have been limited to a  small number of orbits per day. The dataset is meant to be regularly updated with the most recent data.

 

 

2.4          Accelerometer Products

The Level 1B accelerometer data (ACCx_PR_1B) are not distributed to the regular users as they are heavily perturbed by a variety of disturbances. The main problems are:

a) High number of steps, spikes, pulses observed in linear and angular acceleration;

b) Strong dependency on temperature variations.

Intensive investigations resulted in the conclusion that a fully automatic correction of the disturbances is not reliable, at least in the foreseeable future. Therefore, the accelerometer data is corrected by a team of experts and delivered together with all corrections as Level 2 Cat-1 data in product ACCxCAL_2_ (Corrected and calibrated accelerometer observations).


2.5          Level 2 - Cat2 Products

2.5.1    Important Notes

FAC - Field Aligned Currents, and IBI - Ionospheric Bubble Index

Following the correction of the Magnetic Level 1B data (MAGx_LR_1B), a new Baseline (02) of FAC and IBI products has been introduced since 16/07/2015. For the product types: FACxTMS_2F and IBIxTMS_2F, data have been therefore regenerated since, approximately, the beginning of the mission. For the product type: FAC_TMS_2F (FAC from combined S/C A and C measurements) the coverage does not start from beginning of mission, but from 15/04/2014.

EEF – Equatorial Electric Field

Equatorial Electric field data have been made available for the first time since 27/08/2015, only for S/C Alpha and Bravo. Unfortunately, the unavailability of the ASM instrument on S/C Charlie makes impossible the production of EEF data for such spacecraft.

Changes in L2 Cat-2 data format

From 19 January 2017 the L2 Cat-2 products (EEF, IBI, TEC and FAC) are delivered in CDF format as usual, but the extension changes from .DBL to ".cdf". These changes are applicable from 19 Jan 2017 onwards. There are no changes to the previously disseminated files.

 

2.5.2    Quality Information

Detailed quality information for the following products is available: FAC - single S/C solution, FAC - dual S/C solution, IBI, TEC, and EEF.

 

3.    Previous Datasets

In the 'Previous' folder on the ESA FTP server users can find three previous Product Baselines. This folder does not assure interoperability. Therefore in order to correctly select past interoperable Product Baselines, the Baseline History is presented below for the Level 1b and Level 2 - Cat 2 data.

 

3.1          Product Baseline: 03 (Validity: 25/11/2013 - 17/07/2015)

Table 3: L1B Product Baseline 03 contained in Previous Folder

Level 1b Product Baseline Sat A Product Baseline Sat B Product Baseline Sat C
Magnetic products (MAGNET) 03 03 03

 

3.1.1    Magnetic Products

3.1.1.1 Important Notes

 

A schema of the interoperability rules for Baseline "03" Magnetic products is sketched in Figure 2. Details are described below.

  • Previous Dataset 03 contains Magnetic products NOT CORRECTED from external disturbances effects (see Sect. 2.2.2).
  • MAGx_LR_1B product types from 26/11/2013 to 04/07/2014
    Both file counters 01 and 02 exist, e.g. for a given day one could find SW_RPRO_MAGA_LR_1B_20140404T000000_20140404T235959_0301_MDR_MAG_LR.cdf and SW_OPER_MAGA_LR_1B_20140404T000000_20140404T235959_0302_MDR_MAG_LR.cdf

    These two file types differ in the BNEC calculation: an updated set of Euler angles (rotation from Common Reference Frame of the Star Tracker to the VFM sensor frame) has in fact been used in the 0302 subset. If one wants to carry on a study or an investigation involving different magnetic product types (e.g. MAGx_LR_1B and MAGx_HR_1B), the two File Counters should not be mixed, and only the 0301 ones should be used together. Starting from 05/07/2014 all the magnetic data has been produced with the new set of Euler angles (0302).
  • The interoperable set of GPS and Orbit products (GPSxNAV_1B, GPSx_RO_1B, GPSx_RN_1B, MODx_SC_1B) for the Magnetic products before 19/10/2014, both 0301 and 0302 datasets, is only the 0301 one. As already stated in Sect. 2.1, modifications of the orbit processing were introduced on 19/10/2014: all the Orbit products were regenerated since beginning of mission with Product Baseline/File Counter 0302, but the magnetic products of Baseline "03" were not regenerated. This is the reason why both 0301 and 0302 GPS Orbit datasets are included in the "Current" folder of the ESA FTP server.
  • Data with File Counter 03, 02-17 January 2015
    Due to a problem in the L1b data processor leading to wrong rotation angles and, therefore, accuracy degradation on magnetic data, Swarm data with Product Baseline 03 and File Counter 02 produced from 2 to 17 January were not correct.

    They have therefore been regenerated and are now available with Product Baseline 03, File Counter 03. Users who have already downloaded 0302 data from the time interval above are advised to discard them and download the 0303 version instead. The old 0302 version has also been removed from the ESA FTP server.
  • Data with File Counter 03, 21/02/2015 – 29/03/2015
    All the Swarm data with Product Baseline 03 have been regenerated between 21/02/2015 and 29/03/2015. The reason for the regeneration was a bug in the HDR files which led to an incomplete coding of the [DSD] (Data Set Descriptor) blocks: the HDR files with version 0302 reported always one single [DSD] block even for those products, like MAGx_LR_1B, MAGx_CA_1B and MAGx_HR_1B, which include more than one dataset. The HDR files with version 0303 have been corrected.

    As a positive counter effect of such regeneration, a discontinuity in the VFM Temporal Calibration File (TCF) biases and non-orthogonality factors occurred on day 10/03/2015 on S/C A magnetic data has been removed. On the ESA FTP server, only 0303 versions can be found and 0302 versions have been removed. Versions 0303 and 0302 for such period are interoperable but 0303 data are of better quality, therefore, users who have already downloaded 0302 data from the time interval above are advised to discard them and download the 0303 version instead.

Figure 2: Interoperability of Baseline 03 magnetic data with respect to Baseline 03 Orbit products. Only Product Baselines/File Counters with the same colour can be used together.

 

3.1.1.2 Quality Information

 

The magnetic products of Previous Dataset 03 are affected by a residual field which is under investigation: even after the ASM-VFM inter-calibration and the subtraction of the stray fields, a difference in the absolute F field measured by the scalar magnetometer (ASM) and the |B| module obtained from the vector components (VFM) still exists (about 2 nT on average). Scientific applications in which the required accuracy of the magnetic field is of such comparable values need to take this information into account.

With respect to previous Baselines 01 and 02, the main improvements introduced in Baseline 03 are the following:

  • The q_STR_VFM values reported in the header file (.HDR) of the SW_OPER_MAGx_LR_1B File Counter 0302 were not correct. Please see here for further details.
  • The computation of Flags_F, Flags_B and Flags_Platform in the magnetic products MAGx_LR_1B and MAGx_HR_1B is now reliable and fully in line with the products definitions
  • The computation of "F_error" (MAGx_LR_1B) and "B_error" (MAGx_LR_1B and MAGx_HR_1B) is now reliable
  • The transformation q_NEC_CRF has been improved: in the previous baselines data, q_NEC_CRF was calculated at 1 Hz then interpolated to 50 Hz for obtaining the high resolution BNEC. This introduced discontinuities in the quaternion transformation and therefore occasional jumps in BNEC of the order of 10-15 nT. Now q_NEC_CRF is directly calculated at 50 Hz for the high resolution data and the issue has been fixed.
  • Each time the VFM instrument is switched on, a duplicated timestamp can occur. This kind of event, in two cases prevented the correct computation of the 50 Hz data records and the data processor failed. Now this contingency is correctly handled and the lost production has been recovered. Magnetic products for the following days and S/C are now available: S/C A 07/01/2014, S/C B 27/11/2013.
  • After the ASM onboard Swarm Charlie stopped its operations on 5 November 2014, magnetic data have not been produced for that spacecraft. Now the magnetic products for Swarm Charlie are again available with the contribution of the VFM inputs alone. These are the main consequences on data content and quality:
    • MAGC_HR_1B, ASMCAUX_1B and VFMCAUX_1B products are produced as before, with the only difference being that the ASM-VFM inter-calibration is no longer performed and the TCF calibration parameters used (biases, scale factors, etc...) are fixed and equal to the last ones computed before the ASM C failure (i.e. 5 November); Flags_B is always equal to 1.
    • MAGC_LR_1B are produced as before but the scalar field, F, is always zero, and Flags_F = 255
    • MAGC_CA_1B are not produced


3.2 Product Baseline: 02 (Validity: 15/04/2014 - 12/05/2014)

Previous Dataset 02 refers to data produced up to the end of the mission Commissioning Phase, using a data processor and a set of calibration parameters not yet consolidated. ESA therefore strongly recommends to NOT use such data for science studies, and to contact swarm@eo.sppa.org for further information.

Table 4: L1B Product Baselines 02 present in Previous Folder.

Level 1b

Product Baseline Sat A

Product Baseline Sat B

Product Baseline Sat C

Orbit and Attitude products (ORBATT)

02

02

02

Magnetic products (MAGNET)

02

02

02

 

3.3 Product Baseline: 01 (Validity: 25/11/2013 - 14/04/2014)

Previous Dataset 01 refers to data produced up to the end of the mission Commissioning Phase, using a data processor and a set of calibration parameters not yet consolidated. ESA therefore strongly recommends to NOT use such data for science studies, and to contact swarm@eo.sppa.org for further information.

Table 5: L1B Product Baselines 01 presented in Previous Folder.

Level 1b

Product Baseline Sat A

Product Baseline Sat B

Product Baseline Sat C

Orbit and Attitude products (ORBATT)

01

01

01

Magnetic products (MAGNET)

01

01

01

 

3.3.1    Level 2 - Cat 2 Products

Level 2 – Cat 2 products with Baseline 01 (FAC and IBI) are obtained using NON corrected magnetic products as input. They are therefore interoperable with Magnetic products of baseline "03" only.

Table 6: L2 Cat-2 Product Baselines 01 present in Previous Folder.

 

Sat Alpha

Sat Bravo

Sat Charlie

Level 2 - Cat2

Product Baseline

Validity

Product Baseline

Validity

Product Baseline

Validity

Field Aligned Currents – Single

01

26/11/2013 to 31/12/2014

and

09/05/2015 to 15/07/2015

01

28/11/2013 to 31/12/2014

and

09/05/2015 to 15/07/2015

01

04/12/2013 to 05/11/2014

and

09/05/2015 to 15/07/2015

Field Aligned Currents - SW A and C combined

01

04/12/2013 to 05/11/2014

01

04/12/2013 to 05/11/2014

01

04/12/2013 to 05/11/2014

Ionospheric Bubble Index

01

09/05/2015 to 15/07/2015

02

09/05/2015 to 15/07/2015

01

09/05/2015 to 15/07/2015

 

3.3.1.1 Important Notes

 

FAC: The FAC products, both single-spacecraft and combined A-C spacecraft solutions, are available with Product Baseline 01 from 25/11/2013 up to 31/12/2014. Then a technical problem within the data processing caused the stop of the operations for about four months. From 09 May 2015 the production of the Field Aligned Currents (FAC) derived from single spacecraft measurements has been restored.