===================================================================== === QPACK === ====================================================================== ===== 0. Table of Contents ============================================= 1. Status, bug reports, etc. 2. General information 3. Package organisation and structure 4. Getting started 5. The fields of Q 5.1 Report levels 5.2 Names and directories 5.3 Atmospheres 5.4 Tag groups 5.5 Spectroscopy 5.6 Hydrostatic equilibrium and RTE 5.7 Calculation grids 5.8 Sensor 5.9 Data Reduction 5.10 Thermal noise 5.11 Retrieval/error/random quantities 5.12 Conditional simulations 5.13 Retrievals 5.14 Measurement 6. Qpack strategy 6.1 Doing a measurement characterization 6.2 Doing an optimal inversion 6.3 Generating random sets ===== 1. Status, bug reports, etc. ====================================== Qpack is being implemented by Patrick Eriksson and Carlos Jimenez. The package is under development and quick changes from the outline below can be made. The package is shared following the philosophy of the GNU public license, this including no warranty what so ever. Comments, suggestions and bug reports shall be sent to: patrick@rss.chalmers.se jimenez@rss.chalmers.se The present official version is Qpack 1.0. Qpack is still under development but version 1.0 meets the basic needs for doing inversions basic error characterisation and to generate random set of simulated spectra. The only retrieval method handled so far is OEM. Due to our limited time, there is no user guide for Qpack beside this README file. But don't hesitate to contact us with questions if you have problems with Qpack. Good luck! ===== 2. General information ============================================ Qpack is a package of Matlab functions for retrieval work using ARTS as computational engine. Beside functions for the actual inversion and setting up the retrieval quantities, the package includes functions to characterise the retrieval and to optimise the calculation grids. The input to ARTS is provided in two ways, by control file templates and by a structure with settings, denoted as Q. The final control files are created by the Qtool in AMI (ARTS Matlab interface, found at arts/ami). All input not given by the template files must be specified by Q. The fields of Q are defined at the end of this file. For each retrieval/error quantity there exist a do-field, for example Q.TEMPERATURE_DO. If the do-field is set to zero, the quantity will be neglected totally and the other fields of the quantity can be left undefined. For retrieval/error quantities, the do-field can have a value between 0 and 3: 0 neglect the quantity totally, 1 neglect during retrieval but calculate error for this quantity, 2 treat as observation error, 3 retrieve this quantity However, some quantities, such as calibration error, cannot be retrieved and the value 3 is not allowed. See below in the section on "Qpack strategies". Fields of Q ending with "_ON" shall be treated as booleans. For example, Q.CHBIN_ON=0 means that no binning of channels shall be performed. If a on-field is set to zero, the other fields of the quantity can be left undefined. The Q structure shall be defined in a function taking Q as only input and also returning Q. These functions shall not be called directly, the Q structure is instead obtained by the function qpack which calls the specified function. The function qpack starts with setting all do and on variables to zero. In this way it is not necessary to change all old files with definitions of Q when a new retrieval/error variable is introduced. The qpack function makes also some initial manipulations not handled elsewhere, such as setting the global variable REPORT_LEVEL used by the out.m in AMI to the value of Q.QP_LEVEL. See further the help of qpack. Most input files must be in the ARTS ascii format. This is true for all grids read directly by ARTS. Some files read by Qpack functions can either by ascii or binary (for example, sensor definitions), but to be safe, use ascii as this format always works. The file format for the different quantities is commented below the first corresponding variable. Please note that the 10-log (decades) is used as altitude coordinate when defining covariance matrices. All covariance matrices are read by the AMI function sFromFile (see on-line help for a general definition of the file format). Some quantities are expected to be calculated before doing an inversion or characterisation of the retrieval performance. This typically reflects the degree of influence of the atmospheric state on the quantity. For instance, quantities such as H, Sx and Se are pre-calculated, while Kx is not. The following main quantities shall always be pre-calculated: H, Sx and the inverse of Se. Together with these main variables, some derived variables are created for higher flexibility. The order for the pre-calculations is H, Sx and Seinv. The pre-calculated quantities are stored using the Matlab format (.mat files). See "Qpack strategy" section for more information. See next sections for more information. They will provide more details about the package organisation and structure, some tips about how to get started, a description of the Q fields, and some advice about how to run Qpack. ===== 3. Package organisation and structure ============================ Qpack is structured into several sub-folders: === ./ === The only function in the main folder is a function to start up the Qpack. === Templates === The control file templates are placed here. The extension of the templates is ".tmplt". === Main === This folder contains the main functions of Qpack. The main functions (beside qpack.m) start with "qp_", while so called in-line functions evoked by the cfile templates starts with "qpi_". === CLS === The functions for doing inversions by constrained least squares, that is, OEM and Tikhonov regularisation. Ordinary least squares is also handled by these functions for simplicity. These functions starts with "qpcls_", beside the function doing the actual inversions that is called just qpcls.m. === Setup === This folder contains various help functions to create and "optimise" the input files. The functions in this directory starts with "qpmake_" and "qpopt_". === Samples === This folder contains a set of sample files to get started with Qpack. The example case corresponds to the 501.4 GHz band of Odin-SMR, but please note that the files are only provided as examples and do not make a perfect basis for simulating the Odin observations. These files form a full scale, but not extremely large, test case. ===== 4. Getting started ================================================ In the directory Sample you find a definition of the Q structure matching the files in the sub-folders. Copy this file (sample_q.m) to your test directory. Edit the path of the directories to match your environment. For example, if you want the pre-calculated quantities to be stored in the folder /home/xxx/Test, you set Q.OUT to this folder. After this, you should be ready to test Qpack. ===== 5. The fields of Q ================================================ The Qpack fields are described here. Check also Qpack/ChangeLog for an understanding of Qpack and possible changes not reflected here. The information here is a fairly update of the present Qpack version, but we might have missed something. To check in the Changelog make a search of the field from the top of the file to see in chronological order - from most recent - all the information related to that field. Notice also that in some cases there is an exact equivalence between a Q field and an ARTS workspace variable. In this case more information can be found in the ARTS user's guide (AUG). 5.1 Report levels QP_LEVEL ARTS_LEVEL 5.2 Names and directories ARTS ARTS_ERROR_IGNORE OUT TMP_AREA SPECTRO_DIR CALCGRIDS_DIR SENSOR_DIR RETRIEVDEF_DIR 5.3 Atmospheres APRIORI_VMR APRIORI_PTZ SELTAGS_VMR SETUP_VMR SETUP_PTZ 5.4 Tag groups RETRIEVAL_TAGS SELTAGS OTHER_TAGS SPECTRO_TAGS 5.5 Spectroscopy LINEFORMAT LINEFILE LINESHAPE LINESHAPE_FACTOR LINESHAPE_CUTOFF CONTINUA SPECTRO_DO SPECTRO_TAGS INTENS_ON POSITION_ON AGAM_ON SGAM_ON NAIR_ON NSELF_ON PSHIFT_ON 5.6 Hydrostatic equilibrium and RTE HSE_IN_ON HSE_RETRIEVAL_ON HSE_PREF HSE_ZREF PLATFORM_ALTITUDE STEPLENGTH_RTE GROUND_ALTITUDE GROUND_EMISSION REFRACTION_ON REFR_METHOD REFR_LFAC EMISSION_ON 5.7 Calculation grids P_ABS F_MONO ZA_PENCIL 5.8 Sensor ANTENNA_ON ANTENNA_ZA ANTENNA_FILE ANTENNA_ORDER ANTENNA_MOVE DSB_ON DSB_FILE DSB_FPRIMARY DSB_LO BACKEND_ON BACKEND_FREQS BACKEND_FILE FREQSWITCH_ON FREQSWITCH_THROW FREQSWITCH_REVERSE 5.9 Data Reduction BINVIEW_ON BINVIEW_DATA BINNING_ON BINNING_FILE KRED_ON KRED_N KRED_DEPTH LRED_ON LRED_HISTEP LRED_LOSTEP LRED_INDTAGS LRED_DEPTH LRED_KGRIDS Q.LRED_N 5.10 Thermal noise MEASNOISE_DO MEASNOISE_COVMAT MEASNOISE_SCFAC CALINOISE_DO CALINOISE_COVMAT 5.11 Retrieval/error/random quantities RETRIEVAL METHOD CLS_SPECIES_POS_ON CLS_NONLIN_ON CLS_RECALC_ABS_ON CLS_RECALC_WFS_NITER CLS_GA_START_VALUE CLS_GA_FAC_WHEN_OK CLS_GA_FAC_WHEN_NOT_OK CLS_GA_UPPER_LIMIT CLS_STOP CLS_MAX_ITER 5.12 Conditional simulations NUMBER_DO 5.13 Retrievals RETRIEVAL METHOD CLS_SPECIES_POS_ON CLS_NONLIN_ON CLS_RECALC_ABS_ON CLS_RECALC_WFS_NITER CLS_GA_START_VALUE CLS_GA_FAC_WHEN_OK CLS_GA_FAC_WHEN_NOT_OK CLS_GA_UPPER_LIMIT CLS_STOP CLS_MAX_ITER 5.14 Measurement VALID_CHANNELS; VALID_CHANNELS_RANGE; === 5.1 Report levels === QP_LEVEL The report level for the Qpack functions (beside messages directly from ARTS). The number 0 means no reports, while the numbers 1-3 mean increasing level of verbosity. The function out.m in AMI is used for printing inside the Qpack functions. ARTS_LEVEL ARTS is called with this report level for the screen. === 5.2 Names and directories === ARTS The full path to the ARTS executable to run, e.g. /u/patrick/arts/src/arts ARTS_ERROR_IGNORE If set to 0, an ARTS error will also result in a Matlab error. That is, the Matlab execution will stop. If set to 1, Matlab will only give a warning and will try to continue to the execution (but this will not always be possible). The latter can be an advantage when performing batch inversions. For some cases qpcls can manage to continue after an ARTS error. Default is 0. OUT Name or full path for storing pre-calculated quantities. For example: OUT = 'out' OUT = '/u/patrick/SMR/out' TMP_AREA All the calculations are performed in temporary sub-folders to TMP_AREA. The name of the temporary folders is TmpXXXXXX where XXXXXX is a random number. The sub-folder is deleted when the calculations are finished. SPECTRO_DIR Full path of the directory for line files and definition of continua. CALCGRIDS_DIR Full path of the directory where the calculation grids are stored (f_mono, za_pencil and p_abs). SENSOR_DIR Full path of the directory for sensor files. RETRIEVDEF_DIR Full path of the directory for grids and definition of covariance matrices. === 5.3 Atmospheres === APRIORI_VMR Full path for the a priori VMR profiles. There shall exist a file for all needed species. This data are only used during the actual inversion or when generating random data. APRIORI_PTZ Full path for a priori pressure, temperature and altitude file. The full file name, including extension, shall be given, this in contrast to APRIORI_VMR where only the main path shall be specified. SELTAGS_VMR Full path for the a priori VMR profiles for the tags specified in SELTAGS. This is to handle 'raw_vmrsReadFromFiles' arts method. SETUP_VMR SETUP_PTZ One or several atmospheres to be used during the pre-calculations. The format for the VMR profiles and the PTZ files follows the a priori atmosphere (APRIORI_VMR_DIR and APRIORI_PTZ). Each atmosphere corresponds to a string in an ARTS string array (see RETRIEVAL_TAGS). When setting up grids and similar operations, all given atmospheres are considered to avoid "over-fitting" to a special atmospheric condition. When only one atmospheric state can be considered, the first atmosphere is selected. See below for its use with random data. ==== 5.4 Tag groups === RETRIEVAL_TAGS The tag groups to be retrieved. The tag groups shall be given as a Matlab string containing an ARTS string array (also called a string-string). For example: RETRIEVAL_TAGS = '"O3","ClO"' SELTAGS A sub group of tags which should be read from files. This corresponds to arts seltags variable. SELTAGS = '"O2"' Default is SELTAGS empty. OTHER_TAGS The tag groups beside the retrieval tags. Format as above. === 5.5 Spectroscopy === Information about the spectroscopic calculations is found in the AUG chapter "Gas absorption". LINEFORMAT Format of the database. This is to handle the case when the spectral data is from other catalogs. Possible competition: 'Hitran', 'Jpl','Myran2', and 'Arts' e.g. Q.LINEFORMAT = 'Mytran2', reads from a Mytran format; If no specified, then a 'Arts' format is expected. LINEFILE Name of file defining which transitions to consider for the line- by-line calculations. The line file should be in the ARTS format. LINESHAPE LINESHAPE_FACTOR LINESHAPE_CUTOFF Line shape variables. Corresponds to the line-shape workspace variable (WSV) in ARTS. CONTINUA The name of the file defining continua. This file shall look like the part of an ARTS control file where the continua are described. The cont_descriptionInit command shall not be part of the file. === 5.6 Hydrostatic equilibrium and RTE === Information about the RTE calculations is found in the AUG chapters "Basic radiative transfer" and "Line of sight 1D". HSE_IN_ON Boolean to ensure that the atmosphere fulfils hydrostatic equilibrium. HSE_RETRIEVAL_ON Boolean to tell if hydrostatic equilibrium shall be assumed for temperature retrievals. If this field is set to 1, temperature weighting functions are calculated with hydrostatic eq. This field only affects the calculation of weighting functions. If z_abs shall be modified after an inversion to ensure that HSE is fulfilled, this is governed by HSE_IN_ON. Notice that is possible to calculate the temperature weighting functions without hydrostatic equilibrium but still forcing the retrieved atmosphere to fulfil HSE. HSE_PREF Pressure of the reference point for hydrostatic equilibrium. HSE_ZREF Altitude of the reference point for hydrostatic equilibrium. PLATFORM_ALTITUDE Corresponds to the ARTS WSV z_plat. STEPLENGTH_RTE Corresponds to the ARTS WSV l_step. GROUND_ALTITUDE GROUND_EMISSION Corresponds to the ARTS WSVs z_ground and e_ground, respectively. If GROUND_ALTITUDE is set to be empty, the ARTS WSM groundAtBottom is called. Otherwise, groundSet is used where the ground temperature is obtained by interpolating t_abs to the ground altitude. REFRACTION_ON Boolean to turn on refraction. REFR_METHOD The method to use for calculation of refractive index. It is assumed that the method does not need any specific input. Allowed so far 'Unity', 'Boudoris', and 'BoudourisDryAir', see AUG for more info. For example: REFR_METHOD = 'Boudouris' REFR_LFAC Corresponds to the ARTS WSV with the same name. EMISSION_ON Boolean to include emission in the calculations. === 5.7 Calculation grids === P_ABS F_MONO ZA_PENCIL The file (with extension) in GRID_DIR defining p_abs, f_mono and za_pencil, respectively. The files must be in ascii format. === 5.8 Sensor === Information about the sensor calculations is found in the AUG chapter "Sensor modelling". ANTENNA_ON Boolean to consider the antenna pattern. ANTENNA_ZA The zenith angles observed by the sensor (the middle points of the antenna pattern). This file is only read if an antenna is considered. If the antenna is neglected, the viewing angles given by ZA_PENCIL are used to define the observation direction ANTENNA_FILE The file in SENSOR_DIR defining the antenna pattern. See hAntennaFromFileAdv for file format. ANTENNA_ORDER Treatment of the antenna pattern, linear (=1) or cubic (=3). ANTENNA_MOVE Movement of the antenna (unit is degrees) during the integration. A constant scanning velocity is assumed. The value 0 means a fixed antenna during the integration. If the movement is the same for all zenith angles, ANTENNA_MOVE can be set to be a vector of length 1. Otherwise ANTENNA_MOVE shall have the same length as ANTENNA_ZA. DSB_ON Boolean to consider the image sideband. DSB_FILE The file in SENSOR_DIR defining the response of the sideband filter. See hMixerFromFile for file format. DSB_FPRIMARY Some frequency of the primary band (to indicate if the upper or lower band is the primary band). DSB_LO The LO frequency. BACKEND_ON Boolean to consider the backend (spectrometer). BACKEND_FREQS The frequencies recorded by the sensor (the middle points of the backend channels). This file is only read if a backend is considered. If no backend and mixer are included, F_MONO defines the frequencies of the spectral values. If no backend is included, but a mixer, the frequencies of F_MONO inside the primary band will be set as the observation frequencies. BACKEND_FILE The file in SENSOR_DIR defining the backend channel response. The response of all channels are assumed to be identical. See hBackendFromFile for file format. BACKEND_IF Boolean to determine if backend frequencies are given in RF or IF. Default is 0, which means that RFs are given. To use BACKEND_IF: DSB_LO must be set, and the upper band must be set to be primary band (DSB_FPRIMARY > DSB_LO). FREQSWITCH_ON Boolean to include a frequency switch procedure. If selected, BACKEND_IF must also be selected and DSB_LO be specified. FREQSWITCH_THROW The size of the frequency switch throw. This is the total size of the throw. The actual throw is +/-FREQSWITCH_THROW/2. This means that the two line "images" will be placed symmetrically around the nominal transition frequency. If frequency throw is included, the backend must have channels equally spaced in frequency. The throw must be a multiple of the channel spacing. FREQSWITCH_REVERSE The default is that the line appearing at the lower frequency is weighted negetivaly (thus looking as an absorptionb line). This can be changed be setting this field to 1. === 5.9 Data reduction === Information about the data reduction is found in the AUG chapter "Data reduction". BINVIEW_ON Boolean to turn on binning of subsequent spectra. BINVIEW_DATA Corresponds to the input vector bins of hBinView. BINNING_ON Boolean to turn on binning of neighbouring channels. BINNING_FILE Full path of file defining the binning pattern. See hBinFile for file format. KRED_ON Boolean to turn on data reduction by the Kx-matrix version of the Hotelling transformation. The basic idea is to do a Hotelling transformation of the spectral space. But as the spectral variability can be decomposed as: Sy = KxSxKx' + KbSbKb' + Se and we are interested in capture the variability of the parameters to be retrieved (x), the transformation is calculated only from that variability: KxSxKx' = E V E' where E are the eigenvectors and V the eigenvalues. The Hotteling transformation is derived as: 1. Notice that KxSxKx' = Kx sqrt(Sx) sqrt(Sx)' Kx' = Kx sqrt(Sx) (Kx sqrt(Sx))' where sqrt(Sx) is one of the possible square roots of Sx. 2. Then a SVD Kx sqrt(Sx) = U M V' where U and V are left and right singular vectors. 3. The eigenvectors are then U as from (1): E V E' = U M V' (U M V')' = U M M U' KRED_N Number of eigenvectors to keep in the data reduction. KRED_DEPTH How to do the SVD of Kx sqrt(Sx) during reduction, possibilities: Different assumptions are implemented for sqrt(Sx): 1.Q.KRED_DEPTH = 0 sqrt(Sx) = I i.e. Sx is assumed to be the identity matrix 2.Q.KRED_DEPTH = 1 sqrt(Sx) = sqrt(diag(Sx)) i.e. Sx is assumed to be diagonal 3.Q.KRED_DEPTH = 2 sqrt(Sx) = chol(Sx) i.e. full Sx is used and the sqrt is done by a Choleski decomposition. LRED_ON Boolean to turn on data reduction by the limb version of the Hotelling transformation. By limb it means a special reduction based on combining the reduction based on Kx with a prior optimisation of the spectral input from a limb observation regarding each specific point of a retrieval grid. This reduction is useful for a retrieval method where each element of the retrieved species state vector is obtained independently, so the spectral input in the limb observation can be optimised for each element. The basic idea is to decide which elements for the spectral input are left for each retrieval point, and then to proceed with a normal Hotelling transformation of the new spectral space. In practice is done by doing the Hot transf in a Kx padded with zeros to leave only the rows (part of the scan) and columns (the retrieval tag depending on LRED_INDTAGS) corresponding to each retrieval tag and point. LRED_HISTEP This parameter gives the upward distance from the retrieval point to select what upper part of the scan to include in the reduction. If a tangent altitude corresponding to a zenith angle is there, the corresponding spectrum is kept. LRED_LOSTEP As above but to determine the downward distance. LRED_INDTAGS Flag, 1 only the Kx corresponding to the retrieval tag is left, 0 the whole Kx is used. LRED_DEPTH As KRED_DEPTH LRED_KGRIDS This gives the retrieval grids where the reduction method (e.g neural nets) will be doing the retrieval. Notice that they can be different from SPECIES_KGRIDS in a Q structure use to generate rnd data (to allow the calculation of Kx with SPECIES_KGRIDS in a fine grid, independent of the retrieval grid). LRED_N As KRED_N === 5.10 Thermal noise === Some information about the thermal noise is found in the AUG chapter "Measurement errors". MEASNOISE_DO Treatment of measurement thermal noise (0-2). MEASNOISE_COVMAT The file specifying a covariance matrix to generate measurement noise. The file data shall have three columns, where the first column holds frequencies and the second column gives the standard deviations and the third one gives correlation length (in frequency as frequency is the abcisa here. Linear interpolation is applied to get values for other frequencies (but no extrapolation). See further the AMI function sFromFile. MEASNOISE_SCFAC Scaling factors for the standard deviation of the measurement thermal noise. The thermal noise covariance values read from MEASNOISE_COVMAT are multiplicated with factors squared (i.e. si_new = factor * si). The length of this vector shall be the same as for ANTENNA_ZA when the noise shall be scaled. For no scaling, just don't define the field or make the vector empty ([]). CALINOISE_DO Treatment of calibration noise (0-2). This noise is correlated between subsequent viewing angles, in contrast to the measurement noise. CALINOISE_COVMAT As MEASNOISE_FILE but for the calibration noise. === 5.11 Retrieval/error/random quantities === SPECIES_KGRIDS The retrieval grid for each RETRIEVAL_TAGS as a string-string. For example: SPECIES_KGRIDS = '"grid1.aa","grid2.aa"' The grids shall be defined in files found in RETRIEVDEF_DIR. SPECIES_COVMATS Definition file of Sx for each retrieval tag as a string-string. The file format of these files are specified in the on-line help for the AMI method sFromFile. The altitude coordinate when specifying the covariance matrices is the 10-log of the pressure, or decades. A decade is about 16 km. The std is given in relative units e.g 0.3 means 30% of species apriori concentration. TEMPERATURE_DO Treatment of temperature (0-3). TEMPERATURE_KGRID The file defining the retrieval/error grid for temperature. TEMPERATURE_COVMAT The file defining the temperature covariance matrix. Notice that the std is given in absolute units e.g. K and not in relative units as the species. TEMPERATURE_FAST Flag to select a faster version of the function to calculate weighting functions if hydrostatic equilibrium is turned on (HSE_RETRIEVAL_ON=1). Default is here 1. POINTING_DO Treatment of pointing off-sets (0-3). POINTING_STDV Standard deviation for the pointing off-set if 'gaussian' or half amplitude of pdf if 'uniform', realizations assumed to be centred around the nominal zenith angle. POINTING_DELTA Perturbation to apply for calculation of the weighting function. Unit is degrees. POINTING_PDF Treatment of pointing offset pdf, 'gaussian' and 'uniform' can be used to fill this field. This field is only considered when generating random data sets. For inversions and error characterisation, the PDF is implicitly set to gaussian by the use of covariance matrices. POINTING_COR Set to 1, pointing error is perfectly correlated, i.e. it's a pointing offset; set to 0, totally uncorrelated. So far no more refined schemes set (e.g linear drifts). Only considered for generation of random data sets. A perfect correlation is assumed elsewhere. FREQUENCY_DO Treatment of frequency off-sets (0-3). FREQUENCY_STDV Standard deviation for the frequency off-set [Hz]. FREQUENCY_DELTA Perturbation to apply for calculation of the weighting function. Unit is Hz. CONTABS_DO Treatment of continuum absorption (0-3). CONTABS_ORDER Polynomial order for the continuum fitting. CONTABS_LIMITS A vector of length 2 specifying the frequency limits of the fit. A -1 in the first position of this vector means that the lower limit shall be set to the first value of F_MONO, and -1 in the second position means that the upper limit shall be set to the last value of F_MONO. CONTABS_KGRID The file defining the retrieval/error grid for continuum absorption. CONTABS_COVMAT The file defining the continuum absorption covariance matrix. The standard deviations shall be given in percentages with respect to the absorption of the species in CONTABS_REF_SPECIES. The given percentages are converted to absorption values [1/m] by calculating the absorption of the reference species and multiplicate the percentages with the absorption at the fitting points. CONTABS_REF_SPECIES Reference species when defining the standard deviations in the covariance matrix for continuum absorption. See further above for CONTABS_COVMAT. EGROUND_DO Treatment of ground emission (0-3). EGROUND_LIMITS A common value for the ground emission is used inside each specified frequency range. Values outside the range of F_MONO can be given. The values are given as a vector and gives the starting point of each range, beside the last value that gives the end point of the last range. Range 1 ends where range 2 starts etc. The ranges for EGROUND_LIMITS=[f1 f2 f3 f4] are: f1 <= f < f2 f2 <= f < f3 f3 <= f <= f4 An example: if the ground emission is assumed to be different in primary and image bands, but to be constant inside each band, EGROUND_LIMITS can be set to [0 f_lo 1e99] where f_lo is the LO frequency. If parts of F_MONO are outside the specified ranges, no retrieval of the ground emission will be performed for those parts. If EGROUND_LIMITS is set to be empty, this means that each frequency shall be treated separately. EGROUND_COVMAT Definition file of Sx/Sb for the ground emission. Standard deviations and correlation lengths are given as functions of frequency. The mean of the first and last element of F_MONO inside each range given by EGROUND_LIMITS is used when interpolating the given values. EGROUND_MINMAX A vector with length 2 specifying the lower and upper limit for allowed values on the ground emission. If the vector is set to be empty, all values are allowed. These limits are applied only during the iterations of an inversion. When convergence has been reached, the retrieved values are returned even if they are outside the given min and max values. The min/max values can accordingly be used to keep the ground emission inside the range 0 and 1 during the iterations to avoid unphysical conditions. POLYFIT_DO Treatment of polynomial fit to baseline ripple (0-3). POLYFIT_ORDER Order of polynomial fit. POLYFIT_COVMATS Definition file of Sx/Sb for each polynomial coefficient, given as a string-string. The number of files shall be POLYFIT_ORDER+1. The standard deviations are given in intensity unit. The correlation length refers to the zenith angles and the unit is accordingly degrees. PPOLYFIT_DO PPOLYFIT_ORDER PPOLYFIT_COVMATS PPOLYFIT_LIMITS Variables for piecewise polynomial baseline fit. The variables PPOLYFIT_DO, PPOLYFIT_ORDER and PPOLYFIT_COVMATS act as the corresponding POLYFIT variables. All the polynomials have the same order and a priori uncertainty. The variable PPOLYFIT_LIMITS is a vector with the limits between the different polynomials. The end limits must be specified and the length of PPOLYFIT_LIMITS is accordingly the number of polynomials + 1. The limits are defined as for EGROUND_LIMITS. TB_REFLOADS_DO Treatment of loads used for load switching calibration (0-3). It is assumed that a calibration procedure is performed by observing a cold and a hot load, and that the cold one is also used as reference load during the observations of the atmosphere. The cold load corresponds to load 1 and the hot load to load 2 in Equation 71 on page 250 in: Eriksson, P., Microwave radiometric observations of the middle atmosphere: Simulations and inversions, Ph.D. thesis, School of Environmental Sciences, Chalmers University of Technology, Sweden, 1999. Note that the load temperatures here correspond to V^1 and V^2 (I^1 and I^2 are the nominal values). The load are here denoted as the cold and hot load, but the actual temperatures can be reversed. As for PROPCAL, the baseline is not affected by calibration imperfection. TB_REFLOADS_NOMINAL TB_REFLOADS_STDV Nominal brightness temperature and a priori uncertainty for the loads. Both variables shall be a vector of length 2. The first value corresponds to the cold load, and the second to the hot load. PROPCAL_DO PROPCAL_TB_REF PROPCAL_STDV Treatment of proportional calibration error (0-3). These fields deal with a calibration error that is proportional to the spectra, with a variable zero point. The baseline is not affected by the calibration imperfection. That is, we have ym = (1+c)*(y-y0) + bl, where bl is the baseline ripple, y0 the reference point, given by PROPCAL_TB_REF, y is the true spectrum, c the calibration error and ym the obtained spectrum. The a priori uncertainty for c is given by PROPCAL_STDV in units of brightness temperature. Q.PSF_DO Treatment of pressure shift parameter (0-3). Q.PSF_FILE Name on file defining which transitions to consider for pressure-shift retrieval(error analysis). The allowed number of species to consider is 1, but the number of transions is not restricted to 1. The linefile should be in the ARTS format and be put in SPECTRO_DIR. Q.PSF_STDV Standard deviation for the pressure-shift parameter. The unit is Hz. Q.PSF_DELTA Perturbation to apply for calculation of the weighting function. The unit is Hz. Q.FREQ_CALIB_DO Treatment of frequency calibration uncertainty/error (0-3). It can include translation and/or stretch of the monochromatic frequency grid to be fitted polynomially. This is a generalization of FREQUENCY_DO. Q.FREQ_CALIB_ORDER Order of polynomial fit. Q.FREQ_CALIB_COVMAT The file defining the covariance matrix. The first column holds frequencies and the second column gives the standard deviations, and the third column gives the correlation length (This last may not be useful, but is introduced to use the same format as others.). The unit is Hz. See sFromFile in more details. The file must be put in RETRIEVEDEF directory. Q.FREQ_CALIB_DELTA Perturbation to apply for calculation of the weighting function. The unit is Hz. SPECTRO_DO Flag to calculate the spectroscopic parameters weighting function SPECTRO_TAGS Specify the tags for whose lines the weighting function should be calculated. This is very useful for the case when one wants to omit a tag from the calculation (e.g., HNO3 for which the number of lines is very large), or to calculate only the weighting function for particular lines. INTENS_ON -- to calculate it for the line intensities POSITION_ON -- to calculate it for the line position; AGAM_ON -- to calculate it for agam ; SGAM_ON -- to calculate it for sgam ; NAIR_ON -- to calculate it for nair ; NSELF_ON -- to calculate it for nself; PSHIFT_ON -- to calculate it for pressure shift; Q.SAVE Additional field to save the the weighting function for different spectro parameters. This is very useful in the case when one the number of lines for which the weighting parameters should be calculated is very large and one wants to calculate the weighting function for each parameter separated. All the other quantities are calculated only once and they are save under the name given by Q.OUT. The default value for Q:SAVE is set to Q.OUT. An example of the control file to calculate the weighting function and to camculate/plot the spectroscopic error is given in Samples/SampleWfss. === 5.12 Conditional simulations === NUMBER_DO If different from 0 a random set following the statistics given in the _DO variables will be calculated. Those _DO variables larger than 0 are included in the calculations, but notice than so far only species, temperature, pointing offset, polynomial of order 0 and measurement noise can be part of random realizations. NUMBER_DO gives the number of rnd realizations for each variable. The number of spectra created is also NUMBER_DO, i.e for spectrum i the i realizations for each variable defined a state i used to generate spectrum i. If SETUP fields are given (see below 6.3 Generating random sets) NUMBER_DO is a vector with as meany elements as SETUP atmospheres, each element gives the number of realizations for that atmosphere. === 5.13 Retrievals === Retrievals by optimal estimation (OEM), Tikhonov regularization (TR) and ordinary (weighted) least squares (WLS) are here handled with the same functions as far as possible. The name constrained least squares is used as a common name for the methods (WLS is not a CLS method, but is included here for simplicity). NOTE that so far only OEM is working. RETRIEVAL METHOD String with the acronym of the retrieval method to use. Allowed options are so far: 'oem' CLS_SPECIES_POS_ON Flag to include a positive constraint for the species retrieval. The constraint is obtained by retrieving the log of the species profiles. This feature is only activated for non-linear retrievals or when generating random realizations. In the last case, a lognormal pdf is used to generate the species realizations instead of the normal pdf. If set to a scalar, the given value is applied to all retrieval species. For example, [1] means a positive constrain is applied for all species. Otherwise, the field must be a vector with length equalling the number of retrieval tag groups (e.g. [0 0 1 0]). CLS_NONLIN_ON Boolean to turn on nonlinear retrievals. The Marquardt- Levenberg scheme is used for the iterations. See qp_oem for more details. All CLS variables below are only needed for non-linear inversions. CLS_RECALC_ABS_ON This flag determines if the absorption will be recalculated for each state during non-linear inversions, or if the absorption will only be scaled by the species amounts. The latter option is of course faster. Default is RECALC_ABS_ON=1. This field is neglected if temperature is retrieved. With temperature as an inversion variable, absorption must be recalculated for each iteration. This flag can also be used when, for example, the self-broadening of water can change notably between the a priori and true states. CLS_ABS_SPECIES_ON Absolute vmr will be retrieved (the old Qpack retrieved the ratio x/x_a instead) if set to one. Is this option set to one, CLS_WFS_UNIT will be set to 'vmr'. CLS_RECALC_WFS_NITER This field specifies the number of iterations between recalculation of the weighting functions. If the weighting functions are not recalculated for an iteration, the last calculated values are used. All values for the field <= 1 are treated as 1, which means that the weighting functions are recalculated for each iteration. This is also the default value. With CLS_RECALC_WFS_NITER>1, time is saved by not calculating the weighting functions for each iteration, but this is counteracted by the fact that more iterations can be needed, and there will probably be more failures (requiring an increase of gamma) during the iterations. If time is in fact saved, and that convergence is still obtained, must be determined by experiment. CLS_WFS_UNIT The unit in which the weigthing function are calculated. Allowed values are 'frac' and 'vmr'. CLS_GA_START_VALUE Start value for gamma in the Marquardt-Levenberg iteration. CLS_GA_FAC_WHEN_OK The factor with which gamma is decreased when an iteration is succesful. CLS_GA_FAC_WHEN_NOT_OK The factor with which gamma is increased when an iteration is not succesful. CLS_GA_UPPER_LIMIT When gamma equals or is above this value, the iterations are stopped. CLS_STOP Stop criterion for the iteration. The quantity used to check if convergence is reached is the change of the state vector from one iteration to next, scaled with Sx and the length of x. See qp_oem for details. CLS_MAX_ITER An upper limit on the number of iterations to be performed. === 5.14 Measurement === VALID_CHANNELS Turns on rejection of unwanted channels of a measurement by restricting a minimum and maximum allowed brightness temperature or opacity. Only Option: 'Tb'. Option 'opacity' not yet implemented. VALID_CHANNELS_RANGE Corresponding range (e.g. = [-10., 175.]). Channels with values outside this range are removed from the inversion, simply by setting spectra (modeled and measured) and weighting functions to 0. Options: 'Tb', brightness temperature range in [Kelvin]. ===== 6. Qpack strategy =============================================== 6.1 Doing a measurement characterisation 6.2 Doing an optimal inversion 6.3 Generating random sets === 6.1 Doing a measurement characterisation === The _DO fields are treated as: - 0 the variable is ignored, i.e. no error is calculated - 1 calculates error as Dy K S K' Dy' but it's not treated as observation error, i.e it was not included in the derivation of Se and Dy - 2 treats the variable as observation error, i.e it is included in Se and used in deriving Dy, and of course the error is calculated as 1. - 3 the variable will be retrieved, so it's part of the vector state x and the retrieval error will be calculated by adding the contribution from all the variables treated as 1 and 2. The statistics for the characterisation are described by the _COVMATS and _KGRID fields. Use qpcls_invchar where H, Sx and Seinv must be pre-calculated. The following sequence will do the job: > Q = qpack('your_q_fun'); > qp_H(Q); > qp_Sx(Q); > qp_Se(Q); > qpcls_invhar(Q); === 6.2 Doing an OEM retrieval === The retrieval variables are given by _DO = 3 and the variables treated as observation errors by _DO as 2. Variables _DO 1 or 0 are ignored here. The retrieval grids are given by _KGRID fields, and the statistics to construct the Sx matrix are taken from _COVMATS. The typical sequence is: > Q = qpack('your_q_fun'); > qp_H(Q); > qp_Sx(Q); > qp_Se(Q); > X = qpcls(Q,y); where y is the spectrum/scan, or a simulated measurement. To generate the spectrum/scan corresponding to a Q, do the following: > Qy = qpack('another_q_fun'); > qp_H(Qy); > y = qp_y(Qy); === 6.3 Generating random sets ==== NUMBER_DO gives the number of realizations. Variables with _DO fields > 0 are disturbed around the a priori state with the variability given by _COVMAT (so far only species, temperature, pointing offset, polynomial baseline of order 0, and measurement noise). _KGRID fields are only used if a reduction based on Kx is done, in which case the retrieval grid is needed. APRIORI_VMR_DIR APRIORI_PTZ SETUP_VMR_DIRS SETUP_PTZ If no SETUP fields are given the APRIORI fields give the atmosphere where to take the species profiles to be randomly modified. If SETUP fields are given the APRIORI fields will be ignored. The idea is to generate sets of profiles-spectra where the profiles correspond to a more than one mean state. This can get a bit tricky, for instance if as part of the sensor a reduction based on Kx is done, the APRIORI fields will be used as a priori atmosphere when generating the Kx, so it might be a good idea if the APRIORI atmosphere is a mean state from the SETUP atmospheres. The typical sequence is: > Q = qpack('your_q_fun'); > qp_rnd_atm(Q); > qp_rnd_sensor(Q); > qp_rnd_atmxsensor(Q); where qp_rnd_atm creates pencil beam spectra, qp_rnd_sensor creates the random realizations for the sensor part, qp_rnd_atmxsensor applies the sensor to the pencil beam spectra. Another possible sequence is: > Q = qpack('your_q_fun'); > qp_H(Q); > qp_rnd_atm_mem_red(Q); > qp_rnd_sensor_mem_red(Q); > qp_rnd_atmxsensor_mem_red(Q); where the "memory reduction" functions are used. This is recommended to generate random realizations when the sets to generate are very large and memory problems occur. The trick is to apply the H matrix as soon as possible and not at the last step as it is done in qp_rnd_atmxsensor after using qp_rnd_atm and qp_rnd_sensor. Notice that the H matrix has to be pre-calculated. For this to be effective of course the Hd part of H has to correspond to a large reduction, e.g. eigenvector reduction by LRED_ON or KRED_ON. The drawback of these new functions is that all intermediate spectra and sensor realizations are saved in the transformed space, so it is not easy to have a feeling about what it has been produced. Therefore, if memory problems is not an issue it is still advisable to use the previous set of functions.