====================================================================== === QPACK === ====================================================================== ===== Status, bug reports, etc. ====================================== Qpack is being implemented by Patrick Eriksson. The package is under development and quick changes from the outline below can be made. The working order will be to get a fairly complete system fixed as quickly as possible with only the most common options working. Other, less common options, will be implemented later when there is a need for those options. In a first stage, species, temperature (withot hydrostatic eq.) and pointing off-set will be the only variables that can be retrieved, and thermal noise (both measurement and calibration) will be the only observation uncertainties. The package is shared following the philosophy of the GNU public license, this including no warranty what so ever. Comments, suggestions and bug reports shall be sent to: patrick@rss.chalmers.se ===== General information ============================================ Qpack is a package of Matlab functions for retrieval work using ARTS as computational engine. Beside functions for the actual inversion and setting up the retrieval quantities, the package includes functions to characterize the retrieval and to optimize the calculation grids. The input to ARTS is provided in two ways, by control file templates and a structure with settings, denoted as Q. The final control files are created by the Qtool in AMI. All input not given by the template files must be specified by Q. The fields of Q are defined at the end of this file. For each retrieval/error quantity there exist a do-field, for example Q.TEMPERATURE_DO. If the do-field is set to zero, the quantity will be neglected totally and the other fields of the quantity can be left undefined. For retrieval/error quantities, the do-field can have a value between 0 and 3: 0 neglect the quantity totally 1 neglect during retrieval but calculate error for this quantity 2 treat as observation error 3 retrieve this quantity However, some quantities, such as calibration error, cannot be retrieved and the value 3 is not allowed. Measurement thermal is always turned on. Fields of Q ending with "_ON" shall be treated as booleans. For example, Q.CHBIN_ON=0 means that no binning of channels shall be performed. If a on-field is set to zero, the other fields of the quantity can be left undefined. The Q structure shall be defined in a function taking Q as only input and also returning Q, called a QDF. These functions shall not be called directly, the Q structure is instead obtained by the function qpack which calls the specified QDF. The function qpack starts with setting all do and on variables to zero. In this way it is not necessary to change all old files with definitions of Q when a new retrieval/error variable is introduced. The qpack function makes also some initial manipulations not handled elsewhere, such as setting the global varaible REPORT_LEVEL used by the out.m in AMI to the value of Q.QP_LEVEL. See further the help of qpack. Variables with a low influence of the atmospheric state are pre-calculated. Quantities such as H, Sx and Se are pre-calculated, while Kx is not. Most input files must be in the ARTS ascii format. This is true for all grids read directly by ARTS. Some files read by Qpack functions can either by ascii or binary (for example, sensor definitions), but to be safe, use ascii as this format always works. The file format for the different quantities is commented below the first corresponding variable. Please note that the 10-log (decades) is used as altitude coordinate when defining covariance matrices. Some quantities are expected to be calculated before doing an inversion or characterization of the retrieval performance. The following main quantities shall be pre-calculated: H, Sx and Se. Together with these main variables, some derived variables are created for higher flexibility. [!! Describe this !!] The standard order for the pre-calculations is Sx, H and Se, but it is only necessary to calculate Sx before H when eigenvector data reduction is applied. On the other hand, H must always be calculated before Se (to include a possible data reduction in Se by Hd). The pre-calculated quantities are stored using the Matlab format (.mat files). ===== Package organization and structure ============================ Qpack is structured into several sub-folders: === ./ === The only function in the main folder is a function to init the Qpack. === Templates/ === The control file templates are placed here. The extension of the templates is ".tmplt". === Main/ === This folder contains the main functions of Qpack. The main functions (beside qpack.m) start with "qp_", while so called in-line functions evoked by the cfile templates starts with "qpi_". === OEM/ === The functions for doing inversions by OEM and characterizing these retrievals are found in this directory. [!! Fill in later !!] These functions starts with "qpoem_", beside the function doing the actual inversions that is called just qpoem.m. === Setup/ === This folder contains various help functions to create and "optimize" the input files. (I am not sure how much stuff that will be implemented, but this is at least the general idea.) The functions in this directory starts with "qpmake_" and "qpopt_". === Samples/ === This folder contains a set of sample files to get started with Qpack. The example case corresponds to the 501.4 GHz band of Odin-SMR, but please note that the files are only provided as examples and do not make a perfect basis for simulating the Odin observations. These files form a full scale, but not extremely large, test case. In this directory you find a definition of the Q structure matching the files in the sub-folders. Copy this file (sample_q.m) to your test directory. Edit the path of the directories to match your environment. For example, if you want the pre-calculated quantities to be stored in the folder /home/xxx/Test, you set Q.OUT to this folder. After this, you should be ready to test Qpack. ===== To include new retrieval and error variables =================== To include a new retrieval variables the follwong steps are needed: 1. Introduce the needed Q-fields. 2. Modify qp_Sx. 3. Modify qp_kinfo. 4. Modify qpi_Kx. 5. Modify qpoem: a. Read a priori state. b. Include variable in map_x. c. Move retrieved values to X. 6. Check if qpoem_invchar must be modified. ===== The fields of Q ================================================ === Report levels === QP_LEVEL The report level for the Qpack functions (beside messages directly from ARTS). The number 0 means no reports, while the numbers 1-3 mean increasing level of verbosity. The function out.m in AMI is used for printing inside the Qpack functions. ARTS_LEVEL ARTS is called with this report level for the screen. === Names and directories === ARTS The full path to the ARTS executable to run, e.g. /u/patrick/arts/src/arts OUT Name or full path for storing pre-calculated quantities. For example: OUT = 'out' OUT = '/u/patrick/SMR/out' TMP_AREA All the calculations are performed in temporary sub-folders to TMP_AREA. The name of the temporary folders is TmpXXXXXX where XXXXXX is a random number. The sub-folder is deleted when the calculations are finished. SPECTRO_DIR Full path of the directory for line files and definition of continua. CALCGRIDS_DIR Full path of the directory where the calculation grids are stored (f_mono, za_pencil and p_abs). SENSOR_DIR Full path of the directory for sensor files. RETRIEVDEF_DIR Full path of the directory for grids and definition of covariance matrices. === Atmospheres === APRIORI_VMR_DIR Full path for the directory holding the a priori VMR profiles. There shall exist files for all needed species. This data are only used during the actual inversion. APRIORI_PTZ Full path for a priori pressure, temperature and altitude file. The full file name, including extension, shall be given, this in contrast to APRIORI_VMR_DIR where only the directory shall be specified. SETUP_ATMS One or several atmospheric profiles directories to be used during the pre-calculations. The directories shall be given as a character matrix where each row corresponds to a directory. When setting up grids and similar operations, all given atmospheres are considered to avoid "over-fitting" to a special atmospheric condition. When only one atmospheric state can be considered, the first atmosphere is selected. How to handle ptz? ==== Tag groups === RETRIEVAL_TAGS The tag groups to be retrieved. The tag groups shall be given as a Matlab string containing an ARTS string array (also called a string-string). For example: RETRIEVAL_TAGS = '"O3","ClO"' OTHER_TAGS The tag groups beside the retrieval tags. Format as above. === Spectroscopy === LINEFILE Name on file defining which transitions to consider for the line- by-line calculations. The line file should be in the ARTS format. LINESHAPE LINESHAPE_FACTOR LINESHAPE_CUTOFF Line shape variables. Corresponds to the lineshape workspace variable (WSV) in ARTS. CONTINUA The name of the file defining continua. This file shall look like the part of an ARTS control file where the continua are described. The cont_descriptionInit command shall not be part of the file. === Hydrostatic equilibrium === HSE_IN_ON Boolean to ensure that the input atmosphere fulfills hydrostatic equilibrium. The field is only used when reading the atmosphere for the first time HSE_RETRIEVAL_ON Boolean to tell if hydrostatic equilibrium shall be assumed for the retrieval. If this field is set to 1, temperature weighting functions are calculated with hydrostatic eq. and z_abs is recalculated for each iteration. Note that this field can be important for tropospheric water retrievals even if the temperature is not retrieved. === RTE === PLATFORM_ALTITUDE Corresponds to the ARTS WSV z_plat. STEPLENGTH_RTE Corresponds to the ARTS WSV l_step. GROUND_ALTITUDE GROUND_EMISSION Corresponds to the ARTS WSVs z_ground and e_ground, respectively. The ground temperature is set to the the first element of t_abs. REFRACTION_ON Boolean to turn on refraction. REFR_METHOD The method to use for calculation of refractive index. It is assumed that the method does not need any specific input. For example: REFR_METHOD = 'Boudouris' REFR_LFAC Corresponds to the ARTS WSV with the same name. EMISSION_ON Boolean to include emission in the calculations. === Calculation grids === P_ABS F_MONO ZA_PENCIL The file (with extension) in GRID_DIR defining p_abs, f_mono and za_pencil, respectively. The files must be in ascii format. === Sensor === F_ORDER ZA_ORDER These variables define if the spectra shall be treated as piece- wise linear or cubic when setting up the sensor H-matrix. The value 1 means piece-wise linear and the value 3 piece-wise cubic. ANTENNA_ON Boolean to consider the antenna pattern. ZA_SENSOR The zenith angles observed by the sensor (the middle points of the antenna pattern). If the antenna not is considered, ZA_SENSOR is set to equal ZA_PENCIL. ANTENNA_FILE The file in SENSOR_DIR defining the antenna pattern. See hAntennaFromFileAdv for file format. ANTENNA_ORDER Determines if the antenna pattern shall be treated to be either piece-wise linear (1) or cubic (3). See further F_ORDER. ANTENNA_MOVE Movement of the antenna (unit is degrees) during the integration. A constant scanning velocity is assumed. The value 0 means a fixed antenna during the integration. DSB_ON Boolean to consider the image sideband. DSB_FILE The file in SENSOR_DIR defining the response of the sideband filter. See hMixerFromFile for file format. DSB_ORDER Determines if the sideband filter response shall be treated to be either piece-wise linear (1) or cubic (3). See further F_ORDER. DSB_FPRIMARY Some frequency of the primary band (to indicate if the upper or lower band is the primary band). DSB_LO The LO frequency. BACKEND_ON Boolean to consider the backend (spectrometer). F_SENSOR The frequencies observed by the sensor (the middle points of the backend channels). If the backend not is considered, F_SENSOR is set to equal F_MONO. BACKEND_FILE The file in SENSOR_DIR defining the backend channel response. The response of all channels are assumed to be identical. See hBackendFromFile for file format. BACKEND_ORDER Determines if the channel response shall be treated to be either piece-wise linear (1) or cubic (3). See further F_ORDER. === Data reduction === BINVIEW_ON Boolean to turn on binning of subsequent spectra. BINVIEW_DATA Corresponds to the input vector bins of hBinView. BINNING_ON Boolean to turn on binning of neighboring channels. BINNING_FILE Full path of file defining the binning pattern. See hBinFile for file format. KRED_ON Boolean to turn on data reduction by the Kx-matrix version of the Hotelling transformation. KRED_N Number of eigenvectors to keep in the data reduction. === Thermal noise === The thermal noise is always treated to be an observation uncertainty (corresponding to a do-variable = 2). CHCORR_ON Boolean to turn on correlation of the thermal noise in neighboring channels. CHCORR_DATA A vector giving the correlation between the noise in one channel and the neighboring ones. The vector starts with giving the value for the closest channel. An example: CHCORR_DATA = [0.6 0.3]. MEASNOISE_FILE The file in SENSOR_DIR specifying the standard deviation of the measurement noise. The file data shall have two columns, where the first column holds frequencies and the second column gives the standard deviations. Linear interpolation is applied to get values for other frequencies (but no extrapolation). This noise is treated to be totally uncorrelated between different zenith angles. CALINOISE_DO Treatment of calibration noise (0-2). This noise is partly correlated between subsequent viewing angles, in contrast to the measurement noise. CALINOISE_FILE As MEASNOISE_FILE but for the calibration noise. CALINOISE_CORR As CHCORR_DATA but giving the correlation between subsequent viewing angles. === Retrieval/error quantities === SPECIES_KGRIDS The retrieval grid for each RETRIEVAL_TAGS as a string-string. For example: SPECIES_KGRIDS = '"grid1.aa","grid2.aa"' The grids shall be defined in files found in RETRIEVDEF_DIR. SPECIES_COVMATS Definition file of Sx for each retrieval tag as a string-string. The file format of these files are specified in the on-line help for the ARTS method sFromFile. The altitude coordinate when specifying the covariance matrices is the 10-log of the pressure, or decades. A decade is about 16 km. SPECIES_POSITIVE_ON Flag to include a positive constraint for the species retrieval (by retrieving the log of the species profiles). This feature is only activated for non-linear retrievals. TEMPERATURE_DO Treatment of temperature (0-3, see the top of the file). TEMPERATURE_KGRID The file defining the retrieval/error grid for temperature. TEMPERATURE_COVMAT The file defining the temperature covariance matrix. POINTING_DO Treatment of pointing off-sets (0-3). POINTING_STDV Standard deviation for the pointing off-set. POINTING_DELTA Perturbation to apply for calculation of the weighting function. Unit is degrees. === The stuff down to next === is not implemented, but will be later. CONTABS1_DO Treatment of continuum absorption. CONTABS1_ORDER Polynomial order for the continuum fitting. CONTABS1_LIMITS A vector of length 2 specifying the frequency limits of the fit. If the vector is left empty (but exist), the limits are set to cover all frequencies of f_mono. CONTABS1_KGRID The file defining the retrieval/error grid for continuum absorption. CONTABS1_COVMAT The file defining the continuum absorption covariance matrix. CONTABS2_DO CONTABS2_ORDER CONTABS2_LIMITS CONTABS2_KGRID CONTABS2_COVMAT All as the corresponding variable for CONTABS1, for a fit of the continuum absorption in another frequency band (such as the image band.) BL_POL_BACKEND_DO Treatment of polynomial baseline ripple. The ripple is assumed to be additive to the spectra after the backend and F_SENSOR is used to set up the corresponding weighting functions. BL_POL_BACKEND_ORDER Polynomial order of this baseline fit. === === Retrieval method === OEM is so far the only available retrieval method. OEM_NONLIN_ON Boolean to turn on nonlinear retrievals. The Marquardt- Levenberg scheme is used for the iterations. See qp_oem for more details. All OEM variables below are only needed for non-linear inversions. OEM_GA_START_VALUE Start value for gamma in the Marquardt-Levenberg iteration. OEM_GA_FAC_WHEN_OK The factor with which gamma is decreased when an iteration is succesful. OEM_GA_FAC_WHEN_NOT_OK The factor with which gamma is increased when an iteration is not successful. OEM_GA_LOWER_THRESHOLD When gamma equals or is below this value, gamma is set to zero. OEM_STOP Stop criterion for the iteration. The quantity used to check if convergence is reached is the change of the state vector from one iteration to next, scaled with Sx and the length of x. See qp_oem for details. OEM_MAX_ITER An upper limit on the number of iterations to be performed.