2004-05-03 Mattias Ekström * qpack-1-0-65 * CLS/qpcls.m: Removed save of variables and printout of cfile. The save required that a fixed number of out argument was determined and this is not allways wanted. * Main/qp_assemble_K.m: Changed unit of periods to Hz, so that it properly describes the length of the period in the spectra. It should be more intuitive this way. 2004-04-29 Mattias Ekström * qpack-1-0-64 * Main/qpack.m: Added field Q.SINEFIT for sinusoidal baseline fit. It is accompanied by Q.SINEFIT_PERIODS and Q.SINEFIT_COVMATS where Q.SINEFIT_PERIODS are the periods of the sinusoidal curves given in units of 1/Hz. SINEFIT works similarly to POLYFIT. * Main/qp_assemble_K.m: Added section for extending the K matrix. Note that two waves, a sine and a cosine, is used to retrieve both the phase and amplitude of a single sine wave. * Main/qp_kinfo.m: Added sinusoidal baselines as retrieval identities in kname. * Main/qp_Sxb.m: Extending the covariance matrix for the sinus waves. Note that there are two terms per period, a sine and a cosine. 2004-03-05 Patrick Eriksson * qpack-1-0-63 * Templates/cls_iter2_kx.tmplt: Moved call of refrCalc to be outside IF statement. I think that I fixed this, but ... 2004-03-04 Patrick Eriksson * qpack-1-0-62 * Main/qp_H.m: Missed to rename a variable correctly. 2004-03-04 Patrick Eriksson * qpack-1-0-61 * Main/qp_H.m: Removed another bug for new frequency switch implementation. 2004-03-02 Patrick Eriksson * qpack-1-0-60 * Main/qp_H.m: Fixed bugs for the frequency switch, after some test runs with Erwan. 2004-03-02 Patrick Eriksson * qpack-1-0-59 * README: Added field BACKEND_IF. * Main/qpack.m (Q.FREQSWITCH_REVERSE): Added field BACKEND_IF. * Main/qp_H.m: Changed implementation of frequency switch. It was before implemented as a shift of the backend, which is not how it is done in reality. The scheme also brakes down for a SSB reciever. The frequency shift is now implemented correctly as a LO throw. Code exists, but yet no test calculations. 2004-02-27 Carmen Verdes * qpack-1-0-58 * CLS/qpcls.m: Removed an empty line (line 210). This was giving some errors because it was between '=' and function value * CLS/qpcls.m: a small correction of the output variables when qp_iter1 is called (now it outputs the weighting function for spectroscopic parameters as well). * CLS/qpcls_invchar.m: Remove the line when the all retrieval variables were saved into a matrix (Q.OUT_retr_Char.mat) 2004-02-25 Carmen Verdes * qpack-1-0-57 * Main/qpack and Templates/*: added Q.SELTAGS (specifies the seltags) and Q.SELTAGS.VMR (specifies the filenames) fields to handle the case of raw_vmrsReadFromFiles. * README: Added some more documentation * Main/qp_plotspectro: remove the hard-coded uncertainties for pressure shifts. * Samples/SampleWfss/: just some small modifications (design) 2004-02-20 Carlos Jimenez * qpack-1-0-56 * Main/qp_rnd_atm.m, qp_rnd_atm_sensor.m, qp_rnd_atmxsensor.m and equivalent mem_red: some debugging and including output file information in headings. Bug for reading "H2O-MPM89" fixed and some files that were not needed removed. * README: small update in rnd section, qp_H is not needed to run before the rnd functions in normal configuration, qp_rnd_atmxsenor does it. See mscript help for more info. 2004-02-13 Patrick Eriksson * qpack-1-0-55 * Templates/rnd.tmplt: Removed some crap code. There was some code that looked like pasted into the document my mistake. 2004-02-11 Patrick Eriksson * qpack-1-0-54 * README: Changed functionality for GROUND_ALTITUDE=[]. Now groundAtBottom is called for this case. The solution with having the field GROUNDATBOTTOM_ON had the consequence that ground altitude had to be set to an OK value anyhow. Now the ground variables are always set (good or bad?). Added Carmen's new spectroscopic fields to table in Sec 5. * Templates/iter1.tmplt: Removed usage of Q.GROUNDSET_ON. This field was not defined in README or set by any function. Moved the call of refrCalc to be outside IF statement, which it was in version 1-0-51. * Templates/rnd.tmplt: Updated this template to work with GROUND_ALTITUDE=[], which was not the case before! 2004-02-11 Patrick Eriksson * qpack-1-0-53 * qpack_startup.m: Removed a line for Carmen's personal use. * README: Defined fields to include frequency switch. * Main/qpack.m: Added default values for frequency switch fields. * Main/qp_H.m: Included call of hFreqSwitch. 2004-01-13 Carmen Verdes * qpack-1-0-52 * CLS/qpcls.m: added a line to save the output quantities * Main/qpack.m: added fields for spectroscopic parameters: SPECTRO_DO: Flag to calculate the spectroscopic parameters weighting function SPECTRO_TAGS : Specify the tags for whose lines the weighting function should be calculated. This is very useful for the case when one wants to omit a tag from the calculation (e.g., HNO3 for which the number of lines is very large), or to calculate only the weighting function for particular lines. INTENS_ON -- to calculate it for the line intensities . POSITION_ON -- to calculate it for the line position; AGAM_ON -- to calculate it for agam ; SGAM_ON -- to calculate it for sgam ; NAIR_ON -- to calculate it for nair ; NSELF_ON -- to calculate it for nself; PSHIFT_ON -- to calculate it for pressure shift; * Main/qpack.m: included the field Q.GROUNDATBOTTOM_ON in order to handle the case for ground altitude taken from the bottom (use of 'groundAtBottom') * Main/qpi_K.m: change it to calculate the Kb for spectroscopic parameters if Q.SPECTRO_DO is 1, and for the parameter which are specified * Main/qp_assemble_K.m: read into the spectroscopic weighting function Kb, K * Main/qp_invar: changed to plot the spectroscopic parameters errors as well (calling qp_plotspectro.m). This is done only if exist do_plots and the calculation of the spectroscopic parameters weighting function is performed, i.e Q.SPECTRO_DO field is set to 1. * Main/qp_plotspectro: added. This script is executed inside of Main/qp_invar if the field Q.SPECTRO_DO is set to 1; * CLS/qpcls_invchar.m: changed the script in order to output the quantities needed corresponding to spectroscopic weighting function (Kb, kb_names) * Samples: added subdirectory SamplesWfss containing all the function and inputs to provide a spectroscopic error calculation. 2004-01-06 Patrick Eriksson * qpack-1-0-51 * Templates/abs.tmplt: Created this template, to make it easier to just calculate absorption. Before many unnecessary Q-fields were needed to exist. * Main/qp_abs.m: Function format given in header was wrong. Changed to use new template file abs.tmplt. Added the option to extract abs_per_tg. 2003-12-04 Patrick Eriksson * qpack-1-0-50 * Main/qpack.m: Added default values (0) for parameters introduced by Sho in last commit. 2003-11-19 Sho Tsujimaru * qpack-1-0-49 * Main/qp_Sxb.m, qp_assemble_K.m, qp_kinfo.m, qp_x2X.m, qp_x2arts.m: Two new retrieval items are added. Namely, pressure shift and calibration error of frequency grid. The latter is a generalization of frequency-offset retrieval implemented by Patrick, and can treat translation and stretch by polynomial fitting. For the moment, the weighting function for the pressure-shift parameter is calculated by numerical difference. But this should be replaced by Carmen's for spectroscopic paramters. See README for further details. * CLS/qpcls.m: Added several out arguments for convenience. For nonlinear retrieval of pressure-shift parameter, original linefile which gets updated in iterations is copied to another file as backup and is recovered after retrieval. 2003-10-09 Patrick Eriksson * qpack-1-0-48 * Templates/basic.tmplt: f_mono is now also saved. * Main/qp_abs.m: Fixed an error in the header. Added f_mono and PTZ as output. 2003-09-05 Patrick Eriksson * qpack-1-0-47 * The ground altitude can now be set to be empty. The WSM groundOff is then used. See further README, where the old text has also been improved. * Main/qp_x2X.m: Continuum absorption ended up as temperature (there was "if temperature or continuum", followed by "if continuum"). This bug was found by Sho. 2003-07-09 Patrick Eriksson * qpack-1-0-46 * Main/qp_Sxb.m: Fixed a bug pointed out by Sho. qp_abs was called with Q instead of Qtmp. 2003-06-24 Carlos Jimenez * qpack-1-0-45 * Wrong version, but I did not tag, so commiting again and tagging now. 2003-06-24 Carlos Jimenez * qpack-1-45 * README: updating file, mainly grammar and layout. 2003-05-28 Patrick Eriksson * qpack-1-0-44 * Samples/article_figs.m: Added this QDF for creation of article figures. * Main/qp_invchar.m: Do-level 2 is only considered now. This means that Se con be loaded, and is not calculated inside the function. Removed writing of temporary file, included by Viju. * Main/qp_errorplot.m: Made list on error quantities complete. * Main/qp_kinfo.m (name): Changed to 'Reference load temperature(s)'. 2003-05-26 Patrick Eriksson * qpack-1-0-43 * README: A small rearrangenment. 2003-05-26 Patrick Eriksson * qpack-1-0-42 * README: Removed _ORDER fields that now can not be used (that is 1 is the only valid option). * Main/qpack.m: Default for fields mentioned above is now 1. The fields can accordingly be left out from QDFs. 2003-05-05 Carlos Jimenez * qpack-1-0-41 * Forgot version number 2003-05-05 Carlos Jimenez * qpack-1-0-40 * Testing commit, no changes made. 2003-05-01 Carlos Jimenez * qpack-1-0-39 * Main/qp_rnd_atm_mem_red, qp_rnd_atm.m: slight modification, yo has to be saved as yo' so it agress with atmxsensor mscripts. 2003-04-30 Carlos Jimenez * qpack-1-0-38 * Main/qp_rnd_atm_mem_red, qp_rnd_atm.m: sort of a bug when creating the reference spectra yo from a batch of spectra affecting when Q.SETUP_VMR was used. The reference spectra is calculated for Q.APRIORI_VMR, it is user responsability to assure that this corresponds to the mean of the SETUP_VMR profiles. I think the spectra calculated corresponded to the last SETUP_VMR. This has been corrected. * Main/qp_Sxc.m: new mscript as qp_Sx, but giving covaraince matrices for temperature and species instead of reading the sx files. Read function help for details. 2003-02-28 Carlos Jimenez * qpack-1-0-37 * Main/qp_rnd_atm_mem_red.m: yo reference was saved before as the f_mono/za_pencil spectra, good to leave it like this, as we have later yosensor also saved. in the new mem_red functions both yo and yosensor were the same and already transformed, so this is corrected. 2003-02-27 Carlos Jimenez * qpack-1-0-36 * Main/qp_rnd_atm_mem_red.m, qp_rnd_sensor_mem_red.m, qp_rnd_atmxsensor_mem_red.m: adding new set of fuctions to generate random realizations when the sets to generate are very large and memory problems occur. The trick is to apply the H matrix as soon as possible and not at the last step as it is done in qp_rnd_atmxsensor after using qp_rnd_atm and qp_rnd_sensor. Fo this to be effective of course the Hd part of H has to correspond to a large reduction, e.g. eigenvector reduction by Q.LRED_ON or Q.KRED_ON. The drawback of these new functions is that all intermediate spectra and sensor realizations are saved in the transformed space, so it is not easy to have a feeling about what it has been produced. Therefore, if memory problems is not an issue it is still recomemdable to use the previous set of functions. 2003-02-17 Carlos Jimenez * qpack-1-0-35 * Main/qp_Hd.m: keep the debugging effort, another bug! The selection of tangent altitudes for the limb reduction was based on f_mono and za_pencil instead of f_sensor and za_sensor. The bug was not detected as it worked surprisingly well by pure luck due to the sizes of f_mono and za_pencil compared with f_sensor and za_sensor for the cases checked. But a careful checked when aplying the reduction to real SMR spectra showed the inconsistency. 2003-02-17 Carlos Jimenez * qpack-1-0-34 * Main/qp_rnd_atmxsensor.m: bug, realizations of baselines order 0 were not added to rnd spectra. 2003-02-07 Carlos Jimenez * qpack-1-0-33 * Main/qp_Hd.m: bug found when doing a L_RED, the program crash in qp_assemble_K because Kpost and K had different sizes if Kpost corresponded to POLYFIT_DO. The problem was that Kx was calculated with Hs = 1, with f_mono ans so on. This beacuse I did Hs * Kx inside limbKx in qp_Hd so the previous calculation ok Kx was with Hs = 1. SO now Kx is calculated with the right Hs and nothing is done inside limbKx. * Main/qp_H.m: another bug, qp_Sx has to be calculated not only when KRED_ON but when LRED_ON as well. 2002-12-27 Carlos Jimenez * qpack-1-0-32 * Main/qp_rnd_sensor.m: adding random realizations of constant baseline offsets as given by Q.POLYFIT_DO. So far implemented only for Q.POLYFIT_ORDER = 0. * Main/qp_rnd_atmxsensor.m: adding random realizations of constant baseline offsets as given by Q.POLYFIT_DO. So far implemented only for Q.POLYFIT_ORDER = 0. * Main/qp_Hd.m: in limbKx changing slightly the selection of tangent altitudes to be left in Kx for the reduction, same algorithm but better selection in extremes cases (when the retrieval alttitude is outside the ztan range). 2002-12-03 Patrick Eriksson * qpack-1-0-31 * Note that there should be a zero after 1-. * qpack_version.m: The function returns now also a string if an output argument is selected. 2002-11-19 Carlos Jimenez * qpack-1-30 * Main/qp_rnd_atm.m: fixing subfunction param2tag with new tag saving scheme. Before files were saved as xxx.O3.aa e.g. The problem is having some isotopes or freq ranges there, so we go now for e.g xxx.O3-666.aa. Freq range is not conflicting as the rnd realizations are done obly for main species so they won't be duplicate e.g. O3-666. 2002-11-15 Carlos Jimenez * qpack-1-29 * Main/qp_rnd_atm.m: when using lognorm generating a small number of realizations, statistics are far from true. We ensure that the mean is the expected mean by removing the bias in the set, but then it's possible to get negative values even when using lognorm. It's important to remove the bias e.g when doing inversions, to warranty that the inversion bias is not coming from the set. To avoid problems of getting negative numbers into ARTS, negative values are searched and replaced by 0. 2002-11-13 Patrick Eriksson * qpack-1-28 * README: Improved slightly the description around HSE. * Main/qp_x2X.m: HSE is now runed as part of this file if HSE_IN_ON=1. Before there could be cases when the setting of HSE_IN_ON did not had full effect. * Templates/hse.tmplt: Added this template to simply run HSE. 2002-11-05 Carlos Jimenez * qpack-1-27 * Main/qp_rnd_atm.m: major changes in the generation of lognorm realizations, now correlations are included and the definition of Sx is in the noraml space. Trying to explain it. If 2 variables x and @ are related by x = log(@) and @ is log-normally distibuted, then x is normally distributed. That means the mean and std of both variables are related. In Qpack the covariance to be given when generating lognorm realizations is Sx, not S@!!! This is decided so only use in the calculations covariance matrices in a normal space. But how do you calculate Sx from a given S@? Because you will be interested in generating lognormal species realizations with a certain mean and std. Well, a new ami function sForLogNorm can be used to derive the correspoding Sx from your especification for S@, check the function for more info. Notice that the covariance mapping is a pseudo mapping as nothing is done about the correlation lengths, i.e. the correlation lengths defined in S@ are applied in Sx. This new approach has also implications when doing OEM inversions in the log(VMR/apriori) space. If VMR/apriori=@, then the inversion is done in the space x=log(@). If @ is lognorm with S@, then x is normal with Sx. Before the covariance matrices were not mapped, i.e Sx was taken a S@. Now a more real Sx will be used. However, for the S@ we usually employ the differences are small and not practical implications are expected, e.g for a std(@)= 0.5 the corresponding std(x) = 0.47. This is good sign as similar behaviour is expected for the correlations, so the error by not mapping the correlations is also small. 2002-10-30 Carlos Jimenez * qpack-1-26 * Main/qp_rnd_atm.m: new coding from lognormal realizations. It works fine but correlation lengths are not included so far, someone outthere with ideas? If you generate e.g a set of species random realization the mean of the set will be the species mean and the std the species std given in the covmat. Notice then that the covmat is not any more describing a normal distibution, but the std of a lognormal distribution, with the correlation lengths ignored. So the random realizations are NOT the log of a set of normal realizations fullfilling the mean and std as described in the covmat. 2002-10-21 Patrick Eriksson * qpack-1-0-25 * Main/qp_Hd.m: The calling of hBinfile had two bugs. The bugs were found and reported by the Bern group. 2002-10-08 Carlos Jimenez * qpack-1-0-24 * Main/qp_rnd_atm.m: - param2tag modified to deal with all possibe tag names. - lognormal pdf for species instead of normal if Q.CLS_SPECIES_POS_ON = 1. It seems to give reasonable results, but the recipe to be furhter checked in the literature. * Main/qpack.m: Q.POINTING_COR=1 added, it was missing. * README: random part slightly improved. * Main/qp_H.m: saving of [Q.OUT,'.f_sensor'] and [Q.OUT,'.za_sensor'] always done as the data might be needed later by qp_assemble_K, qp_Se and qp_Sxb. Running qp_Sx here if reduction done as Sx is needed. 2002-08-28 Patrick Eriksson * qpack-1-0-23 * Samples/sample_q.m: The file contained an incorrect tag name (put there for test purposes). 2002-08-28 Patrick Eriksson * qpack-1-0-22 * Introduced the field ARTS_ERROR_IGNORE to control what qpack shall do with ARTS errors. See further README. 2002-08-15 Patrick Eriksson * qpack-1-0-21 * qpack_version.m: Missed to update version number in last commit. 2002-08-15 Patrick Eriksson * qpack-1-0-20 * CLS/qpcls.m: The averaging kernel matrix is now part of the output argument S. * Main/qpack.m: I had missed to set Q.FREQUENCY_DO = 0 as default. 2002-08-08 Patrick Eriksson * qpack-1-0-19 * A slight change of the fields for HSE. This is now valid (from README): HSE_IN_ON Boolean to ensure that the atmosphere fulfills hydrostatic equilibrium. HSE_RETRIEVAL_ON Boolean to tell if hydrostatic equilibrium shall be assumed for temperature retrievals. If this field is set to 1, temperature weighting functions are calculated with hydrostatic eq. The field names can now be a bit misleading but I guess it is best not to introduce new names. This means that temperature can now be retrieved by using analytical expressions for the WFs (HSE_RETRIEVAL_ON=0), but the atmosphere can still be forced to follow HSE by setting HSE_IN_ON = 1. 2002-07-17 Patrick Eriksson * qpack-1-0-18 * Again two main changes: 1. Modifications to make Qpack platform independent. 2. Retrieval of frequency off-sets included. 1: All functions have been revised to not include hard-coded file seperators. The Matlab function fullfile is now used to append path and file names. The functions unix and dos are now used to call arts to enable better error handling. 2: Qpack modified to make use of the new WF option in arts. 2002-07-12 Patrick Eriksson * qpack-1-0-17 * Two main things fixed: 1. Bugs releated to ground emission fixed. 2. It is now possible to make retrievals where no species profile will be retrieved. * Templates/: Changed templates to handle case 2. * Main/qp_x2arts: Fixed bugs for point 1. * Main/qp_kinfo: Bugs here as well for point 1. * CLS/qpcls.m: Made changes to handle no species retrieval. 2002-07-09 Carlos Jimenez * qpack-1-0-16 * CLS/qpcls.m: adding S to output of arts_failure. 2002-07-03 Patrick Eriksson * qpack-1-0-15 * CLS/qpcls.m: Made kx_index a field of the S structure to faciliate the extraction of data for each retrieval identity. 2002-05-31 Patrick Eriksson * qpack-1-0-14 * qpack_version.m: Fixed the version number. PLEASE, remember to set the version number in qpack_version before making a commit. 2002-05-17 Carlos Jimenez * qpack-1-0-13 * Allowing more than 1000 tranistions in /Setup/qpopt_linefile by adding a reallocate function 2002-05-15 Patrick Eriksson * qpack-1-0-12 * Some changes have been done to allow basic error characterisation in qpcls. This needed (to keep things relatively simple) that the function qp_Seinv was removed, and that part is now covered by qp_Se. This means that for the precalculations the standard function chain is: qp_Sx(Q); qp_H(Q); qp_Se(Q); * Main/qp_Se.m: This file now saves Se and its inverse if there is no output argument. The input arguments beside Q are now optional. * Main/qp_Seinv.m (Seinv): This function is removed. * CLS/qpcls.m: Added a new output argument, a structure giving some basic retrieval characteristics. See the function header. 2002-05-08 Carlos Jimenez * qpack-1-0-11 * small correction in README * small correction in Setup/qpopt_linefile for not deleting hbar if not created (line 337) 2002-04-07 Patrick Eriksson * qpack-1-0-10 * The old fields SPECIES_POSITIVE_ON, RECALC_ABS_ON and RECALC_WFS_ON are renamed to reflect that they are specifications for the inversion method, instead of atmospheric or sensor values. The meaning of RECALC_WFS_ON is also slightly changed. The new names are CLS_SPECIES_POS_ON CLS_RECALC_ABS_ON CLS_RECALC_WFS_NITER CLS_SPECIES_POS_ON and CLS_RECALC_ABS_ON work as the corresponding old fields. CLS_RECALC_WFS_NITER is the number of iterations between recalculation of the WFs. This gives more flexibility. For example, CLS_RECALC_WFS_NITER=2 means that WFs are recalculated every second iteration. All values <=1 for CLS_RECALC_WFS_NITER will be treated as 1, meaning recalculation for each iteration. This is also default. For all these feilds, see further README. * CLS/qpcls.m: Absorption is now always recalculated if temperature is retrieved. If this is not done, the temperature WFs will not be correct. Changes to include CLS_RECALC_WFS_NITER. More output messages. If WFs are recalculated or not, is shown in a new column. * Main/qpack.m: The default values for the discussed fields are Q.CLS_SPECIES_POS_ON = 0; Q.CLS_RECALC_ABS_ON = 1; Q.CLS_RECALC_WFS_NITER = 1; 2002-04-07 Carlos Jimenez * qpack-1-0-9 * Found a big bug, as WFS were recalculated when doing iterative inversions but not used, i.e., the 1st iteration WFS were always re-used. Fixed and see below. * New Q field called RECALC_WFS_ON to have the possibility of recalculating WFS or not when doing iterative inversions. CLS/qpcls.m and related mscripts accordingly updated. Default is RECALC_WFS_ON=1. 2002-04-03 Carlos Jimenez * qpack-1-0-8 * CLS/qpcls.m small bug related to previous work. 2002-04-03 Patrick Eriksson * qpack-1-0-7 * CLS/qpcls.m: The function handles now (most?) ARTS crashes. The G structure is then set to reasonable values. The field G.failure is set to 'ARTS crash'. All other return variables are set to be empty. * Main/qp_iter1.m: Returns empty variables if there is a ARTS crash. 2002-03-23 Patrick Eriksson * qpack-1-0-6 * CLS/qpcls.m: Removed the change done for 1-0-4. I was too quick to change was Carmen brought up the subject, it looked strange to divide the cost for with the length of y. However, it is the correct thing to do if the cost for y is handled liekwise. If you look at the equations, you see that it is the sum of the two costs that shall be minimized (without any normalisation to the vector lengths). If you then scale one of the costs, you must handle the other cost in exactly the same way. The costs values are now scaled with the length of y (as before 1-0-4). This scling is nice as it makes it easier to judge if a retrieval has been succesful or not. The final cost for y (cost_y) shall not exceed 1 too much. However, how well you can trust this depends on how well the measurement uncertainty covariance matrix (Se) matches the true magnitude of the thermal noise. If you overestimate the magnitude of thermal noise, the final cost for y will be smaller than 1 for a good fit. 2002-03-21 Patrick Eriksson * qpack-1-0-5 * Added the field TEMPERATURE_FAST to select which version of kTemp that is used when calculating temperature WFs with HSE. Default is the fast version. * CLS/qpcls.m: Added more output arguments for the second call version: [Dy,Kx,kx_names,kx_index,kx_aux] = qpcls( Q ) 2002-03-21 Patrick Eriksson * qpack-1-0-4 * CLS/qpcls.m: The cost for the state vector was normalised with the wrong length (ny instead of nx). This bug was found by Carmen. Thanks for that. 2002-02-16 Carlos Jimenez * qpack-1-0-3 * New fields for retrieving calibration missing in qpack. 2002-02-13 Patrick Eriksson * qpack-1-0-2 * Included two different ways of retrieving calibration variables. The two ways are denoted as TB_REFLOADS and PROPCAL. See README for descriptions. The calibration options handled by the plotting functions. * Main/qp_x2y.m: Added this function to handle the new calibration retrieval. The purpose of this function is to handle all parameters that are not included in ARTS.The baseline fit is moved to this function from qp_x2arts. 2002-01-22 Patrick Eriksson * qpack-1-0-1 * Finished the implementation of continuum absorption that works with an updated version of ARTS. No heavy testing but it seems to work. It is important not to include too high altitudes as the condition number for a matrix to invert as part of OEM becomes too poor. * Made the function qpcls_errorplot that cpmplements qpcls_invchar by plotting the error for individual sources. * The lower left figure created by qpcls_invchar shows now the correlation "averaging kernels" instead of the correlation FWHM. 2002-01-18 Oliver Lemke * qpack-1-0-0 * Version released with arts-1.0. 2002-01-18 Oliver Lemke * qpack-1-0-0-rc1 * OK, this is release candidate 1. Just tagged this as a final version for testing. Hope next version will be the final release. 2002-01-18 Patrick Eriksson * qpack-1-0-0-pre1 * Main/qp_invchar.m: Included functionality for continuum absorption and baseline parameters. * Continuum absorption: Changed the definition for CONTABS_LIMITS so it is easier to adjust to F_MONO. See README. The internal unit for continuum absorption is now 1/km. This to get x-values closer to 1 and get a better rank when inverting matrices for OEM. The unit 1/km is just used internally, retrieves values are in 1/m. Adapted qp_Sxb and qpi_K to these changes. HOWEVER, the continuum absorption is not totally working and cannot be used. There will be an error if you try. Continuum absorption has to wait until 1.1. * Main/qp_contabs_freqs.m: Added this function to handle frequencies for continuum absorption. * Main/qp_abs.m: Added this function to calculate the absorption for a Q (corresponding to qp_y). 2002-01-16 Patrick Eriksson * qpack-1-0-0 (Tag removed) * README: Some small changes and some new text. * Main/qpack.m: Added some to the header text. * CLS/qpcls.m: Wrote some xtext for the function header. 2002-01-15 Patrick Eriksson * qpack-0-0-49 * Setup/qpopt_fmono_by_interp.m: Renamed to qpopt_fmono. See below. * Setup/qpopt_fmono.m: The function qpopt_fmono_by_interp is renamed to this name. This as the old name did a wrong impression and I don't think there will be another similar function, so this name should suffice. A small bug fixed. Heading text improved. * Setup/qpopt_linefile.m: Created and added this function that generates a linefile for a frequency band. The strategy for the function resembles the one for qpopt_fmono. See the function header for more details. 2002-01-12 Patrick Eriksson * qpack-0-0-48 * README: Added description of the fields ANTENNA_MOVE and MEASNOISE_SCFAC. These fields allow that the integration time differs between the zenith angles. * Main/qp_H.m: Included Q.ANTENNA_MOVE. * Main/qp_Se.m: Included Q.MEASNOISE_SCFAC. See README. 2002-01-04 Patrick Eriksson * qpack-0-0-47 * qpopt_fmono_by_interp: Small changes to the messages given. * Some fixes in the sample case. 2001-12-10 Patrick Eriksson * qpack-0-0-46 * qpcls.m: Small changes to the godness structure G. The field names are now all in lower case. 2001-12-10 Patrick Eriksson * qpack-0-0-45 * The Q-fields F_SENSOR and ZA_SENSOR are replaced with the fields BACKEND_FREQS and ANTENNA_ZA, respectively. These names are much more informative than the old ones, they tell you the real meaning, and there is no risk of confusion with the f_sensor and za_sensor variables created by qp_H. Sorry, this means that you have to change your control files. I have been thinking about this change a long time and I tried to avoid it. However, yeasterday I myself got confused and lost quite some time, and I decided to make this change. * CLS/qpcls.m: The cost is now normalised with the length of y to make it more easy to check if the final cost is OK. If you have described the observation uncertainties correctly, the final cost should be close to 1, otherwise you haven't found the minima of the cost function. I cannot say what values above 1 of the final cost that can be accepted, but it should probably not be much above 1. 2001-11-29 Patrick Eriksson * qpack-0-0-44 * Another try to remove the OEM folder. 2001-11-28 Patrick Eriksson * qpack-0-0-43 * CLS/qpcls.m: Some cleaning in the function to make the code nicer. I did some actual changes, without changing the basic algorithms, so there could be a warning for bugs. * Main/qpack_version.m: Added this function to enable and handle version nmumber for Qpack. NOTE: This file should be edited before doing a commit. * Main/qpack.m: The command 'qpack -v' gives the version number. This feature was added to be consistent with ARTS. 2001-11-28 Patrick Eriksson * qpack-0-0-42 * Fixed a bug for qpcls appearing for linear retrievals and when fitted spectra were calculated (it worked if the non-linear gamma fields were specified). * Updfated the Sample folder to follow the changes in ARTS for continua. 2001-11-22 Viju O.-J. * qpack-0-0-41 * Templates/iter1.tmplt: Removed the keyword "CloudSat" from AtmFromRaw. * Templates/basic.tmplt: Removed the keyword "CloudSat" from AtmFromRaw. 2001-11-22 Patrick Eriksson * qpack-0-0-40 * Main/qp_eground_lims.m: Put in warning message if a ground emission range is totally outside F_MONO. * Main/qp_x2arts.m: Fixed a bug for cases when part of F_MONO is outside the ground emission ranges. E_GROUND is now read for each iteration. 2001-11-21 Patrick Eriksson * qpack-0-0-40 * qpack_startup.m: The Setup folder added to the search path. * Main/qp_ympb.m: A function similar to qp_y, but this one returns the monochromatic pencil beam spectra (MPB), that is, H is not applied. The function allows also that F_MONO and ZA_PENCIL are changed from the definition in Q. * Setup/qpopt_fmono_by_interp.m: The first function in Setup. This function "optimizes" f_mono with respect to interpolation error. See the function header for further information. 2001-11-18 Patrick Eriksson * qpack-0-0-39 * CLS/qpcls.m: It is not now demanded that gamma=0 to stop the iterations. The lower treshold for gamma is removed. If the upper limit for gamma is reached, the iteration number is now set to 9999. 2001-11-16 Patrick Eriksson * qpack-0-0-38 * Main/qp_x2arts.m: As suspected, there was a bug for the ground emission, that now is fixed. A non-linear inversion test for AMSU-B worked very well. 2001-11-15 Patrick Eriksson * qpack-0-0-37 * Non-linear CLS inversions did not work. The code is running but something is wrong! What? Maybe the ground emission. 2001-11-15 Patrick Eriksson * qpack-0-0-36 * README: A new Q-field, EGROUND_MINMAX: A vector with length 2 specifying the lower and upper limit for allowed values on the ground emission. If the vector is set to be empty, all values are allowed. These limits are applied only during the iterations of an inversion. When convergence has been reached, the retrieved values are returned even if they are outside the given min and max values. The min/max values can accordingly be used to keep the ground emission inside the range 0 and 1 during the iterations to avoid unphysical conditions. * Main/qp_invchar.m: Plotting for ground emission now included. 2001-11-15 Patrick Eriksson * qpack-0-0-35 * Some extra information in README about definition ot covariance matrices. 2001-10-26 Carmen Verdes * qpack-0-0-34 * I removed my last changes and I returned back to the old version ( qpack-0-0-31 ) 2001-10-26 Patrick Eriksson * qpack-0-0-33 * I just discovered that the OEM directory still existed. This is a new attempt to remove OEM. 2001-10-23 Carmen Verdes * qpack-0-0-32 * fast and dirty changes needed for intercomparison: * basic.tmplt: I added a new flag Q.READ.ABSCOEFF for reading the absorption coefficients. If this is set in the structure then the absorption coefficients are read from the specified file, otherwise they are calculated. This is needed for ARTS intercomparison case 3. Also I changed the template file to output the y_mono , f_mono, za_pencil vectors. * qp_y.m: I changed it to output the y_sensor vector. 2001-10-22 Patrick Eriksson * qpack-0-0-31 * It is now not necessary to always specify F_SENSOR and ZA_SENSOR. These Q-fileds are only needed when you include a backend and an antenna pattern, respectively. The function qp_H now stores the frequency and zenith angle grids that corresponds to your instrument simulation. See further README. * Ground emission is now a retrieval/error variable. See README for instructions on how to specify the input. This part has not been tested in detail. I leave this to Viju. Viju, can you test this? If you find anything strange, please send me your control file (the file where Q is specified) and needed input to run the case. 2001-10-18 Carlos Jimenez * qpack-0-0-30 * Test 2001-10-18 Carlos Jimenez * qpack-0-0-29 * Bug in qp_rnd_atm when doing uniform poiting fixed. * Added CLS/qpcls_batch.m, as qpcls but to invert a set of spectra produced by qp_rnd_atmxsensor.m, see the help function. 2001-10-15 Carlos Jimenez * qpack-0-0-28 * Bug in hse part in qp_rnd_atm fixed * New Q field Q.POINTING_COR, set to 1 the pointing error for all z_tan in a scan are prefectly correlated, set to 0 the errors are uncorrelated. Should we go for a more general approach as with the other errors? * qp_Hd hRedLimb modified, if whole Kx and equal retrieval grid for all species reduction only done for first one as it is identical for all TAGS. 2001-10-15 Carlos Jimenez * qpack-0-0-27 * New Q-field: OEM_GA_UPPER_LIMIT to set an upper limit to gamma, some inversions can go funny -numerical problems?-. qcls.m udpated. * New function qcls_batch.m, as qcls but taking as input a structure used to generate a random set instead of y and doing the inversion of all the spectra in the random set. See the function help. * qp_Hd updated by including hRedKx.m and hRedLimb.m as subfunctions, also updated. * qp_H updated, removing do_reduction check and using defalult values from qpack. * qp_rnd_atm and qp_rnd_sensor updated to include the possibility of give a previous random structure Qr to copy rnd realizations for the new Q instead of generating them for Q (valid e.g if Q and Qr have different sensor but the rnd realizations are needed to be the same) * new Q field: LRED_INDTAGS, for the limb reduction, set to 1 uses for in the reduction only the Kx for the corresponding retrieval tag, set to 0 uses the whole Kx. 2001-10-15 Patrick Eriksson * qpack-0-0-26 * Big internal changes, but just small changes in the control files. Starting with Q-field changes: * New Q-field: RETRIEVAL_METHOD The retrieval method must now be specified. So far is 'oem' the only allowed option, but there will bemore later (see below for CLS). * New Q-field: RECALC_ABS_ON See README. Can be left out. * Big news! The do-levels 1 and 2 are now fully working (before it was only do-level 2 for noise that worked). Accordingly, much more variables can now be placed in Se. For variables beside noise, this is done by adding H*Kb*Sb*Kb'*H to Se (for variables with _DO=2). Note that data reduction can be needed to have enough memory. * The inversion characteristion is now handled in a general function called qp_invchar. For CLS, use qpcls_invchar. In the inversion characterization, variables with _DO = 1 or 2 are included in Se. * As Se is not needed for doing an inversion and in the inversion characterization Se can be different, Se is not longer saved by QP_SE and input/output of QP_SE is changed. If you will do an OEM (CLS) inversion, you shall now use QP_SEINV (qp_Seinv). See further below. * OEM functions and folder renamed to CLS. CLS stands for constrained least squares which will be used as a common name for OEM, Tikhonov regularization and (ordinary, weighted) least squares. So far is only OEM handled, but I plan to include the other two options some day (or night, as right now?) in the future. The functions qpcls and qpcls_invchar are used exactly as qpoem and qpoem_invchar. * Some parts are moved out from qpcls and are called qp_x2arts and qp_x2X. * qp_ycalc renamed to qp_y to be consistent. * Started implementation of ground emission but not yet finished. * To summerize, the standard set of function calls to make an OEM inversion is now: qp_Sx(Q); qp_H(Q); qp_Seinv(Q); qpcls(Q,y); To simylate a measurent: y = qp_y(Qy); 2001-10-10 Carlos Jimenez * qpack-0-0-25 * qp_rnd_atm fixed to be able to get parameterizatrions as retrieval tags e.g. rnd realizations of H2O-PWR98 means disturbing H2O profiles, not H2O_PWR98 profiles! 2001-10-09 Carlos Jimenez * qpack-0-0-24 * Changes in qp_rnd_atm, hse added to the rnd realizations, so if Q.HSE_IN_ON is set to 1 and temperature rnd realizations are done, the correponding z_abs for each t_abs fulfill hse according to the Q.HSE fields given in the Q srtucture defining the rnd data. The hse is calculated by arts/ami/physics/hseCalc, mscript that mimics the hse calculations inside arts, to get consistent results. Q.ALTIUDE_DO added in Templates/rnd.tmplt to flag do_z. * Changes in qp_rnd_atm, Q.POINTING_PDF added so the pointing offset can be defined as 'gaussian' or 'uniform'. As it will be nearly always gaussian, as default is set to 'gaussian', so if Q.POINTING_PDF is not given 'gaussian' is assumed. 2001-10-05 Patrick Eriksson * qpack-0-0-23 * Just included some references to AUG in README 2001-09-28 Patrick Eriksson * qpack-0-0-22 * Mainly changes to sync Qpack with all changes in ARTS. * qpoem can be runed for non-linear inversions again. * Continued work on retrieval of continuum absorption, but not finished. * ycalc.tmplt replaced with basic.tmplt. 2001-09-28 Patrick Eriksson * qpack-0-0-21 * Kx is now an output argument beside Dy when calling qpoem with only one input argumnent. 2001-09-27 Carlos Jimenez * qpack-0-0-20 * cleaning in /Main, qp_rnd, qp_rndxH, not in use any more 2001-09-20 Patrick Eriksson * qpack-0-0-20 * Small modifications in qpoem_invchar. The vertical resolution is not longer a return argument. 2001-09-17 Carlos Jimenez * qpack-0-0-19 * qp_rnd_sensor updated _DO > 0 makes realizations 2001-09-17 Carlos Jimenez * qpack-0-0-18 * qp_H saving also Hs (sensor matrix without reduction) * qp_Hd alowing all + hRedKx or hRedLimb * README updated in NUMBER_DO 2001-09-17 Patrick Eriksson * qpack-0-0-17 * Piecewise polynomial baseline fit included (PPOLYFIT). See README for more details. * The code of qp_Sx changed, but not the functionality (I hope). * The code of qp_Se also changed. The definition of the MEASNOISE_DO field changed to be consistent with the other do fields. Calibration noise can now be used again, but there is now no variable to tune the correlation between different viewing angles. A perfect correlation is assumed. * The field APRIORI_VMR_DIR changed to APRIORI_VMR as the field does not give the name of a directory. I know that this results in that you have to change a lot of control files, but this thing has been bugging for a long time and it is better do the change now than later. * The Sample directory updated to work with the new way of defining the thermal noise. 2001-09-03 Carlos Jimenez * qpack-0-0-16 * Big changes!!! * added qp_rnd_atm, qp_rnd_sensor, qp_rnd_atm_sensor mscripts to generate random sets, see the mscripts help to find out about their usage. The qp_rnd, qp_rndxH and qp_rndcheck are removed. The new mscripts do practically the same, but now the random realizations are done by Matlab mscripts, as the corresponidng part in arts is going to be removed. rnd.tmplt has also been updated to remove the old stuff and generate spectra in brightness temperature, after Qpack change of emission unit. * qp_H and qp_Hd slighly modified to allow the use of /ami/hmatrix/hRedKx and /ami/hmatrix/hRedLimb, the 1st one is reduction based on Kx, the 2nd a special reduction to use with limb observations optimized for each retrieval point and tag (obviously only to be used with a suitable reduction method, e.g an ensemble of neural nets). See the hRedKx and hRedLimb help and README for info. Notice that hRedLimb is still under tests, so do not use till further notice. * qp_Se updated to do rnd realizations using mscript sFromFile Major change, definition of measurement noise, check README. Only measurement noise, calibration to be decided how to implement it, other _DO = 2 also not implemented as in original qp_Se. * qp_Sx updated to also use sFromFile polynomial fit incorporated, continua still not implemented. * README is updated detailing new fields and so on. 2001-08-31 Patrick Eriksson * qpack-0-0-15 * The inclusion of polynomial fit finished. * It is now (more?) important to set f_sensor and za_sensor. These files will be used in some cases even if the corresponding sensor part is not included. If the backend or antenna not are included set F_SENSOR=F_MONO or ZA_SENSOR=ZA_PENCIL, respectively. * A bug in qpoem for non-linear inversion found. Not fixed!! 2001-08-31 Patrick Eriksson * qpack-0-0-14 * Started to include polynomial fit of baseline ripple. Not finished. * The unit used now for emission spectra is Rayleigh-Jeans brightness temperature. One reason to this is that the Sx matrix easily is singular when making baseline fits and using intensities. (I have written in AUG that brightness temperatures cannot be used for practical retrievals, but I think that that statement is wrong if the calibration is done correctly. More about this later. For simulations there is no problems.) 2001-08-24 Patrick Eriksson * qpack-0-0-13 * qpoem_inchar has now return variables. 2001-08-06 Carlos Jimenez * qpack-0-012 * /Main/qp_rnd changed so the spectra is saved without any sensor matrix applied, /Main/qp_rndxH added to do the multiplication. Why? Provides flexibility as it is possible to apply a previous reduction from a structure Qi to a new rnd set given by Q. See help. 2001-07-30 Carlos Jimenez * qpack-0-0-11 * Reduction based on weighting functions implemented, qp_Hd.m updated and new ami function/arts/ami/hmatrix/hRedKx.m added. New fields in Q structure KRED_ON, KRED_N, KRED_DEPTH. * Functions to generate random spectra /Main/qp_rnd and corresponding template /Templates/rnd.tmplt added, based on arts methods to generate random realizations, only species, temperature and pointing. They are PROVISIONAL, as a more detailed implementation not relying in arts methods is needed. 2001-06-13 Patrick Eriksson * qpack-0-0-10 * Smaller changes to cfile templates and qpoem. 2001-04-24 Patrick Eriksson * Qpack-0-0-9 * Put in temperature retrieval with HSE. Two new Q fields: PREF and ZREF. * Updated templates to changes in ARTS regarding HSE, the ground and refraction. * Fixed a bug in qpoem_invchar. 2001-04-18 Patrick Eriksson * Qpack-0-0-8 * More work on qpoem. Bugs fixed and some changes for input and output. Test retrievals have been performed for species, pointing off-set and temperature (yet only without hydrostatic eq.). However, the sample case is not suited for temperature retrievals but the iteration seems to work properly. * A maximum number of iterations introduced (OEM_MAX_ITER). * The positive constraint for non-linear species retrievals can now be toggled (SPECIES_POSITIVE_ON). No heavy tests for this option, but so far it looks good. 2001-04-17 Patrick Eriksson * Qpack-0-0-7 * Added functions etc. to easily simulate measurements corresponding to a Q structure. Noise is not yet included. The function doing the work is called qp_ycalc. A suitable Q for this function si found in Sample and it is called ycalc_q. * The Q-fields for the apriori data are renamed. See README. This to make it possible to select VMRs and PTZ from different directories. * Continua added. 2001-04-12 Patrick Eriksson * Qpack-0-0-6 * The OEM retrievals seem to work for species and pointing off-set. No bigger bug checks. 2001-04-11 Patrick Eriksson * Qpack-0-0-5 * Mainly a back-up commit. * I have started to work on the retrieval stuff (qpoem.m etc.) 2001-03-29 Patrick Eriksson * Qpack-0-0-4 * Included: oem_iter1.tmplt, qp_Se.m, qp_kinfo.m, qpi_Kx.m, qp_Hd.m, qpoem_invchar.m and qpoem_iter1.m * Modefied the sample files somewhat to better match Odin. * Changes to README and sample_q 2001-03-26 Patrick Eriksson * Qpack-0-0-3 * Made an organization into sub-folders. * The README file extended and a first version almost ready. * Created a sample case (found in Sample/). 2001-03-21 Patrick Eriksson * Qpack-0-0-2 * First, not finished, version of the README file. 2001-03-19 Carlos Jimenez * Qpack-0-0-1 * Initializing Qpack under cvs