%
% To start the document, use
%  \levela{...}
% For lover level, sections use
%  \levelb{...}
%  \levelc{...}
%
\levela{Data types, math operations and file formats}
 \label{sec:formats}
 
 This section defines the data types, basic mathematical operations
 and file formats supported by ARTS. The implementation of vectors,
 matrices and arrays are based on the Matrix Template Library (MTL)
 package, while files are created and read by using the Hierarchical
 Data Format (HDF). Some information and help to install these packages
 is given below.

%
% Document history, format:
%  \starthistory
%    date1 & text .... \\
%    date2 & text .... \\
%    ....
%  \stophistory
%
\starthistory
  001027 & Started by Patrick Eriksson. \\
%  00???? & First version finished by ??.\\
\stophistory


\levelb{Data types}
 \label{sec:formats:datatypes}

 \begin{table}[t]
  \begin{tabular}{p{4.5cm} p{4.5cm} p{4.5cm}}
   \verb|NUMERIC|        & \verb|INDEX|         & \verb|STRING|         \\
  \end{tabular}
  \caption{The atomic data types of ARTS.}
  \label{table:format:atomic}
 \end{table}

 \begin{table}[t]
  \begin{tabular}{p{4.5cm} p{4.5cm} p{4.5cm}}
   \verb|MATRIX|         & \verb|SPARSE|         & \verb|MAYBESPARSE|    \\
   \verb|SYMMETRIC|      & \verb|BANDEDSYM|      & \verb|COVARMATRIX|    \\
   \verb|VECTOR|         &                       &                       \\
  \end{tabular}
  \caption{Numeric vector and matrix types in ARTS for which mathematical 
           operations are defined.}
  \label{table:format:nummath}
 \end{table}

 \begin{table}[t]
  \begin{tabular}{p{4.5cm} p{4.5cm} p{4.5cm}}
   \verb|ARRAYofVECTOR|  & \verb|ARRAYofMATRIX|  & \verb|ARRAYofINDEX|   \\
   \verb|ARRAYofSTRING|  &                       &                        \\
  \end{tabular}
  \caption{Array containers for the atomic types.}
  \label{table:format:atomicarrays}
 \end{table}

\levelc{Atomic types}
 \label{sec:formats:atomic}

 The most basic, the atomic, data types of ARTS are shown in Table
 \ref{table:format:atomic}. The \verb|NUMERIC| data type is either
 set to be double or float in \verb|arts.h|. If \verb|NUMERIC| is set
 to be double, the calculations will be more accurate and there is a
 smaller risk to encounter numerical problems in e.g. matrix inversions.
 On the other hand, when \verb|NUMERIC| is set to be float the calculations
 will be more rapid (about double as fast). The type selected for
 \verb|NUMERIC| is also reflected in the size of output files.
 
 A variable of type \verb|INDEX| is a positive integer. This type is
 used for indexing e.g. vectors and matrices. The type is also used
 for all functions flags, i.e. to make a selction among a limited number of
 choices. Accordingly, characters or strings shall not be used flags. 

 A string is not a true ``atomic'' data type as it consists of a number
 of characters, but as characters are not used in ARTS, strings are
 the most basic text type in ARTS.
  


\levelc{Numeric mathematical types}
 \label{sec:formats:nummath}
 
 Numeric values can be stored in vectors or a number of matrix types
 (Table \ref{table:format:nummath}) for which mathematical operations
 are possible. The allowed operations are given in Section 
 \ref{sec:formats:maths}.
 The vectors are treated in mathematical operations as column vectors.
 
 A \verb|MATRIX| is a general matrix and a \verb|SPARSE| is a general
 sparse matrix. The type \verb|MAYBESPARSE| is a structure type that
 allows a matrix to be either sparse or ``full''. 
 
 Two types exist for symmetric matrices, where \verb|SYMMETRIC| is the
 general type. The banded-symmetric type (\verb|BANDEDSYM|) is
 intended for matrices where all the non-zero elements are close to
 the diagonal of the matrix. Diagonal matrices are treated as a
 special case of banded-symmetric matrices. Covariance matrices are
 symmetric matrices. The type \verb|COVARMATRIX| is a structure that
 covers both general and banded symmetric matrices and can thus be
 used efficently for most covariance matrices.

 !! Write something about indexing when the switch to MTL is ready. !!


\levelc{Arrays based on atomic types}
 \label{sec:formats:atomic_arrays}
 
 Arrays correspond to vectors but are not treated as mathematical
 objects, they are only used as containers to hold different data.
 The arrays (as vectors and matrices) have 0-based indexing, that is,
 the first element has index 0 (not 1). The existing arrays for the
 atomic data types are listed in Table \ref{table:format:atomicarrays}.
 A \verb|MATRIX| is basically an array of vectors, but if the vectors
 have different length they must be stored as an \verb|ARRAYofVECTOR|.


\levelc{Other data types}
 \label{sec:formats:others}

 \begin{table}[t]
  \begin{tabular}{l l l}
   \verb|LineRecord| & \verb|ARRAYofLineRecord| & \verb|ARRAYofARRAYofLineRecord| \\
   \verb|OneTag|     & \verb|Tags|   & \\
   \verb|LOS| & & \\
  \end{tabular}
  \caption{ARTS data types using a mix of atomic types (structures).}
  \label{table:format:structures}
 \end{table}
 
 !! Stefan or Axel can you check Table \ref{table:format:structures} and
 write something here !!
 
 The \verb|LOS| is a structure to describe the line of sight (LOS).
 The structure holds for example the pressures along the LOS. To make the
 calculations more efficient for 1D calulations, only one half of the
 LOS is stored.  For this reason, the LOS structure includes indecies
 to describe the iteration order. The structure also contains the
 index for ground reflections and the geometrical step length along
 the LOS. The LOS is further described in Section \ref{sec:los}.


\levelb{Mathematical and logical operations}
 \label{sec:formats:maths}
 
 \levelc{The atomic types} 
 All standard mathermatical (\verb|+|,\verb|-|,\verb|*|,\verb|/| etc.)
 and logical operations (\verb|==|, \verb|!=| etc.) of C++ can of
 course be used for \verb|NUMERIC| and \verb|INDEX|. Among the C++
 standard operations for strings, comparation using \verb|==| and
 \verb|!=| are probably the most important.

 
 \levelc{Vectors and matrices} 
 
 Appendix \ref{app:math} gives examples on usage and allowed
 mathematical operations for the different vector and matrix types.
 
 \levelc{Arrays} 
 No mathematical or logical oeprations are defined for the arrays (see
 Table \ref{table:format:atomicarrays}). The arrays are only used as
 containers to store data.


\levelb{File formats}
 \label{sec:formats:files}
 
 All ARTS data can be stored to, or loaded from, binary files using
 HDF.  For some data types an ASCII file format also exists (Table
 \ref{table:format:aa}). The default extension for ASCCI files is
 \verb|.aa| (ARTS ASCII) and for binary files it is \verb|.ab| (ARTS
 binary).
 
 \levelc{ASCII}
  \label{sec:formats:file:ascii}
  
  All data types based on \verb|NUMERIC| and \verb|INDEX| that can be
  represented by \verb|ARRAYofMATRIX| are stored using a common ASCII
  file format. Table \ref{table:format:aa} contains the data types that
  fulfills this criteria. Numeric ASCII files have the following 
  structure: \\
  {\footnotesize \begin{verbatim} 
# The file can start with an arbitrary number of comment lines.  
# These lines starts with the hash symbol (#) 
# The first row after the comment lines give the number of matrices 
# in the array. After this follows, for each matrix, a row giving 
# the matrix size followed by the data in row order.  
2 
2 3 
1.1 2.2 3.3 
4.4 5.5 6.6 
1 1 
3.1415
 \end{verbatim} 
}

 \noindent
 Index arrays (\verb|ARRAYofINDEX|) are stored as integer vectors to
 make the files easier to inspect.
     
 The sizes given in the file must be compatible with the data type
 of the variable that is read. Vectors can be given both as
 columns or row matrices. When reading varaibles of INDEX type,
 the data is checked to only contain non-negative integers.
 
 The types \verb|STRING| and \verb|ARRAYofSTRING| are stored using a
 similar file format. String ASCII files have the following structure:
 {\footnotesize \begin{verbatim} 
# The file can start with an arbitrary number of comment lines.
# These lines starts with the hash symbol (#)
# The first row after the comment lines give the number of strings
# in the array, followed by the strings (one on each row).  
3
String 1
String 2
String 3
 \end{verbatim} 
}

 \begin{table}[t]
  \begin{tabular}{p{4.5cm} p{4.5cm} p{4.5cm}}
   \verb|NUMERIC|        & \verb|VECTOR|         & \verb|MATRIX|          \\
   \verb|ARRAYofVECTOR|  & \verb|ARRAYofMATRIX|  &                        \\
   \verb|INDEX|          & \verb|ARRAYofINDEX|  &                        \\
   \verb|STRING|          & \verb|ARRAYofSTRING|  &                        \\
  \end{tabular}
  \caption{ARTS data types that can be stored using the ASCII
           file formats (.aa).}
  \label{table:format:aa}
 \end{table}
 

\levelc{Binary}
 \label{sec:formats:file:binary}
  
 Binary files are created and read by using HDF 4 (Sec.
 \ref{sec:formats:hdf}). The Vdata format is applied. Most data types
 are stored using a common approach but for some data types a special 
 format is used.

\leveld{General binary file format}
 \label{sec:formats:file:binary:general}
 
 The binary files for data types that can be treated as special cases
 of a matrix or can be broken down to a number of matrices have a
 common layout. In this context strings are treated as vectors of
 characters and scalars as 1x1 matrices. For example, the data types
 in Table \ref{table:format:aa} meat this criteria.
 
 A Vdata contains only a single scalar, vector or matrix. The file for
 an array contains thus a number of Vdatas. The fields of a Vdata
 contains a single number or character. The field order for the common
 format is thus throughout 1. Matrices are in row order, that is, 
 the data order is (1,1), (1,2), (1,3),\dots,(2,1),(2,2),\dots.
 
 The matrix dimensions are stored as an attribute to each Vdata. The
 name of the attribute is \verb|SIZE|. The order of \verb|SIZE| is 2
 where the first value is the number of rows and the second value is
 the number of columns.\footnote{The order of the size attribute can
   be increased to accomodate information needed for e.g. sparse
   matrices.}. The data type of \verb|SIZE| is unsigned 4 byte
 integers (= HDF type \verb|DFNT_UINT32|). Vectors and strings are
 treated as column objects (i.e. 1 column).
 
 The data type of the file data is indicated by the class name of the
 Vdata. \verb|INDEX| data are stored as 4 byte unsigned integers and
 the class name is set to \verb|UINT| (= HDF type \verb|DFNT_UINT32|).
 The class name for characters of strings is \verb|CHAR| and the file
 data type is 1 byte characters (= HDF type \verb|DFNT_CHAR|).

 The record size for floating point values can either be 4 or 8 bytes.
 The corrresponding class names are \verb|FLOAT| and \verb|DOUBLE|
 (= HDF type \verb|DFNT_FLOAT32| and \verb|DFNT_FLOAT64|, respectively).
 The type of \verb|NUMERIC| determines the file type when writing from
 ARTS. Data is automatically converted to the type of \verb|NUMERIC|
 when reading file data.
 
 The field name of a Vdata (remember that a Vdata has here only a
 singel field) describe the structure of the data. Treated structure
 types are \verb|SCALAR|, \verb|VECTOR|, \verb|MATRIX| and \verb|STRING|.
 The following combinations of data structure and data type are allowed:
 \begin{verbatim}
    SCALAR: UINT, FLOAT, DOUBLE 
    VECTOR: UINT, FLOAT, DOUBLE 
    MATRIX: FLOAT, DOUBLE 
    STRING: CHAR
 \end{verbatim} 
 The Vdata name is used to handle arrays. However, for simplicity
 reasons and for consistency with the ASCII format, index arrays
 (\verb|ARRAYofINDEX|) are stored as index vectors (field name
 \verb|VECTOR| and Vdata class \verb|UINT|). For other type of
 arrays, the Vdata name is set to the data structure name followed
 by a sequential number (starting at 1). The length of the array
 is given by a seperate Vdata holding an index number (\verb|SCALAR|,
 \verb|UINT|), named as \verb|N_|{\it string\_type}. An example should
 clarify this. A file holding a matrix array of length 3 has the
 the Vdatas \verb|N_MATRIX|, \verb|MATRIX1|, \verb|MATRIX2| and
 \verb|MATRIX3|.

 The described approach to store binary data results in that each
 ARTS data type has a corresponding format for binary files. This 
 gives an automatic check that a file matches the data type of
 an ARTS variable when reading from a binary file. The drawback
 is that, for example, a file holding a matrix cannot be read to create
 an matrix array of length 1, which is possible for the ASCII format,
 but consistency was emphasized when designing the binary format.
 In addition, the size attribute is used to check that the data
 have the expected size, for example, vectors are expected to only
 have one column.
 

\leveld{Other binary file formats}
 \label{sec:formats:file:binary:others}

 Spectroscopic data and tag information are better stored using
 structure data fields (constructs). !! Or? !!


\leveld{Display tools}
 \label{sec:formats:file:binary:display}

 The content of the binary files can be displayed using some command
 line HDF utilities. A first utility is
 \verb|vshow|. The syntax is \\

 \verb|vshow |{\it filename}\verb| +|

 \noindent
 where {\it filename} is the binary file of interest. The final \verb|+|
 indicates that the values of the stored data shall be displayed. Without
 the \verb|+| symbol, only the data structure is reported.

 Another utility is \verb|hdp|. Type \verb|hdp -h| for some on-line help.
 To display the values of the stored data, type\\
 
 \verb|hdp dumpvd |{\it filename}



\levelb{MTL}
 \label{sec:formats:mtl}

 The MTL home page is found at \\

 \verb|http://www.lsc.nd.edu/research/mtl/| \\

 \noindent
 The MTL version used presently is 2.1.2. !! Stefan or Oliver, please check
 the version number and write something about installing MTL. !!


\levelb{HDF}
 \label{sec:formats:hdf}

 The HDF home page is found at \\

 \verb|http://hdf.ncsa.uiuc.edu/| \\

 \noindent
 The HDF 4 data format is used. The present version of ARTS has been
 tested with HDF 4.1r3. The precompiled binaries were used.
 
 HDF is not supplied with ARTS, it must be installed seperately as a
 library. For example, to install HDF on a Linux system, try the
 following:
 \begin{itemize}
  \item[1] Download the precompiled version for your system. Unpack.
  \item[2] Copy the contents of /bin, /include, /man and /lib to the 
           corresponding sub-directories of /usr/local. You need to be 
           superuser to do this.  
 \end{itemize}


%%% Local Variables: 
%%% mode: latex 
%%% TeX-master: "uguide" 
%%% End:

