%
% To start the document, use
%  \levela{...}
% For lover level, sections use
%  \levelb{...}
%  \levelc{...}
%
\levela{Data types, math operations and file formats}
 \label{sec:formats}
 
 This section defines the data types, basic mathematical operations
 and file formats supported by ARTS. The implementation of vectors,
 matrices and arrays are based on the Matrix Template Library (MTL)
 package, while files are created and read by using the Hierarchical
 Data Format (HDF). Some information and help to install these packages
 is given below.

%
% Document history, format:
%  \starthistory
%    date1 & text .... \\
%    date2 & text .... \\
%    ....
%  \stophistory
%
\starthistory
  001027 & Started by Patrick Eriksson. \\
%  00???? & First version finished by ??.\\
\stophistory


\levelb{Data types}
 \label{sec:formats:datatypes}

 \begin{table}[t]
  \begin{tabular}{p{4.5cm} p{4.5cm} p{4.5cm}}
   \verb|NUMERIC|        & \verb|size_t|         & \verb|string|         \\
  \end{tabular}
  \caption{The atomic data types of ARTS.}
  \label{table:format:atomic}
 \end{table}

 \begin{table}[t]
  \begin{tabular}{p{4.5cm} p{4.5cm} p{4.5cm}}
   \verb|MATRIX|         & \verb|SPARSE|         & \verb|MAYBESPARSE|    \\
   \verb|SYMMETRIC|      & \verb|BANDEDSYM|      & \verb|COVARMATRIX|    \\
   \verb|VECTOR|         &                       &                       \\
  \end{tabular}
  \caption{Numeric vector and matrix types in ARTS for which mathematical 
           operations are defined.}
  \label{table:format:nummath}
 \end{table}

 \begin{table}[t]
  \begin{tabular}{p{4.5cm} p{4.5cm} p{4.5cm}}
   \verb|ARRAYofVECTOR|  & \verb|ARRAYofMATRIX|  & \verb|ARRAYofsize_t|   \\
   \verb|ARRAYofstring|  &                       &                        \\
  \end{tabular}
  \caption{Array containers for the atomic types.}
  \label{table:format:atomicarrays}
 \end{table}

\levelc{Atomic types}
 \label{sec:formats:atomic}

 The most basic, the atomic, data types of ARTS are shown in Table
 \ref{table:format:atomic}. The \verb|NUMERIC| data type is either
 set to be double or float in \verb|arts.h|. If \verb|NUMERIC| is set
 to be double, the calculations will be more accurate and there is a
 smaller risk to encounter numerical problems in e.g. matrix inversions.
 On the other hand, \verb|NUMERIC| is set to be float the calculations
 will be more rapid (about double as fast). The type selected for
 \verb|NUMERIC| is also used when writing numeric data to binary files.
 
 A variable of type \verb|size_t| is a positive integer. This type is
 used for indexing e.g. vectors and matrices. The type is also used
 for all functions flags, i.e. to make a selction among a limited number of
 choices. Accordingly, characters or strings shall not be used flags. 

 A string is not a true ``atomic'' data type as it consists of a number
 of characters, but as characters are not used in ARTS, strings are
 the most basic text type in ARTS.
  
 !! Do we need  \verb|int| ? !!


\levelc{Numeric mathematical types}
 \label{sec:formats:nummath}
 
 Numeric values can be stored in vectors or a number of matrix types
 (Table \ref{table:format:nummath}) for which mathematical operations
 are possible. The allowed operations are given in Section !!.
 The vectors are treated at mathematical operations as column vectors.
 
 A \verb|MATRIX| is a general matrix and a \verb|SPARSE| is a general
 sparse matrix. The type \verb|MAYBESPARSE| is a structure type that
 allows a matrix to be either sparse or ``full''. 
 
 Two types exist for symmetric matrices, where \verb|SYMMETRIC| is the
 general type. The banded-symmetric type (\verb|BANDEDSYM|) is
 intended for matrices where all the non-zero elements are close to
 the diagonal of the matrix. Diagonal matrices are treated as a
 special case off banded-symmetric matrices. Covariance matrices are
 symmetric matrices. The type \verb|COVARMATRIX| is a structure that
 covers both general and banded symmetric matrices and can thus be
 used efficently for any covariance matrix.


\levelc{Arrays based on atomic types}
 \label{sec:formats:atomic_arrays}
 
 Arrays correspond to vectors but are not treated as mathematical
 objects, they are only used as containers to hold different data.
 The arrays (as vectors and matrices) have 0-based indexing, that is,
 the first element has index 0 (not 1). The existing arrays for the
 atomic data types are listed in Table \ref{table:format:atomicarrays}.
 A \verb|MATRIX| is basically an array of vectors, but if the vectors
 have different length they must be stored as an \verb|ARRAYofVECTOR|.


\levelc{Other data types}
 \label{sec:formats:others}

 \begin{table}[t]
  \begin{tabular}{l l l}
   \verb|LineRecord| & \verb|ARRAYofLineRecord| & \verb|ARRAYofARRAYofLineRecord| \\
   \verb|OneTag|     & \verb|Tags|   & \\
   \verb|Los| & & \\
  \end{tabular}
  \caption{ARTS data types using a mix of atomic types (structures).}
  \label{table:format:structures}
 \end{table}
 
 Stefan or Axel can you check Table \ref{table:format:structures} and
 write something here!!
 
 The \verb|Los| is a structure to describe the line of sight (LOS).
 The structure holds the pressures along the LOS. To make the
 calculations more efficient for 1D calulations, only one half of the
 LOS is stored.  For this reason, the LOS structure includes indecies
 to describe the iteration order. The structure also contains the
 index for ground reflections and the geometrical step length along
 the LOS. The LOS is further described in Section \ref{sec:los}.


\levelb{Mathematical and logical operations}
 \label{sec:formats:maths}
 
 \levelc{The atomic types} 
 All standard mathermatical (\verb|+|,\verb|-|,\verb|*|,\verb|/| etc.)
 and logical operations (\verb|==|, \verb|!=| etc.) of C++ can of
 course be used for \verb|NUMERIC| and \verb|size_t|. !! Strings, ==
 and !=, something else? !!
 
 \levelc{Vectors and matrices} 
 
 Usage and allowed mathematical operations for the different vectors
 and matrices are best shown by an example. The following file
 (\verb|test_mat.cc|) is compiled together with ARTS and contains thus
 only defined ARTS commands: {\footnotesize
   \verbatiminput{../../src/test_math.cc} }
 
 \levelc{Arrays} 
 No mathematical or logical oeprations are defined for the arrays (see
 Table \ref{table:format:atomicarrays}). The arrays are only used as
 containers to store data.


\levelb{File formats}
 \label{sec:formats:files}

 All ARTS data can be stored to, or loaded from, binary files using HDF.
 For some data types an ASCII file format also exists.
 
 \levelc{ASCII}
  \label{sec:formats:file:ascii}
  
  All data types based on \verb|NUMERIC| and \verb|size_t| that can be
  represented by \verb|ARRAYofMATRIX| are stored using a common ASCII
  file format. Table \ref{table:format:am} gives the data types that
  fulfills this criteria. The default extension for these ASCII files 
  is \verb|.am|.  These \verb|.am| files have the following 
  structure: \\
  {\footnotesize \begin{verbatim} 
# The file can start with an arbitrary number of comment lines.  
# These lines starts with the hash symbol (#) 
# The first row after the comment lines give the number of matrices 
# in the array. After this follows, for each matrix, a row giving 
# the matrix size followed by the data in row order.  
2 
2 3 
1.1 2.2 3.3 
4.4 5.5 6.6 
1 1 
3.1415
 \end{verbatim} 
}
     
 The sizes given in the file must be compatible with the data type
 of the variable that is read. Vectors can be given both as
 columns or row matrices. For \verb|size_t|, the ordinary type
 conversion of C++ is applied (\verb|(size_t)| when reading and
 \verb|(NUMERIC)| when writing).
 
 The types \verb|string| and \verb|ARRAYofstring| are stored using a
 similar file format. The default extension is here \verb|.as| and
 these files have the following structure:
 {\footnotesize \begin{verbatim} 
# The file can start with an arbitrary number of comment lines.
# These lines starts with the hash symbol (#)
# The first row after the comment lines give the number of strings
# in the array, followed by the strings (one on each row).  
3
String 1
String 2
String 3
 \end{verbatim} 
}


 \begin{table}[t]
  \begin{tabular}{p{4.5cm} p{4.5cm} p{4.5cm}}
   \verb|NUMERIC|        & \verb|VECTOR|         & \verb|MATRIX|          \\
   \verb|ARRAYofVECTOR|  & \verb|ARRAYofMATRIX|  &                        \\
   \verb|size_t|         & \verb|ARRAYofsize_t|  &                        \\
  \end{tabular}
  \caption{ARTS data types that can be stored with the numeric ASCII
           file format (.am).}
  \label{table:format:am}
 \end{table}
 

\levelc{Binary}
 \label{sec:formats:file:binary}
  
 Binary files are created and read by using HDF (Sec.
 \ref{sec:formats:hdf}). 

 


\levelb{MTL}
 \label{sec:formats:mtl}

 The MTL home page is found at \\

 \verb|http://www.lsc.nd.edu/research/mtl/| \\

 \noindent
 The MTL version used presently is 2.1.2 (Check if correct). 
 Further instructions should be given here.


\levelb{HDF}
 \label{sec:formats:hdf}

 The HDF home page is found at \\

 \verb|http://hdf.ncsa.uiuc.edu/| \\

 \noindent
 The present version of ARTS has been tested with HDF 5.1.2.2. The C
 implementation of HDF is used.
 
 HDF is not supplied with ARTS, it must be installed seperately as a
 library. For example, to install HDF on a Linux system, do the following:
 \begin{itemize}
  \item[1] Download the C package for gcc and ``shared''. Unpack.
  \item[2] Copy the contents of /bin, /include and /lib to the corresponding
           sub-directories of /usr/local. You need to be superuser to do 
           this.  
 \end{itemize}


%%% Local Variables: 
%%% mode: latex 
%%% TeX-master: "uguide" 
%%% End:

