\chapter{Technical manual}

\section{Logging}

The Apache log4cxx framework is used for logging in the project. To use a logger in a custom class you should
create a LoggerPtr variable, for example:

\begin{lstlisting}
static log4cxx::LoggerPtr logger(log4cxx::Logger::getLogger("SuperMatrix.matrices.io.SimMatrixReader"));
\end{lstlisting}

As a name of the logger we use relative path of the file with dots as a path separators. This enables grouping
of loggers, filtering of a logfile by a module, etc.

Example of logger usage in the code: 

\begin{lstlisting}
LOG4CXX_WARN(logger, "Warning - empty line in file " << lfilename.c_str() << ". Line no = " << id);
LOG4CXX_INFO(logger, "Reading similarity db from file " << filename);
\end{lstlisting}

Each logger and each logged information has level. Only informations with higher or equal level than level of
logger are written. There are six standard levels and they have the following order: 

\begin{verbatim}
TRACE < DEBUG < INFO < WARN < ERROR < FATAL
\end{verbatim}

Loggers should be configured. For loading custom configuration from file one can use
\verb|PropertyConfigurator::configure(fileName)|. If loggers are not configured in the code, default
configuration will be read from \textit{log4cxx.xml}. Sample \textit{log4cxx.xml}:

\begin{lstlisting}
<?xml version="1.0" encoding="UTF-8" ?>
 <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
  <appender name="A1" class="org.apache.log4j.ConsoleAppender">    
    <param name="Target" value="System.out"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%-4r [%t] %-5p %c %x - %m%n"/>
        </layout>
  </appender>
  <root>
        <priority value="info" />
        <appender-ref ref="A1"/>
  </root>
  <category name="SuperMatrix.matrices.io" >
        <priority value ="error" />
        <appender-ref ref="A1"/>
  </category>
 </log4j:configuration>
\end{lstlisting}

In the above example loggers will write to the console. Loggers with name that starts with
\verb|SuperMatrix.matrices.io| will show only the messages above ERROR level. And level of other loggers is
set to INFO. For further details see \emph{log4cxx} documentation available at
\url{http://logging.apache.org/log4cxx/}.

\section{File formats}
\subsection{SuperMatrix}
\label{sec:SuperMatrixFormat}
SuperMatrix is loaded on the basis of 8 files. Each contains information describing the contents of the matrix.

\subsection*{boundary.txt}
operator/relation boundaries or empty set, if no boundaries are defined for this matrix
\subsection*{info.txt}
	Matrix description
\subsection*{name.clabel}
	Each line contains a label the next column of the matrix.
\subsection*{name.rlabel}
	Each line contains a label the next row of the matrix. The label consists of the parts of speech and the name of a row separated by a colon.

\subsection*{matrix\_css.mat}

	File describes column part of the matrix. The first line of the file contains information about the number of columns, rows and non-zero cells.
	Next lines are representation of rows in the SuperMatrix. Each line consists of pairs of numbers. First is an index of the column and the second 
	is cell value for this index in the line. \\

In example:
\begin{lstlisting}
12 9 28
0 1 1 1 2 3
2 1 3 1 4 1 5 1 6 1 8 1
1 1 3 1 4 1 7 1
0 1 4 2 7 1
3 1 5 1 6 1
9 1
9 1 10 1 
9 1 10 1 11 1
8 1 10 1 11 1

\end{lstlisting}

This matrix has 12 columns, 9 rows and 28 non-zero cells.
Based on the first row, we can read the columns of SuperMatrix with indexes 0 and 1 have value 1, and column with index 2 has value 3.

\subsection*{matrix\_crs.mat}
	File describes row part of the matrix. The first line of the file contains information about the number of columns, rows and non-zero cells.
	Next lines are representation of columns in the SuperMatrix. Each line consists of pairs of numbers. First is an index of the row and the second 
	is cell value for this index in the line.
\label{sec:rowFeat}
\subsection*{row\_features.csv}
	File contains information about features of SuperMatrix rows. \\
\begin{tabular}{l p{10cm}}

label &- parts of speech and the name of a row separated by a colon \\
nz &- number of non-zero cells \\
tf &- term frequency in a row \\ 
TF &- global term frequency \\
entropy &- entropy value of the line\\
\end{tabular}
\subsection*{column\_features.csv}
Just as at \ref{sec:rowFeat}, but for columns.

\label{sec:fileformats}
\subsection{WSD Reader}

Word Sense Disambiguation Reader requires lexcsd application.

To use WSD Reader you need configuration file and another reader which can read your corpus.\\ Format of configuration:
\begin{lstlisting}
work-dir=/home/user/workdir/
classifier-config=/home/user/lexcsd/lexcsd/apps/feature_classifier.ini
models-dir=/home/user/models/
weka-server=/home/user/wekaServer.jar
\end{lstlisting}

\verb|work-dir| is a directory for temporary files. \verb|classifier-config| contents path to configuration file for Feature Classifier (lexcsd application).
Next parameter - \verb|models-dir| - is a directory with models files. Last option points the jar file with weka server. You can find it in lexcsd.

Using WSD Reader:

\begin{verbatim}
wsdreader,config:/home/user/wsdreader/config.cfg,inner:poliqarp
\end{verbatim}


\section{Tools}
\label{ss:architectTools}
\subsection{basedict}\label{sec:basedictTool}
\subsection*{Overview}
Basedict is an application used to create dictionary of lemmas based on corpus(corpora). It can filter corpora for specific part of speech and/or 
minimum frequency of lemmas. As the result is returned file in format $flex;lemma;frequency$, where first line is with this description. Application is located in package /tools/architect2/basedict 

\subsection*{Parameters description}

\begin{tabular}{l p{8cm}}
--tagset arg (=kipi) &   Name of a tagset. \\
--reader arg         &  Name of a reader. \\
--corpus arg         &  Path(s) to corpus. \\
--posfilters arg     &  Filters for part of speech. If set then only specified POS will be stored. If not set then all POS will be stored. \\
--output arg         &  Path for file with results. \\
--pos arg (=1)       &  If true then part-of-speech will be preserved. \\
--disamb-only arg (=1) & If true then only disambiguated lemmas will be saved. \\
--mintf arg (=0)     &  Min frequency of lemmas to store in the output file. \\


\end{tabular}
\subsection*{Usage}
To use basedict you need corpus, proper reader for it and output path. The rest of parameters is optional. To work with multiple corpora you have to add
 \verb|--corpus| parameter, as many as you have corpora.

\begin{verbatim}
$ basedict --corpus /home/user/corpus --reader poliqarp 
--output /home/user/dictionary.txt
\end{verbatim} 

\subsection{builder}\label{sec:builderTool}
\subsection*{Overview}
Builder is a program used to extract from corpora related lemmas. Relation is determined by wccl operators. As the result is returned file with tuples in format $flex:lemma;relation;flex:lemma;frequency$

\subsection*{Parameters description}

\begin{tabular}{l l l}
-t & [ --tagset ] arg  & Name of a tagset \\
-r & [ --reader ] arg  & Name of a reader \\
-c & [ --corpus ] arg  & Path(s) to corpus \\
-w & [ --operator ] arg  & File with WCCL operator \\
-o & [ --outfile ] arg & Path to file with results \\
-d & [ --dump-interval ] arg & Dump interval (in sentences) \\
-a & [ --append-pos ] arg & If true then lemmas will be prefixed with POS \\
\end{tabular}

\subsection*{Usage}
To use builder you need corpus, proper reader for it, wccl operator and output file.  
\begin{verbatim}
$ builder --corpus /home/user/corpus --reader poliqarp 
--operator /home/user/operators/operator.ccl
--output /home/user/output.tuples
\end{verbatim} 

\subsection{windowbuilder}\label{sec:windowbuilderTool}
\subsection*{Overview}
Builder is a program used to extract tuples for building window matrices from Polish corpora.

\subsection*{Parameters description}

\begin{tabular}{l l l}
-t & [ --tagset ] arg (=kipi) & Name of a tagset \\
-r & [ --reader ] arg & Name of a reader \\
-c & [ --corpus ] arg & Path(s) to corpus \\
-o & [ --outfile ] arg (=out.csv) & Path for file with results \\
-d & [ --dump-interval ] arg (=100000) & Dump interval (in sentences) \\
& --row-poses arg & Row POSes. If not set then all poses will be used. \\
& --column-poses arg & Column POSes. If not set then all poses will be used. \\
& --row-words-file arg & File with words for row. Every line consists of  \\
                                  & &     pos:baseform. If not set then all words will be used (after \\
                                  & &     POS filtering) \\
& --column-words-file arg & File with words for columns. Every line consists of \\
                                  & &     pos:baseform. If not set then all words will be used (after \\
                                  & &     POS filtering) \\
& --simple arg (=0) & If set to true, no order of word information will be \\
                                  & &     retained. \\
& --max\_window arg (=100000) & Max window size. \\
\end{tabular}

\subsection*{Usage}
To use windowbuilder you need corpus and proper reader for it.
\begin{verbatim}
$ windowbuilder -c /home/user/corpus -r poliqarp -o /home/user/output.tuples
--max_window 5
\end{verbatim} 

\subsection{doctermbuilder}\label{sec:doctermbuilderTool}
\subsection*{Overview}
Doctermbuilder is a program used to extract tuples for building document$\times$term matrices from Polish corpora. 

\subsection*{Parameters description}

\begin{tabular}{l l l}
-t & [ --tagset ] arg (=kipi) & Name of a tagset \\
-r & [ --reader ] arg & Name of a reader \\
-f & [ --filelist ] arg & Path to corpus (a file list) \\
-o & [ --outfile ] arg (=out.csv) & Path for file with results \\
-d & [ --dump-interval ] arg (=100) & Dump interval (in documents) \\
& --column-poses arg & Column POSes. If not set then all poses will be used. \\
& --column-words-file arg & File with words for columns. Every line consists of \\ & & pos:baseform. If not set then all words will be used (after POS \\
	& & filtering) \\
\end{tabular}

\subsection*{Usage}
To use doctermbuilder you need corpus (as a list of files) and proper reader for it.
\begin{verbatim}
$ doctermbuilder -f /home/user/filelist -r xces -o output.tuples
\end{verbatim} 

\subsection{filtrator2}
\subsection*{Overview}
Filtrator2 is a program used to remove rows/columns from a matrix based on multiple criteria. As the result is returned matrix in SuperMatrix format.

\subsection*{Parameters Description}
\begin{tabular}{l p{8cm}}
--matrix-name arg & Matrix name.\\
--matrix-dir arg & Matrix directory.\\
--out-name arg & Name of output matrix.\\
--out-dir arg (=./wd) & Name of output dir.\\
--words arg  & Name of file with words to extract (only those words  will be kept in the resulting matrix).\\
--mode arg (=both)& Filtering mode. One of: row, column, both.\\
--max-iters arg (=1)  & Maximum iterations for combined row and column filtering.\\
--min-tf arg (=5)& Minimum term frequency.\\
--min-TF arg (=5) & Minimum global term frequency.\\
--min-nz arg (=5)  & Minimum term non-zero count.\\
--min-entropy arg (=-1)& Minimum term entropy.\\
--max-tf arg & Maximum term frequency.\\
--max-TF arg & Maximum global term frequency.\\
--max-nz arg & Maximum term non-zero count.\\
--max-entropy arg & Maximum term entropy.\\
\end{tabular}
\subsection*{Usage}
To use \verb|filtrator2| you need to have matrix(SuperMatrix), specify filtration criteria and output pathes. If you want extract from matrix selected words, you have to set parameters \verb|--words| and \verb|--mode|.

\begin{verbatim}
$ filtrator2 --matrix-name supermatrix --matrix-dir /home/user/matrices/
--out-name newMatrix --out-dir /home/user/matrices/filtered/ --mode column
--words /home/user/columns.txt
\end{verbatim}

\subsection{parallelSimilarity}\label{sec:parallelSimilarity}
\subsection*{Overview}

Parallel similarity computed similarity between given(or all) rows with all other rows of matrix. Application is located in package parallel/similarity

\subsection*{Parameters}
\begin{tabular}{l p{7cm}}
--matrix-dir arg			&Path with supermatrix to be computed\\
--matrix-name arg			&Name of matrix to be computed\\
--measure-of-relatedness arg	&Measure of relatedness with all parameters for FunctionFactory\\
--results-file arg			&Ouput file for results\\
--results-type arg			&Type of storage for results, one of: supermatrix, textfile, densematrix\\
--read-ccs					&Whether matrix is read from ccs file\\
--read-crs					&Whether matrix is read from crs file\\
--minimum-similarity arg	&Minimal value for result to be considered important\\
--k-best-results arg		&How many results should be stored for each computed row\\
--target-words-file arg		&File with specific words to be computed\\
--transform-matrix arg (=0)       & Whether to transform matrix before computing or not (on original matrix)\\

\end{tabular}\\*
Parameters for grid with installed MPI:


\begin{tabular}{l p{10cm}}
-n	&Number of nodes, on which we will compute. 1 of nodes is master node dividing work between slaves.\\
-hostfile	&File with hostnames of available machines. Should be provided by administrator.\\
\end{tabular}

\subsection*{Usage}

To use \verb|parallelSimilarity| besides its own option, we have to also set \verb|mpirun| arguments. 

Computing similarity is based on a comparison of two SuperMatrix rows. In normal way, rows are processing by measure
of relatedness only in case they are taking part in computing similarity. If we set option \verb|--transform-matrix|  then all rows in SuperMatrix will be processing befor process of computing similarity, regardless of whether they participate in it or not. This option is usefull if we are checking similarity for all rows or using \verb|transformation| in measure of relatedness.
 

\begin{verbatim}
mpirun -n 24 --hostfile hostfile ./parallelSimilarity 
--matrix-dir /home/user/matrices --matrix-name matrix
--results-type supermatrix --measure-of-relatedness "lin()"
--results-file /home/user/result.txt
--read-crs --minimum-similarity 0.001 --k-best-results 16000
\end{verbatim}

\subsection{similarity\_results\_labeler}\label{sec:simLabeler}
\subsection*{Overview}
Similarity\_results\_labeler for each word from similarity matrix extract list of k most similar words.
\subsection*{Parameters description}
\begin{tabular}{l p{6cm}}
--matrix-filename arg			& Name of the matrix \\
--matrix-type arg               	& Type of storage matrix, one of: supermatrix, textfile, densematrix \\
--label-file arg                   	& File with row labels \\
--results-file arg                 	& File storing results \\
--minimum-similarity arg (=0.0001)	& Minimum similarity of rows \\
--k-best-results arg (=20)         	& Number of best results to display \\
--selfsim                          	& If similarity to self should be written \\
\end{tabular}
\subsection*{Usage}
To use similarity\_results\_labeler you need matrix, its type, file with labels of rows in matrix and file to store results.

\begin{verbatim}
$./labeler --matrix-filename /path/to/matrix --matrix-type textfile 
--results-file /path/to/output_file --label-file /path/to/label_file 
--k-best-results 5
\end{verbatim}
`
\subsection{tuplesproc}\label{sec:tuplesprocTool}
\subsection*{Overview}
Tuplesproc is a tool used to build co-occurrence matrix based on tuples with related lemmas and dictionary of lemmas. Application supports filtering rows and columns in created matrix by frequency and specific words/relations. As the result is returned matrix in SuperMatrix format (\ref{sec:SuperMatrixFormat}).
\subsection*{Parameters description}

\begin{tabular}{l p{8.5cm}}
--input arg                                   & Directory or file with tuples. Exclusive with input-list. \\
--include arg                                  & File name filter, when using input option only files matching this pattern will be parsed.\\
--file-list arg                                &  Path to a file which contains files list to parse. Exclusive with input option. \\
--out arg                                      &  Output directory. Required! \\
--matrix-name arg                              & Matrix name. Required! \\
--matrix-description arg			& Optional description of a matrix \\
--tokens arg (=0)                              &  Number of tokens in a corpus. \\
--dict arg                                     &  File with (global) term frequency dictionary \\
--row-file arg                                 &  File which contains set of rows to write in SuperMatrix \\
--column-file arg                              &  File which contains set of columns to write in SuperMatrix \\
--relation-file arg                            &  File which contains set of relation to write in SuperMatrix \\
--relations arg                                &  Relations to write in SuperMatrix\\
--row-min-tf arg (=0)                          &  Minimum value of term frequency in rows features\\
--col-min-tf arg (=0)                          &  Minimum value of term frequency in columns features \\
--interval arg (=1000000)                      &  how often to print progress (in lines) during reading of tulpes files\\
\end{tabular}

\subsection*{Usage}
To use tuplesproc you need files with tuples, file with dictionary of lemmas and output directory. You also need to set a name of generating matrix. 

\begin{verbatim}
$ tuplesproc --input /home/user/tuples --dict /home/user/dictionary.txt 
--output /home/user/matrices/ --matrix-name NewMatrix 
\end{verbatim}

There are two ways to set files with tuples. In first, with using parameter \verb|--input|, you can set path to tuples file or directory where the tuples files are. If you have these files in different places you can use option \verb|--file-list|. In this way you have to set path to file which contains pathes to tuples files (each path in separate line). You cannot use this options together. 

Filtering options \verb|--row-file \verb|--column-file| and \verb|relation-file| work in similar way. These parameters take as argument path to the file, which should contains set of specific label of columns/rows or name of relations. Each label or name have to be in separate line. 

\label{sec:tuplesTools}
\subsection{accumulate}
\subsection*{Overview}
Accumulate is a script for summing up frequencies in tuples. Script is located in tools/architect2/scripts.
\subsection*{Parameters description}
\begin{tabular}{l p{8.5cm}}
--infile arg &    Input file with tuples \\
--outfile arg &   Output file \\

\end{tabular}

\subsection{window\_util}
\subsection*{Overview}
Window\_util is a script for proccessing window tuples. Script is located in tools/architect2/scripts/tuples.
\subsection*{Parameters description}
\begin{tabular}{l p{8.5cm}}
-t arg		& Path to tuples file \\
-d		& Removing information about window size \\
-l arg		& Maximum left size of window \\
-r arg		& Maximum right size of window \\
-o arg		& Path to output file \\
\end{tabular}

\subsection{tuples\_merger}
\subsection*{Overview}
Tuples\_merger is a script for merging files with tuples. Script is located in tools/architect2/scripts/tuples.
\subsection*{Parameters description}
\begin{tabular}{l p{8.5cm}}
--tuples arg		& Path(s) to tuples file \\
--tuples-dir arg 	& Directory with tuples to merge \\
--output arg		& Path to output file \\
--remove-words arg	& Path to file with words to exclude \\
--remove-flex arg	& Path to file with parts of speech to exclude \\
\end{tabular}

\subsection{tuple\_filter}
\subsection*{Overview}
Tuple\_filter is a script for filtering tuples. Script is located in tools/architect2/scripts.
\subsection*{Parameters description}
\begin{tabular}{l l p{8.5cm}}
-i & [--in] arg 		& File with tuples \\
-o & [--out] arg		& Output file \\
& --col-pos arg		& File with parts of speech to remove from columns \\
& --row-pos arg		& File with parts of speech to remove from rows \\
& --col-words arg	& File with words to remove from columns \\
& --row-words arg	& File with words to remove from rows \\
-d & 			& Remove wrong lines \\
\end{tabular}

\section{Design}

\section{API description}

\todo{Imported from Doxygen?}

\section{Function Factory}
\todo{Old FF description}

\section{Function Factory 2}
\label{sec:ff2}

Function Factory 2 is the successor class to Function Factory, created with ease of creating new MSRs by combining existing weight, similarity functions, filters etc. in mind. It uses a similar notation in its string format:

$id(arg1=val1,...,argn=valn)$

Where:

$id$ --- the identifier string of the described entity

$arg1$ --- the identifier string of the entity's argument

$val1$ --- the value of the given argument

One of the major changes in comparison to Function Factory is the possibility of argument nesting: val1 may be another Function Factory 2 string, describing a complex object:

$id\_outer(arg1=val1, arg2=id\_inner(arg21=val21))$

Function Factory 2 strives to provide helpful and human-readable information on parsing errors. The following errors are possible (and handled):

\begin{itemize}
	\item invalid call format (other than id(arg=val,...))

	\item invalid function object identifier (for example: an unknown filter identifier)

	\item invalid argument for a given function object (for example: a 'noise' parameter for a filter, which doesn't have one)

	\item invalid argument type for a given argument (boost::lexical\_cast exceptions)

	\item missing a required argument for a given function object

\end{itemize}

\subsection{Creating objects}

The Create methods are responsible for creating objects. Each function object type has its corresponding method, for example filters have the CreateFilter method. Each method takes a string argument with the description of the desired object, most of them also take the pointer to the SuperMatrix (except for local weights and similarity functions).

\subsubsection{Compare Functions}

Method: CreateComparator(string, SuperMatrixPtr)

Identifier: cmp

Arguments:

\begin{tabbing}

weight -- \= string -- \= string describing the weighing object for the matrix\\
selector -- \> string -- \> a string describing the selector for the matrix\\
relative\_selector -- string -- a string describing the relative selector for pairs of terms\\
simfunction -- string -- a string describing the similarity function for the matrix\\
filter -- string -- a string describing the filter for the matrix\\
\end{tabbing}
\subsubsection{Filters} 

Method: CreateFilter(string, SuperMatrixPtr)

%\paragraph{FeatureFilter01}
%Identifier: feature01

%Arguments: 

%k \tab\tab\tab-- size\_t -- 2 by default
\classDesc{FeatureFilter01}{feature01}
\argdescThree{k}{size\_t}{2}

\classDesc{FilterVector}{vector}
\argdescTwo{filter}{string}

\classDesc{NoiseKFilter}{noisek}
\argdescTwo{k}{size\_t}

\classDesc{NoiseKTFFilter}{noisektf}
\argdescTwo{k}{size\_t}
\argdescTwo{mintf}{size\_t}

\subsubsection{Weights} 

Method: CreateWeigher(string, SuperMatrixPtr)

\classDesc{CRMDt}{CRMDt}
none

\classDesc{CRmDmi}{CRmDmi}
none

\classDesc{LogEnt}{logent}
\argdescThree{useFilteredEntropy}{bool}{false}

\classDesc{LogEngNormalize}{logent\_normalize}
\argdescThree{useFilteredEntropy}{bool}{false}

\classDesc{MIWeight}{MI}
\argdescThree{miFilter}{double}{0}

\classDesc{MIWeightQuantified}{MI\_quant}
\argdescThree{miFilter}{double}{0}
\argdescThree{trail}{size\_t}{2}
\argdescThree{zeros}{bool}{false}

\classDesc{MLEWeightFeature}{MLE}
none

\classDesc{NormalizeWeight}{normalize}
none

\classDesc{RFFWeight}{rff}
\argdescTwo{simmatrix}{string}
\argdescTwo{top\_k}{size\_t}
\argdescTwo{sim\_threshold}{double}
\argdescTwo{selector}{string}

\classDesc{RWFChi2}{rwf\_chi2}
\argdescTwo{rankbuilder}{string}
\argdescThree{threshold}{double}{0}

\classDesc{RWFEvertsMI}{rwf\_evertsmi}
\argdescTwo{rankbuilder}{string}
\argdescThree{mi\_threshold}{double}{0}

\classDesc{RWFLogLikelihood}{rwf\_loglikelihood}
\argdescTwo{rankbuilder}{string}
\argdescThree{threshold}{double}{0}

\classDesc{RWFNoWeight}{rwf\_noweight}
\argdescTwo{rankbuilder}{string}

\classDesc{RWFPMIWeight}{rwf\_pmi}
\argdescTwo{rankbuilder}{string}
\argdescThree{discount}{bool}{true}
\argdescThree{threshold}{double}{0}

\classDesc{RWFZScore}{rwf\_zscore}
\argdescTwo{rankbuilder}{string}
\argdescThree{threshold}{double}{0}

\classDesc{RWFZScoreInMatrixWeight}{rwf\_zscore\_in\_matrix}
\argdescTwo{rankbuilder}{string}
\argdescThree{threshold}{double}{0}

\classDesc{RWFLocalZScoreWeight}{rwf\_zscore\_local}
\argdescTwo{rankbuilder}{string}
\argdescThree{threshold}{double}{0}

\classDesc{MISimpleWeight}{MI\_simple}
\argdescThree{discount}{bool}{true}
\argdescThree{threshold}{double}{0}

\classDesc{TScoreWeight}{TScore}
none

\classDesc{Tfidf}{Tfidf}
none

\classDesc{TfidfNormalizeWeight}{Tfidf\_normalize}
none

\classDesc{RowScaledWeight}{row\_scaled}
\argdescTwo{weight\_inner}{string}

\classDesc{ColumnScaledWeight}{column\_scaled}
\argdescTwo{weight\_inner}{string}

\classDesc{BoundaryScaledWeight}{boundary\_scaled}
\argdescTwo{weight\_inner}{string}
\argdescThree{AdjP}{double}{0.8}
\argdescThree{AmISubj}{double}{0.6}
\argdescThree{Coord}{double}{0.8}
\argdescThree{NGen}{double}{0.6}

\classDesc{RWFAnyWeight}{rwf}
\argdescTwo{rankbuilder}{string}
\argdescTwo{weight\_inner}{string}

\classDesc{QuantizationWeight}{quant}
\argdescTwo{weight\_inner}{string}
\argdescThree{trailing\_digits}{size\_t}{2}
\argdescThree{zeros}{bool}{false}

\classDesc{PMINoLog}{pmi\_nolog}
\argdescThree{discount}{bool}{true}
\argdescThree{mi\_threshold}{double}{0}

\classDesc{Chi2}{chi2}
\argdescThree{threshold}{double}{0}

\classDesc{EvertsMI}{evertsmi}
\argdescThree{threshold}{double}{0}


\subsubsection{Similarity Functions} 

Method: CreateSimfunction(string, SuperMatrixPtr)

\classDesc{CosineFilterSimilarity}{cosinefilter}
\argdescThree{percentIter}{double}{0.1}
\argdescThree{minIter}{size\_t}{2}

\classDesc{CosineFunction}{cosine}
\argdescThree{matrixNormalized}{bool}{false}

\classDesc{Dice}{dice}
none

\classDesc{Jaccard}{jaccard}
none

\classDesc{LinSimilarity}{lin}
none

\classDesc{MCFSim}{mcf}
\argdescThree{threshold}{double}{0.14}
\argdescTwo{weight}{string}
\argdescTwo{sim\_inner}{string}

\classDesc{PEntP}{pentp}
none

\classDesc{SHD}{shd}
\argdescThree{alpha}{double}{0.5}

\classDesc{SimCRM}{crm}
\argdescThree{gamma}{double}{0.25}
\argdescThree{beta}{double}{0.2}

\classDesc{filesimfunction}{filesim}
\argdescTwo{sim\_matrix}{string}

\classDesc{DistanceSim}{distancesim}
\argdescTwo{dist}{string}

\classDesc{RatioSim}{ratio}
none

\subsubsection{Selectors} 

Method: CreateSelector(string, SuperMatrixPtr)

\classDesc{FrequencySelector}{frequency}
\argdescThree{min\_nz}{size\_t}{5}
\argdescThree{min\_tf}{size\_t}{5}
\argdescThree{min\_TF}{size\_t}{5}

\classDesc{TScoreSelector}{TScore}
\argdescThree{threshold}{double}{2.576}

\classDesc{TScoreKBestSelector}{TScoreKBest}
\argdescThree{threshold}{double}{2.576}
\argdescThree{k}{size\_t}{350}

\classDesc{RFFSelector}{RFF}
\argdescTwo{TFThreshold}{double}
\argdescTwo{weightThreshold}{double}

\subsubsection{Row Relative Selectors}

Method: CreateRowRelaviteSelector(string, SuperMatrixPtr)

\classDesc{RelativeTScoreKBestSelector}{TScoreKBest}
\argdescTwo{tscore\_threshold}{double}

\subsubsection{Local Similarity Functions}

Method: CreateLocalSimfunction(string)

\classDesc{LocalCosineSimilarity}{local\_cosine}
\argdescTwo{isNormalized}{bool}
\argdescTwo{doFilter}{bool}
\argdescThree{percentInter}{double}{0.1}
\argdescThree{minInter}{size\_t}{2}

\classDesc{IRad}{irad}
\argdescThree{beta}{double}{20}
\argdescTwo{weight}{string}

\subsubsection{Rank builders}

Method: CreateRankBuilder(string, SuperMatrixPtr)

\classDesc{NormalRankBuilder}{normal}
\argdescThree{uniform\_rank}{bool}{false}
\argdescThree{rank\_size}{size\_t}{350}

\classDesc{RoundingBasedPartialOrder}{rounding\_partial}
\argdescThree{uniform\_rank}{bool}{false}
\argdescThree{rank\_size}{size\_t}{350}
\argdescThree{sport\_ranking}{bool}{false}
\argdescThree{idf}{bool}{false}
\argdescThree{idf\_strength}{double}{1}
\argdescThree{divide\_factor}{double}{10}

\subsubsection{Local weighers}

Method: CreateLocalWeigher(string)

\classDesc{LocalLogEnt}{logent}
\argdescThree{useFilteredEntropy}{bool}{false}

\subsubsection{RFF Selectors}

Method: CreateRFFSelector(string, SuperMatrixPtr)

\classDesc{RFFSelector}{RFF}
\argdescTwo{TFThreshold}{double}
\argdescTwo{weightThreshol}{double}

\subsubsection{Distance Functions}

Method: CreateDistfunction(string, SuperMatrixPtr)

\classDesc{EuclidianDist}{euclid}
None

\classDesc{MinkowskiDist}{minkowski}
\argdescThree{p}{double}{2}

\subsection{Implementation details}

FunctionFactory2 provides public methods presented in the previous section to create function objects. The most important protected methods and components of the class are:

\begin{itemize}
	\item a map of legal arguments for classes

	\item a map of function pointers to parsing methods which return specific objects

	\item methods initializing a parser for a specific class (i.e. a specific filter)

	\item methods initializing parsers for an abstract class (i.e. all supported filters)

	\item methods parsing the description of a specific class (i.e. a specific filter)

\end{itemize}

\subsubsection{Legal arguments for classes}
\label{ss:legals}
This is a string $\mapsto$ vector\textless{}string\textgreater{} map. Filters, similarity functions etc. have seperate maps, i.e. legal\_filter\_args for filters. These maps are initialized by methods described in \ref{ss:initSpecific}. Example map entries:

\begin{itemize}
	\item "cosine" $\mapsto$ ["matrixNormalized"]

	\item "cosinefilter" $\mapsto$ ["percentInter","minInter"]

	\item "dice" $\mapsto$ []

\end{itemize}

\subsubsection{Function pointrs returning specific objects}
\label{ss:pointers}
This is a string $\mapsto$ boost::function map. The function arguments and returned value depend on the abstract class. It assigns each identifier a pointer to a method which will parse the arguments and return the desired function object. Filters, similarity functions etc. have seperate maps, i.e. legal\_filter\_functions for filters. These maps are initialized by methods described in \ref{ss:initSpecific}. Example map entries:

\begin{itemize}
	\item "cosine" $\mapsto$ \&smartcomparator::compare::FunctionFactory2::CreateSimfunctionCosine

	\item "cosinefilter" $\mapsto$ \&smartcomparator::compare::FunctionFactory2::CreateSimfunctionCosineFilter

	\item "dice" $\mapsto$ \&smartcomparator::compare::FunctionFactory2::CreateSimfunctionDice

\end{itemize}

\subsubsection{Methods initializing a parser for a specific class}
\label{ss:initSpecific}
These methods inputs data on a single class into maps described in \ref{ss:legals} and \ref{ss:pointers}.

\subsubsection{Methods initializing parsers for an abstract class}
\label{ss:initGeneral}
These methods initialize maps described in \ref{ss:legals} and \ref{ss:pointers}. They contain calls to \ref{ss:initSpecific} methods and are called by the FunctionFactory2 constructor.

\subsubsection{Methods parsing the description of a specific class}
\label{ss:parse}
The maps described in \ref{ss:pointers} point at these methods. They parse single arguments, pass them on to the proper constructor and return the desired object. They contain information on the arguments being obligatory or optional (and initialize optional arguments with a default value).

\subsection{Extending FunctionFactory2}
\label{ss:extendingFF}
In order to add support for a new function class to FunctionFactory2 one must do the following:

\begin{itemize}
	\item Create a method to parse the class (see \ref{ss:parse}).

	\item Create a method to initialize the class parser (see \ref{ss:initSpecific}).

	\item Add a call to this initialization method to \ref{ss:initGeneral}.
\end{itemize}

For example:

Let's say we want to add a Foobar filter to FF2. The Foobar filter takes two arguments:

\begin{itemize}
	\item foo (bool) - optional, false by default

	\item bar (double) - mandatory
\end{itemize}

\subsubsection{Parsing method}

\begin{lstlisting}
filters::FilterFeaturePtr FunctionFactory2::CreateFilterFoobar(std::map<std::string, std::string> & args,
								   Matrices::SuperMatrixPtr matrix)
{
		bool foo = false; //init optional parameter with the dafault value
		double bar; //init the mandatory parameter
		if (args.find("foo") != args.end()) //check for the argument's occurence in the arguments map (created by the calling CreateFilter method)
			GetBool2(args, "mintf", mintf); //assign a value to the bool parameter
		if (args.find("bar") != args.end())
			GetNumber2(args, "bar", bar); //assign a value to the double parameter
		else
			throw std::runtime_error("Foobar filter missing required argument: bar"); //an exception in case of a missing mandatory parameter
		return filters::FilterFeaturePtr(new filters::FoobarFilter(matrix->GetColumnFeatures(), foo, bar)); //call the constructor and return the pointer
}
\end{lstlisting}

\subsubsection{Class parser initialization}

\begin{lstlisting}
void FunctionFactory2::initFilterFoobar()
{
    std::vector<std::string> args; //initialize a legal arguments vector
    args.push_back("foo");
    args.push_back("bar"); //populate the vector
    legal_filter_args["foobar"] = args; //assign the vector to the identifier
    filter_functions["foobar"] = &smartcomparator::compare::FunctionFactory2::CreateFilterFoobar; //assign the parsing method to the identifier
}
\end{lstlisting}

\subsection{Calling the class initializer}
\begin{lstlisting}
void FunctionFactory2::initFilters() //this method already exists
{
	initFilterFeature01();
	initFilterVector();
	initNoiseKFilter();
	initNoiseKTFFilter();
	//the above methods initialize other filters
	initFilterFoobar(); //the call to be added (order is irrelevant)
}
\end{lstlisting}
\section{Adding new objects}
\subsection{Creating Weights}
\subsection*{Structure}
To create a new weight must be added to the package "comparator/weights" header file and source file of the new weight. New weight class must inherit from class WeightFeature.  

Skeleton of new weight header file:
\begin{lstlisting}

#include "../comparator_stdheaders.h"
#include "WeightFeature.h"

namespace smartcomparator
{
        namespace weights
        {
                class SMART_COMPARATOR_API FooBarWeight : public WeightFeature
                {
                public:
                        FooBarWeight(Matrices::SuperMatrixPtr pMatrix, int foo, double bar=0);
                        virtual ~FooBarWeight();
                        virtual void Weight(size_t rowindex, Matrices::CompressedVector *v, std::set<size_t> &filteredIndices);

                };

        }// namespace weights
}// namespace smartcomparator
\end{lstlisting}

After creating a new weight files, you must add information about them in files "comparator\_all.h" and "CMakeLists.txt" (package "comparator"). Now you must add weight to FunctionFactory2, just as in the example in \ref{ss:extendingFF}. 

Method Weight takes as arguments number of row which is weighted (rowindex), pointer to row(CompressedVector *v) and set of previously filtered indices (filteredIndices). 

\subsection*{Using weights}
To use weight we can use FunctionFactory by creating CompareFunctionPtr. In examples in this section m means SuperMatrixPtr m.

\begin{lstlisting}
smartcomparator::compare::CompareFunctionPtr comp = smartcomparator::compare::FunctionFactory::CreateFunction("cmp(simfunction=cosine(), weight=foobar(foo=3, bar=0.15)",m);
comp->TransformMatrix();
\end{lstlisting}

Similarity function is necessary to create Comparator, but doesn't participate in computing weights.

Second way is to create WeightFeature Object
\begin{lstlisting}
smartcomparator::weights::WeightFeaturePtr wptr(new smartcomparator::weights::FooBarWeight(m, 3, 0.15));
\end{lstlisting}

