\chapter{Tutorial}

\section{Working with Python bindings}
\label{sec:pythontutorial}
\lstset{language=Python}

The easiest way to work with SuperMatrix is through the supplied bindings for Python. Unfortunately, not
all functionality is available using only supplied bindings. For example, one cannot create co-occurrence
matrices from text yet. Nevertheless, most important functionalities (such as loading an existing
matrix, creation of Measures of Semantic Relatedness or comparing matrix rows) can be achieved this way.

In this tutorial we assume basic knowledge of Python and access to some pre-built co-occurrence matrix. We
present the tutorial in standard, "pythonic" way using the \verb|>>>| prompt. Nonetheless, we encourage
you to use shell with more features (e.g., IPython). We assume that the matrix is stored in './matrices'
directory. 

We will first present a few lines of code and then explain their meaning. We will describe most important
concepts used by SuperMatrix as we go.

\begin{lstlisting}
>>> import pymatrices, pycomparator
\end{lstlisting}

The first thing to do is to load the two libraries: \verb|pymatrices| and \verb|pycomparator|. The former is a
wrapper library on the \verb|matrices| C++ namespace. It contains most important data structures that holds
the co-occurrence data. The latter is a wrapper library on the
\verb|smartcomparator|\footnote{ \emph{comparator} directory in the source code}. It contains components
needed to construct Measure of Semantic Relatedness (hereinafter called MSR).

\begin{lstlisting}
>>> mdir = './matrices'
>>> mname = 'bnc_subj_summed'
>>> matrix = pymatrices.LoadSuperMatrix(mdir, mname, False, True)
\end{lstlisting}

After issuing the last command there should be a loading progress report. The loading time depends on the
available hardware. 

The first two lines define two variables: mdir points to a directory containing the matrices while mname holds
the name of the matrix, which is loaded in the third line. The \verb|LoadSuperMatrix| function takes four
parameters: directory with matrices, matrix name and two boolean values (flags). The first flag which is set
to \verb|False| in the above example tells SuperMatrix that column-based matrix will not be used. The
second flag says that the row-based matrix will be used. Both matrices are usually kept for efficiency reasons.
On the other hand, only a few MSRs needs access to column-based matrix. Thus, it is usually safe to load only
row-based one only.

SuperMatrix stores sparse matrix together with some precomputed data (see Sec. \ref{sec:fileformats}). The
matrix usually describes words (rows) occurring with other words in some lexico-syntactic relation (columns).
The co-occurrence data is gathered by processing corpora. The matrix that is being used in this tutorial
was constructed from British National Corpus (BNC)\footnote{\url{http://www.natcorp.ox.ac.uk/}}
with the help of MiniPar dependency parser \citep{Lin:93}. The rows correspond to nouns that occur more then
10 times in the BNC, the columns denote other words that can be in any relation with nouns according to
MiniPar (e.g., \emph{appo}, \emph{mod}, \emph{subj-of}). The values in matrix cells correspond to the number
of times a given relation was observed between the two words (row and column).

\begin{lstlisting}
>>>  matrix.NumberOfRows()
40643
\end{lstlisting}

Matrix class has a few methods that can be used for, e.g., inspection of the data. The above method is used to
check the size of a matrix. After writing \verb|matrix.| using the \verb|<tab>| key will show all the methods
available.

\begin{lstlisting}
>>> matrix.GetRowIndex('research')
30589
>>> row_features = matrix.GetRowFeatures()
>>> research_features = row_features[30589]
>>> research_features.GetTermFrequency()
12893
>>> research_features.GetGlobalTermFrequency()
21547
>>> research_features.GetEntropy()
6.5469299999999997
\end{lstlisting}

In the above example we are checking several \emph{features} of the row describing the word \emph{research}.
Currently, we store only a few features (frequencies, label and entropy). The \verb|GetRowIndex| method will
return maximum integer if given word is not present in the matrix (one can check if the returned index is
less then number of rows). The difference between \verb|GetTermFrequency| and \verb|GetGlobalTermFrequency| is
that the first number denotes a number of times a given word was present in any lexico-syntactic relation in the
corpus and the latter denotes a number of times a given word was present in a corpus.

Let's create some MSRs:

\begin{lstlisting}
tfidf = pycomparator.FunctionFactory.CreateFunction('cmp(simfunction=cosine(),weight=tfidf())', matrix)
lin = pycomparator.FunctionFactory.CreateFunction('lin()', matrix)
rwf = pycomparator.FunctionFactory.CreateFunction('rwf_lin(rank_size=1000,uniform_rank=false)', matrix)
\end{lstlisting}

We first construct classical MSR based on \emph{term frequency inversed document frequency} model with the cosine
function \citep{ManRagSch:08}. Then we construct the original Lin's measure \citep{Lin:98} in the second
line. In the third line we construct a Rank Weight Function based on Lin's measure
\citep{DBLP:conf/ai/BrodaPS09}.

The most important thing in construction of an MSR is the string that describes the MSR. For the construction
of \emph{tfidf} we use the recently introduced notation used by \verb|FunctionFactory2| (see Sec.
\ref{sec:ff2}). For the other two the old \verb|FunctionFactory| is used. The latter has the advantage of
compactness, but the disadvantages in flexibility.\footnote{Also, the parsing is performed in much better way
in FunctionFactory2.}

\begin{lstlisting}
>>> tfidf.Compare('house', 'research')
0.045528858937868059
>>> tfidf.Compare('investigation', 'research')
0.33784927672295895
>>> tfidf.Compare('investigation', 'not in the matrix')
-1.0
\end{lstlisting}

The above outlines a basic comparison routines of MSR (tfidf is an MSR). If we try to compare rows that are
not present in the matrix the result is usually -1. The MSRs above use a similarity function for row
comparison, so the bigger the value returned the more similar the words are. Most of the similarity functions
present in SuperMatrix are constrained to \verb|<0,1>|, so values below 0 are used to denote special
conditions. 

\begin{lstlisting}
>>> matrix.GetRowIndex('house')
17257
>>> matrix.GetRowIndex('research')
30589
>>> tfidf.Compare(17257, 30589)
0.045528858937868059
\end{lstlisting}

We can also use row indices in place of words in the above examples. The last method call in the following
example is equivalent to the first one in the previous example. Usage of indices is useful when using the
Python bindings from scripts.

\begin{lstlisting}
>>> pycomparator.print_kbest(tfidf, 'research', 20)
urban (0.474), study (0.465), survey (0.445), nation-building (0.430), rossall (0.402), experiment (0.389), researcher (0.367), datum (0.357), investigation (0.338), discovery (0.335), evidence (0.334), segmentation (0.328), work (0.323), counterparty (0.323), capitalisation (0.313), tympanometry (0.309), institut (0.309), hawtrey (0.309), frizz (0.309), imperfection (0.293),

>>> pycomparator.print_kbest(rwf, 'research', 20)
study (0.175), investigation (0.155), survey (0.138), analysis (0.130), evaluation (0.124), project (0.121), researcher (0.110), science (0.109), training (0.109), discussion (0.108), experiment (0.107), exploration (0.106), datum (0.106), observation (0.105), evidence (0.104), planning (0.104), teaching (0.102), enquiry (0.099), assessment (0.099), interview (0.096),
\end{lstlisting}

The code snipped above shows how can we find 20 words most similar to another word. The \verb|print_kbest| function
requires 3 arguments: an MSR, a target word and number of most similar words. The computation can take a
while as calling this function is equivalent to calling pair-wise comparison for every matrix row with a
target word.

The above example also shows the difference in quality of returned similarity lists by different MSRs. RWF
is clearly better in the above example but the computation takes more time. The difference between RWF and
Lin's measure is not significant for high frequency words.

\section{Using C++}
\label{sec:cpptutorial}
\lstset{language=C++}

\subsection{Loading Existing Matrices}

There are many different ways to load existing matrices. The most convenient way is to use LoadSuperMatrix function from
Matrices::IO namespace. It can be done using code similar to the one presented below:

\begin{lstlisting}
// place with matrices
std::string mdir = "/some/path/supermatrix2/noun/";
// matrix name (dirname)
std::string mname = "kipi_substAdjP_pq";
// loading of matrix
Matrices::SuperMatrixPtr pMatrix = Matrices::IO::LoadSuperMatrix(mdir, mname, false, true);
\end{lstlisting}

LoadSuperMatrix accepts four arguments: directory with matrices (parent directory of matrix), matrix
name\footnote{It will be simplified in future version to just one parameter as we have come to format that
assumes one matrix per directory and it is unnecessary burden to split the path every time matrix should be
loaded.} and two boolean arguments specifying which part of the matrix should be loaded. In the above example
we want to load only row-based matrix, so first boolean argument is set to false (do not load column-based
matrix) and second is set to true. Column-based matrices are rarely needed in typical workflows involving
measuring of semantic relatedness, as most operation in those are row-based.

\subsection{Viewing Contents of Matrices}
After loading SuperMatrix we can check some matrix features. 

\begin{lstlisting}
	std::cout << "Number of rows: " << pMatrix->NumberOfRows() << std::endl;
	std::cout << "Number of columns: = " << pMatrix->NumberOfColumns() << std::endl;
\end{lstlisting}

The above methods are used to check the size of a matrix. 

\begin{lstlisting}
	int research_index =  pMatrix->GetRowIndex("research");
	Matrices::FeatureVectorPtr row_features = pMatrix->GetRowFeatures();
	Matrices::Feature research_feature = row_features->at(research_index);
	std::cout<< research_feature.GetEntropy();
	std::cout<< research_feature.GetTermFrequency()
	std::cout<< research_feature.GetGlobalTermFrequency()
\end{lstlisting}

In the above example we are checking several features of the row describing the word research.

\subsubsection{vector\_extractor}
Program vector\_extractor, for each given word, extract corresponding vector of values from matrix. 

\subsubsection{Parameters description}

\begin{tabular}{l l}
--matrix arg & Matrix path \\
--out-name arg & Name of output file \\
--words arg & Name of file with words to extract \\
--mode arg (=row) & Extraction mode. One of: row or column \\
\end{tabular}


\subsubsection{Usage}
To use builder you need matrix, output file and file with words to extract.
\begin{verbatim}
$ vector_extractor --matrix /home/user/matrices/matrix 
--out-name out.csv --words /home/user/words --mode row
\end{verbatim} 


\subsection{Using Measures of Semantic Relatedness}

There are two main ways of working with pre-build Measures of Semantic Relatedness (MSRs) --- using a factory
of MSR objects or assembling MSR from parts. Second way is more powerful and may allow for finer control on
some aspects of MSRs, but in this short overview we will focus on first one. Assuming that matrix is loaded
into variable $pMatrix$ following code shows how to create simple MSR and compare a few lemmas:

\begin{lstlisting}
	std::string msr = "rwf_lin(rank_size=1000,uniform_rank=false)";
	// Creation of compare function
	smartcomparator::compare::CompareFunctionPtr
			pCmp = smartcomparator::compare::FunctionFactory::CreateFunction(msr,  pMatrix);

	std::cout << "Min value: " << pCmp->MinValue() << std::endl;
	std::cout << "cmp(pies, kot) = " << pCmp->Compare("subst:pies", "subst:kot") << std::endl;
	std::cout << "cmp(pies, drabina) = " << pCmp->Compare("subst:pies", "subst:drabina") << std::endl;
	std::cout << "cmp(pies, drabina) = " << pCmp->Compare("subst:cena netto", "subst:call girl") << std::endl;
	std::cout << "cmp(pies, drabina) = " << pCmp->Compare("bzdura 1", "bzdura 2") << std::endl;
\end{lstlisting}

The \verb=CreateFunction= function receives MSR-description string and matrix and returns ready-to-use MSR.
The language describing MSRs is not yet formalised\footnote{It will be soon with simple EBNF grammar.}. In the
code above we have created Rank Weight Function based on Lin's measure with ranking building from the
bottom\footnote{This means that the maximum value of rank is determined by the number of ranks in case where
there are less important feature then desired ranking size.}. 

To compute semantic relatedness between all pairs of words the \verb|parallelSimilarity| \ref{sec:parallelSimilarity} tool can be used.

\section{Important data structures}

\subsection{Tuples format}\label{ss:tuplesformat}
Tuples are one of the possible basic data structures that can be used as a basis for computing distributional similarity between
words. In the most fundamental way a file with tuples contains lines with:
\begin{lstlisting}
word_a;relation;word_b;frequency
\end{lstlisting}
where relation are taken from, e.g., dependency parser. There are two simple, but still interesting ways for
preparing similarity space: window-based and document-based. In window-based approach you create a tuples in the
following form:
\begin{lstlisting}
word_a;IN_WINDOW;word_b;frequency
\end{lstlisting}
where IN\_WINDOW is a custom name denoting that two words are close in text to each other. You can also keep the distance (counted in
words) in tuples, so the relation can be for example: IN\_WINDOW\_5 or IN\_WINDOW\_3. 

Another interesting and classical similarity space is to use a term by document matrix. Here a relation is defined as "occuring in a given
document", for example:
\begin{lstlisting}
word_a;IN_DOCUMNET;path/to/document;frequency
\end{lstlisting}

There are also some scripts for tuples manipulation, described in \ref{ss:architectTools}.

\subsection{Matrices}

\subsubsection{Buidling matrices}
Before build matrix, first you should obtain tuples. To generate tuples for Polish language you can use one of tools described in \ref{ss:architectTools}. For other languages you should have your own tools which generate tuples in appropriate format.
Tuples format is described in \ref{ss:tuplesformat}. Then you can use \verb|tuplesproc| \ref{sec:tuplesprocTool} to build matrix.

\subsubsection{LSA(how to build LSA matrix)}
\begin{itemize}
  \item generate tuples term $\times$ documents, they should have following form:
  \begin{lstlisting}
  term;IN_DOCUMNET;path/to/document;frequency
  \end{lstlisting}
  For Polish language you can use \verb|doctermbuilder| \ref{sec:doctermbuilderTool}.
  \item build matrix from tuples using \verb|tuplesproc| \ref{sec:tuplesprocTool}
  \item use \verb|weighter| \ref{ss:weighter} with \verb|SVD| transformation and \verb|logent| weight
  \item use \verb|parallelSimilarity| \ref{sec:parallelSimilarity} to generate similarity matrix
  \item use \verb|similarity_results_labeler| \ref{sec:simLabeler} to extract lists of k-best words from similarity matrix

\end{itemize}




\section{How to make K-Best file}

\subsection{Mass runners}
In the need of processing whole KGR10 there were introduced scripts in directory /tools/architect2/mass\_runners. They all are almost identical. The sole purpose of them is to run some tool from architect2 on corpus consisting of huge number of relatively small files.

As there were many files, all scripts can run in parallel and save progress leaving possibility of resurrecting computations in case of some interruption. Since even preparing of list of files can take very long, it is better to provide prepared one. We suggest to save the output of corpus decompression and make it into an index.

All take the same basic parameters, listed below. Apart from them, all parameters listed after ' -{}- ' will be passed to underlying programs. Examples will be shown throughout all further sections.

\subsubsection*{Parameters}
\begin{tabularx}{\textwidth}{l X}
-{}-in & Input directory. If index is used it still should be provided, for it is used to compute output paths.\\
-{}-out & Output directory. \\
-{}-index & Index of all files (not directories!) to be converted. In each line of index shall be one full path to file.\\
-{}-interval & After how many files the progress should be saved.\\
-{}-threads & How many threads should be run at one time.\\
-{}-prefix & Path and prefix of all backup files.\\
\end{tabularx}


\subsection{Merging MWEs}
This step is optional, but if you've got definitions of MWEs it would be a waste if you not use it. The following command should convert the whole corpus.

\begin{lstlisting}[language=bash]
$ python3 mass_runners/mwefinder.py --in CORPUS --out OUTPUT --interval INTERVAL --threads THR_NUM --index FILE_LIST -- --reader READER --mwefiles MWEFILES/* --tagset TAGSET > LOG_FILE 2> ERR_FILE &
$ disown %1    # lest the process die when ssh connection is lost
\end{lstlisting}

\subsubsection*{Where:}
\begin{tabularx}{\textwidth}{l l X}
\toprule
placeholder & example & description \\
\midrule
CORPUS & kgr10 & directory of corpus \\
OUTPUT & kgr10\_mwes & directory for new corpus with MWEs merged \\
INTERVAL & 30000 & after how many files should be prepared a backup \\
THR\_NUM & 30 & number of threads \\
FILE\_LIST & index.txt & list of files in CORPUS \\
READER & xces & format of CORPUS and OUTPUT \\
MWEFILES & Dane/MWEs & directory with definitions of MWEs \\
TAGSET & nkjp & tagset used by CORPUS and OUTPUT \\
LOG\_FILE & log & file where log should be stored \\
ERR\_FILE & err & file where all errors should be written \\
\bottomrule
\end{tabularx}


\subsection{Building supermatrix}
Firstly, we need to count words in the corpus. For this purpose Basedict was created. Run the following command on the corpus, to convert it into lesser number of files containing dictionaries. They all are called in the style of '231.dct'. Using \lstinline{disown} is highly recommended in all commands.

\begin{lstlisting}[language=bash]
$ python3 mass_runners/basedict.py --in CORPUS --out OUTPUT --interval INTERVAL --threads THREADS --index FILE_LIST -- --tagset TAGSET --reader READER > LOG_FILE 2> ERR_FILE &
\end{lstlisting}

Then we need to run WCCL operators and make tuples of Corpus. It is domain of Builder. It's mass\_runner works similarly to Basedict, leaving corpus of files called '151.tps'.

\begin{lstlisting}[language=bash]
$ python3 mass_runners/builder.py --in CORPUS --out OUTPUT --interval INTERVAL --threads THREADS --index FILE_LIST -- --tagset TAGSET --reader READER -w OPERATORS/* LOG_FILE 2> ERR_FILE &
\end{lstlisting}

\subsubsection*{Where:}
\begin{tabularx}{\textwidth}{l l X}
\toprule
placeholder & example & description \\
\midrule
OPERATORS & Dane/Operatory & directory with WCCL operators \\
\bottomrule
\end{tabularx}

We highly prefer to have one dictionary and one file with operators. So we should use script MergeFiles. It can merge both dictionaries and tuples. In each run it lessens the number of files, but there may be need to run it few times. And thus we run:

\begin{lstlisting}[language=bash]
$ python3 mass_runners/merge_files.py --in DIR_DCT --out OUTPUT_DCT --interval INTERVAL --threads THREADS > LOG_FILE 2> ERR_FILE &
$ python3 mass_runners/merge_files.py --in DIR_TPS --out OUTPUT_TPS --interval INTERVAL --threads THREADS > LOG_FILE 2> ERR_FILE &
$ ...
\end{lstlisting}

If there be too little RAM (which can happen in further trials on merging tuples) you can use the script tools/scripts/split\_files.py. It splits tuples into few files that can be merged independently, and then simply be put together.

Now we can build a matrix. Although you may also want to generalise PoSes of your rows. Then you should change them according to this table:

\begin{tabularx}{\textwidth}{X l}
\toprule
from & to \\
\midrule
burk, brev, subst, depr, ger, siebie, ppron12, ppron3 & subst \\
adja, adj, adjc, pact, ppas, num, numcol & adj \\
bedzie, fin , imps, impt, inf, praet, pred, winien & verb \\
adv, pcon, pant, qub & adv \\
adjp, aglt, comp, conj, interj, interp, prep, xxx & IGNORE \\
\bottomrule
\end{tabularx}

The tool to build a matrix is called Tuplesproc. Run:

\begin{lstlisting}[language=bash]
$ tuplesproc  --input TUPLES --out . --matrix-name MATRIX  --dict DICTIONARY --row-file ROWFILE
\end{lstlisting}

\subsubsection*{Where:}
\begin{tabularx}{\textwidth}{l l X}
\toprule
placeholder & example & description \\
\midrule
TUPLES & tuples.tps & file with tuples \\
DICTIONARY & dictionary.dct & original dictionary, containing frequencies for each \textbf{column} \\
ROWFILE & rowfile & file containing names of all rows that should be used. If you generalised PoSes it would contain lines such as ,,verb:spać" \\
MATRIX & GrandMatrix & name of output matrix \\
\bottomrule
\end{tabularx}


\subsection{K-Best file}
Now as you have your dreamed-of supermatrix, you are able to compute similarity. Such computations can take very long time (it's $\omega(n^3)$) for so big matrices, and therefore it would be best to filter it out of unnecessary content. It can be achieved by following command (which deletes all columns that have nothing of interest inside):

\begin{lstlisting}[language=bash]
$ filtrator2 --matrix-name GrandMatrix --matrix-dir . --out-name NotSoGrandMatrix --out-dir . --mode column --min-tf 50 --min-TF 50 --min-nz 5
\end{lstlisting}

Then you should weight it. Albeit it is possible to do it directly in computation, it's better to speed it up by running this command first (weight recommended at this moment):

\begin{lstlisting}[language=bash]
$ weighter NotSoGrandMatrix 'cmp(weight=MI(miFilter=0),filter=feature01())' ./WeightedNotSoGrandMatrix   # this dot is important!
\end{lstlisting}

Now is the big time: at least you are about to run parallelSimilarity!

\begin{lstlisting}[language=bash]
$ mpirun -n THREADS parallel/similarity/parallelSimilarity --matrix-dir . --matrix-name WeightedNotSoGrandMatrix --results-file raw_results --backup-interval INTERVAL --measure-of-relatednes 'cmp(simfunction=cosine())' --k-best-results KBEST --target-words-file ROWFILE &
\end{lstlisting}

\subsubsection*{Where:}
\begin{tabularx}{\textwidth}{l l X}
\toprule
placeholder & example & description \\
\midrule
THREADS & 10 & at least 2, otherwise it won't work \\
KBEST & 101 & number of desired k-best + 1 \\
ROWFILE & mwes\_only.row & file with rows for which similes we will seek \\
\bottomrule
\end{tabularx}

Now we have bunch of raw data, which we must convert into valid k-best file. It is relatively simple:

\begin{lstlisting}[language=bash]
$ tools/similarity_results_labeler/labeler --matrix-filename raw_results --matrix-type supermatrix --label-file WeightedNotSoGrandMatrix/WeightedNotSoGrandMatrix.rlabel --results-file kbest --k-best-results 100
\end{lstlisting}















