7: Performance Evaluation

The classifiers namely Latent Dirichlet Allocation (LDA), Term Frequency-Inverse Document Frequency (TF-IDF), Doc2Vec and Topic modelling and their performances are evaluated to obtain the accuracy. The result of pointwise mutual information (PMI) is used to get the Y label set of data, and multiclass is used in these data.

TF-IDF is a statistical representation that estimates how similar a word is to a document in a set of documents. This is carried out by multiplying two metrics namely the words IDF across a document set and how many times a word exists in a document. TF-IDF was discovered for information retrieval and search of document. It performs by developing proportion to the number of times a word exists in a document but is offset by the document number that comprises the word. So words that are relevant in each document such as what, if and this scores reduced even though they may exist several times since they do not mean much to that document. There are many ways of estimating the term frequency with the easiest being a raw number of examples a word exists in a document. Then, there are ways to adjust the frequency by document length or by raw frequency of document word. The IDF denotes how relevant or rare a word is in the whole segment of document. The nearer it is to zero, the more relevant a word is. This metric can be estimated by considering the total document number dividing it by the document number that comprises a word and estimating the logarithm.