TEXT DOCUMENT CLASSIFICATION SYSTEM WITH BIG DATA TECHNOLOGIES USAGE

Abstract

The aim. The paper considered a model of the document classification system using Big Data technology. When using Big Data technology, a large array of documents accumulates on the server which must be pre-processed and uploaded to the database. In the documents you need to define keywords with a help of which you need to assign them to one or more thematic sections. In addition, the developed system should operate fast and provide automatic learning. Therefore, the development of models and methods of classification of text documents for real time is an urgent task. A very intensive development of these methods has been observed recently with the rapid development of computer technology and with the transition of many organizations into electronic document management. As a result of the study, a method and a system model were developed; a combination of approaches for model training is proposed; the most productive model for system training is determined. Scientific novelty. The paper proposes a new solution for performing accurate Bayesian classification based on Spark. This classifier uses a large number of in-memory server operations to classify a large number of text documents based on a large training dataset using MapReduce. The map phase calculates the number of occurrences of keywords in different distributions of the training data. After that, several reducers calculate the probability of assigning of a document to certain classes based on the calculations obtained at the map stage. The key point of this proposal is to manage a set of text documents keeping them in memory whenever possible. Conclusions. The results of this work could be used for implementation of an effective classification system for text documents that uses an accurate Bayesian classifier developed with the Python programming language in combination with the Hadoop Big Data service.