TEXT DATA CORPORA GENERATION ON THE BASIS OF THE DETERMINISTIC METHOD

Background. The solution to many problems in the field of natural language processing involves the use of corpora of text data, which makes the issue of preparing such corpora topical. At the same time, the formation of corpora based on natural texts is time-consuming and not always expedient. Therefore, an automated generation of corpora based on various methods and algorithms is gaining popularity, which greatly simplifies the preparation of experimental data. Objective. The purpose of the paper is to increase the number of detected defects during testing of software implementations of methods for processing natural text data by developing a new method for generating text data corpora to be used as input for testing. Methods. The deterministic method of text data corpora generation, named CorDeGen, is proposed, which satisfies the following requirements: determinism, dependence on input only from the desired number of terms in the generated corpus, as well as the non-trivial structure of the generated corpus. Based on the proposed method, the algorithm has been developed that implements it, as well as a software implementation on the .NET platform (programming language – C#). The evaluation of the speed and efficiency of the developed method has been done based on the developed software. Results. The performed speed evaluation of the developed CorDeGen method showed a power-law dependence of the time of generating the corpus on the number of terms (input parameter), with a degree of about 1.5. In this study, the feasibility of using the developed method to test the correctness of software implementations is shown by the example of testing the k-means clustering method. Conclusions. Testing of the developed deterministic method of text data corpora generation has shown the effectiveness of using this method in the testing of other natural language processing tasks, such as clustering, instead of natural corpora.