ML base Spacing Correcter

This model is improved version of TrainKoSpacing, using FastText instead of Word2Vec

Performances

Model	Test Accuracy(%)	Encoding Time Cost
TrainKoSpacing	96.6147	02m 23s
자모분해 FastText	98.9915	08h 20m 11s
2 Stage FastText	99.0888	03m 23s

Data

Corpus

We mainly focus on the National Institute of Korean Language 모두의 말뭉치 corpus and National Information Society Agency AI-Hub data. However, due to the license issue, we are restricted to distribute this dataset. You should be able to get them throw the link below National Institute of Korean Language 모두의 말뭉치. National Information Society Agency AI-Hub

Data format

Bziped file consisting of one sentence per line.

~/KoSpacing/data$ bzcat train.txt.bz2 | head
엠마누엘 웅가로 / 의상서 실내 장식품으로… 디자인 세계 넓혀
프랑스의 세계적인 의상 디자이너 엠마누엘 웅가로가 실내 장식용 직물 디자이너로 나섰다.
웅가로는 침실과 식당, 욕실에서 사용하는 갖가지 직물제품을 디자인해 최근 파리의 갤러리 라파예트백화점에서 '색의 컬렉션'이라는 이름으로 전시회를 열었다.

Architecture

Model

Word Embedding

자모분해

To get similar shpae of Korean charector, use 자모분해 FastText word embedding. ex) 자연어처리 ㅈ ㅏ – ㅇ ㅕ ㄴ ㅇ ㅓ – ㅊ ㅓ – ㄹ ㅣ –

2 stage FastText

Becasue of time to handdle 자모분해, use 자모분해 FastText only for Out of Vocabulary charector.

Thresholding

Because middle part of output distribution are evenly distributed.

Use log transform and second derivative result:

How to Run

Installation

• For training, a GPU is strongly recommended for speed. CPU is supported but training could be extremely slow.

• Support only above Python 3.7.

  Requirement

• Python (>= 3.7)

• MXNet (>= 1.6.0)

• tqdm (>= 4.19.5)

• Pandas (>= 0.22.0)

• Gensim (>= 3.8.1)

• GluonNLP (>= 0.9.1)

• soynlp (>= 0.0.493)

Dependencies

pip install -r requirements.txt

Training

python train.py --train --train-samp-ratio 1.0 --num-epoch 50 --train_data data/train.txt.bz2 --test_data data/test.txt.bz2 --outputs train_log_to --model_type kospacing --model-file fasttext

Evaluation

python train.py --model-params model/kospacing.params --model_type kospacing
sent > 중국은2018년평창동계올림픽의반환점에이르기까지아직노골드행진이다.
중국은2018년평창동계올림픽의반환점에이르기까지아직노골드행진이다.
spaced sent[0.12sec/sent]  > 중국은 2018년 평창동계올림픽의 반환점에 이르기까지 아직 노골드 행진이다.  

Directory

Directory guide for embedding model files bold texts means necessary

• model

  • fasttext
  • fasttext_vis
  • fasttext.trainables.vectors_ngrams_lockf.npy
  • fasttext.wv.vectors_ngrams.npy
  • kospacing_wv.np
  • w2idx.dic

• jamo_model

  • fasttext
  • fasttext_vis
  • fasttext.trainables.vectors_ngrams_lockf.npy
  • fasttext.wv.vectors_ngrams.npy
  • kospacing_wv.np
  • w2idx.dic

Reference

TrainKoSpacing: https://github.com/haven-jeon/TrainKoSpacing 딥 러닝을 이용한 자연어 처리 입문: https://wikidocs.net/book/2155