김서영 / data

Go to a project
Toggle navigation Toggle navigation pinning
김서영
0b5e75c9 1 parent f141c599

readme file

Inline Side-by-side
Showing 7 changed files with 129 additions and 896 deletions
......@@ -27,26 +27,134 @@
- 장르 시각화 : matplot, Seaborn 패키지를 통한 시각화
## Model
- 데이터 수집 : [CMU Movie Summary Corpus][CMU]의 Detaset 42,306개, 뮤지컬 줄거리 255개
- 데이터 수집 : [CMU Movie Summary Corpus][CMU]의 Detaset 42,306개, 뮤지컬 줄거리 307개
- 데이터 전처리 : Detaset 토큰화, 불용어 처리 후 정수 인코딩
- 데이터 분석 : 의사결정나무 모델, RNN, k-nearest neighbor < 연구중
- 데이터 분석 : lstm
- 데이터 검증 : k-fold 교차검증을 활용하여 장르별 정확도를 확인
- 데이터 시각화 : 분석된 장르를 장르 단어 분포도로 시각화한다.
## Schedule
__3월__
- 주제 선정
__4월__
- 뮤지컬 줄거리 웹 크롤링
- 테스트 데이터 라벨링
__5월__
- 데이터 전처리
+ 토큰화 (완료)
+ 라벨링 (진행중)
- 데이터 시각화 : 분석된 뮤지컬 데이터를 장르 단위로 시각화한다.
## 과제 수행
### 데이터 수집
[ 장르 분류 모델 학습 데이터 ]
모델 학습을 위한 데이터는 뮤지컬 줄거리로는 양이 충분하지 못해 뮤지컬과 줄거리가 비슷한 영화 데이 터를 활용했습니다. http://www.cs.cmu.edu/~ark/personas/에서 영화 줄거리(영어) 데이터를 42,306개 수집했 습니다.
* 수집한 영화 줄거리 태깅 일부 : 각 영화 줄거리에는 데이터 제공 사이트에서 미리 장르를 다양하게 태깅 된 상태였습니다.
미리 태깅되어 있는 장르를 [‘romance’, ’fantasy’ ,‘thriller’, ‘drama’, ’ history’ , ‘social’ , ‘#N/A’] 같이 6개의 장르로 재분류했습니다.
재분류 방법은 위 표처럼 직접 모든(364개) 장르를 확인하며 위 6가지 정해놓은 장르 중 가장 유사한 장르 를 선택했습니다. 장르 분류가 애매한 경우는 ‘#N/A’으로 처리했습니다. 이러한 방법을 364개의 장르를 재 태깅했습니다.
재태깅된 장르를 기반으로 줄거리를 6가지 장르로 분류해서 장르별 csv파일로 추출했습니다
[ 모델 테스트 데이터 ]
브로드웨이 뮤지컬 줄거리를 크롤링을 통해 수집했습니다. 크롤링한 웹사이트는 https://broadwaymusicalhome.com/shows.htm 입니다. 총 307건의 줄거리를 크롤링하였습니다. (Beautifulsoup 패키지를 사용했습니다)
### 데이터 전처리
[ NLTK 라이브러리 활용 전처리 ]
장르가 NULL인 줄거리를 제외하고 약 35,000개의 줄거리 데이터 전처리를 진행했습니다. 이때단어 토큰화, 불용어 제거를 위해 NLTK 자연어처리 라이브러리를 사용했습니다.
* 전처리 소스코드 일부 (자세한 내용은 주석처리했습니다.)
```from tqdm import tqdm
all_vocab = {}
all_sentences = []
stop_words = set(stopwords.words('english'))
for i in tqdm(allplot):
all_sentences = word_tokenize(str(i)) # 단어 토큰화를 수행합니다.
result = []
for word in all_sentences:
word = word.lower() # 모든 단어를 소문자화하여 단어의 개수를 줄입니다.
if word not in stop_words: # 단어 토큰화 된 결과에 대해서 불용어를 제거합니다.
if len(word) > 2: # 단어 길이가 2이하인 경우에 대하여 추가로 단어를 제거합니다.
result.append(word)
if word not in all_vocab:
all_vocab[word] = 0
all_vocab[word] += 1
all_sentences.append(result)
all_vocab_sorted = sorted(all_vocab.items(), key = lambda x:x[1], reverse = True)
all_word_to_index = {}
i=0
for (word, frequency) in all_vocab_sorted :
if frequency > 1 : # 빈도수가 적은 단어는 제외한다.
i=i+1
all_word_to_index[word] = i
vocab_size = 15000 #상위 15000개 단어만 사용
words_frequency = [w for w,c in all_word_to_index.items() if c >= vocab_size + 1] # 인덱스가 15000 초과인 단어 제거
for w in words_frequency:
del all_word_to_index[w] # 해당 단어에 대한 인덱스 정보를 삭제
all_word_to_index['OOV'] = len(all_word_to_index) + 1
```
[ 전체 줄거리(전체 : 로맨스, 판타지, 스릴러, 역사 줄거리) 단어 인덱스 부여 ]
단어 사용 빈도수가 적은 단어는 제외하기 위해서 빈도순으로 인덱스를 부여했습니다. 15,000순위 미만인 단어 정보는 삭제했습니다.
[ 학습 데이터(= RM_train+TH_train+FN_train+HS_train ) 인코딩 ]
총 학습 데이터는 7000개로 ‘로맨스 : 스릴러 : 판타지 : 역사 = 2 : 2 : 2 : 1’의 비율로 맞추었습니다.
* 인덱스 부여 소스코드 일부 (자세한 내용은 주석 처리했습니다.)
### 모델 학습
다중 분류를 위해 순환 신경망 모델을 사용했습니다.
* LSTM 모델 구현 소스코드 (자세한 설명은 주석 처리했습니다.)
```from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM, Embedding
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
import numpy as np
M_test=Mu_encoded
M_test= np.array(M_test)
max_len = 230
X_train = pad_sequences(X_train, maxlen=max_len)
X_test = pad_sequences(X_test, maxlen=max_len)
model = Sequential()
model.add(Embedding(15002, 120))
model.add(LSTM(128))
model.add(Dense(4, activation='softmax'))
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=4)
mc = ModelCheckpoint('best_model.h5', monitor='val_acc', mode='max', verbose=1, save_best_only=True)
X_train = np.array(X_train)
Y_train = np.array(Y_train)
X_test = np.array(X_test)
Y_test = np.array(Y_test)
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['acc'])
model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=7, batch_size=64, callbacks=[es, mc])
```
뮤지컬 데이터로 테스트 결과 많은 뮤지컬 작품이 로맨스로 분류된 것을 확인하였고, 스릴러가 가장 적은 비율로 분류된것을 확인했습니 다.
[ 실제 줄거리와 비교 ]
* 오페라의 유령 뮤지컬 줄거리의 장르를 예측한 결과 아래와 같이 로맨스로 정확하게 분류되었습니다. 오 페라의 유령같이 장르 특징이 뚜렷한 줄거리의 경우는 항상 정확한 분류가 이루어지는 것을 확인했습니다.
## 기대효과 및 활용방안
공식 티켓 판매처(인터파크, 예스 24공연 등)에도 뮤지컬 장르구분은 존재하지 않았습니다. 이런 사이트에서 축적된 공연 데이터를 통해 더 정확한 장르 구분을 한다면 일반인들의 뮤지컬에 대한 관심과 접근성을 높 일 수 있을것으로 기대합니다. 또한, 국내 창작 뮤지컬에 로맨스 뮤지컬 뿐만 아니라 다양한 장르가 존재함 을 알릴 수 있을 것입니다. 또한, 분류된 결과를 데이터베이스에 구축하고 웹사이트를 구현하면 뮤지컬 장 르 체계화에 도움이 될 것입니다.
## 결론 및 제언
이진 분류의 경우(예시- 로맨스 vs 판타지)에는 테스트이렇 데이터 정확도가 80%에 이르며 높은 정확도를 보였습니다. 하지만 다중 분류(4개)의 경우에 학습한 경우 정확도가 최대 63%까지만 이르며 모델의 성능을 높이지 못했습니다. 원인으로 첫째, 학습한 줄거리 데이터의 길이가 짧아 내용이 불충분한 경우가 있습니다. 둘째, 줄거리 길이는 충분하지만 분류를 위한 특성을 추출하기 어려운 단어가 많은 경우가 있었습니다.
프로젝트 진행 초반에는 한글 줄거리로 국내 뮤지컬 장르를 분류하고자 했습니다. 하지만 국내 뮤지컬 줄 거리의 대부분은 홍보용으로 내용에 대한 설명이 부실한 경우가 많아 분석에 어려움이 있었습니다. 충분한 한국어 뮤지컬 줄거리 데이터를 구하지 못해 영어 자연어처리로 주제가 변경된 부분이 아쉽습니다
### 역할 분담
양윤지 :
영화 데이터 수집&정리, 줄거리 데이터 장르 재태깅, Train,Test데이터 시각화&패딩, LSTM모델구현, 시각화
김서영 :
뮤지컬 데이터 수집&정리, 장르 데이터 전처리, Train,Test데이터 인코딩, LSTM모델로 학습, 테스트 데이터 학
습, 뮤지컬 데이터 분류,
시연 영상 링크 : https://youtu.be/EKjjQ0tHM4s
......
This file is too large to display.
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# <4월 20일 ~ 4월 24일>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"(1) 플레이DB 웹크롤링 -> 에러처리, csv파일로 옮기기 남음\n",
"\n",
"(2) **tf-idf**로 키워드 추출 고려 <br/>\n",
" -> 줄거리가 짧아서 단어 빈도수가 커야 6정도 ==> 따라서 장르별 대표작들 줄거리를 모아서 해보기로 계획\n",
"\n",
"(3) **word2vec**로 문맥이 비슷한 단어 파악하려했지만<br/>\n",
" ->학습 시간이 오래걸리는 문제 <br/>\n",
" ->단어 관계 파악 후에 어떻게 처리할지 고민,,<br/>\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# <다음주 계획>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"-장르별 키워드 생각(크롤링x, 생각대로) <br/>\n",
"ex) 로맨스 = ['사랑','연인','애틋',........]<br/>\n",
"-가볍게 테스트<br/>\n",
"\n",
"\n",
"### 테스트 결과가 잘 나온다면,, 장르별 키워드 추출 모델 만들기\n",
"#### (잘 나오지 못하면,, 새로운 방법 고려)\n",
"---\n",
"\n",
"(1-1) **불용어 대표 사전 제작**<br/>\n",
"\n",
"- 뮤지컬정보관련된거 ('뮤지컬','제작사','콘텐츠', 뮤지컬이름, '창작','티켓', '매진'....)<br/>\n",
"- 고유 명사<br/>\n",
"- 시간관련 명사, 부사 모든것, (시작 ,나이, '최초','전날' '작년', OOOO년 OO월 OO일, '당시')<br/>\n",
"- 장소는 애매,, ('감옥'이런 장소는 유의미,,)(그런데 지명은 무의미)<br/>\n",
"- 감정이 아닌 단어 예를 들면 ('표현', '세상',...) <br/>\n",
"- 흔한 불용어('우리' ,,)<br/>\n",
"- 한국어가 아닌 단어(한자, 영어)<br/>\n",
"\n",
"(1-2) **장르 대표작 30편씩 고르기**<br/>\n",
"=> 장르별 대표작 30편씩은 줄거리 + 그 이외 기사, 대본 가능한대로 긁어와..<br/>\n",
"\n",
"*키워드 추출은 불용어 사전을 통해 불필요한 단어를 제외하고, 나머지 단어들의 사용된 빈도를 기준으로함.<br/>\n",
"*이때 키워드를 사전 api와 비교해서 나오지 않는것은 이름으로 간주하고 제거.<br/>\n",
"\n",
"(1-3) 장르 대표작들을 통해 **‘장르별 대표 키워드 사전 만들기’** => 이때 키워드가 포함된 문장도 추출<br/>\n",
"(왜냐하면 word2vec을 사용하려면 문장 필요)<br/>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. 불용어 처리\n",
"## 2. 어근 추출\n",
"## 3. 인코딩\n",
"## 4. 패딩 \n",
"\n",
"## 5. "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"5699\n",
"9823\n",
"14020\n",
"2727\n",
"1498\n",
"1464\n"
]
}
],
"source": [
"import pandas as pd\n",
"from nltk.corpus import stopwords\n",
"from nltk.tokenize import word_tokenize\n",
"import re\n",
"\n",
"\n",
"thriller_plot = pd.read_csv('/Users/yangyoonji/Documents/2020-1/2020-dataCapstone/data/moviedata/moviePlot/thrillerPlot.csv')\n",
"drama_plot = pd.read_csv('/Users/yangyoonji/Documents/2020-1/2020-dataCapstone/data/moviedata/moviePlot/dramaPlot.csv')\n",
"fantasy_plot = pd.read_csv('/Users/yangyoonji/Documents/2020-1/2020-dataCapstone/data/moviedata/moviePlot/fantasyPlot.csv')\n",
"history_plot = pd.read_csv('/Users/yangyoonji/Documents/2020-1/2020-dataCapstone/data/moviedata/moviePlot/historyPlot.csv')\n",
"social_plot = pd.read_csv('/Users/yangyoonji/Documents/2020-1/2020-dataCapstone/data/moviedata/moviePlot/socialPlot.csv')\n",
"romance_plot = pd.read_csv('/Users/yangyoonji/Documents/2020-1/2020-dataCapstone/data/moviedata/moviePlot/romancePlot.csv')\n",
"# /Users/김서영/Desktop/datacap/data/moviedata/moviePlot/romancePlot.csv\n",
"\n",
"#romance_plot = pd.read_csv('/Users/김서영/Desktop/datacap/data/moviedata/moviePlot/romancePlot.csv')\n",
"#thriller_plot = pd.read_csv('/Users/김서영/Desktop/datacap/data/moviedata/moviePlot/thrillerPlot.csv')\n",
"#drama_plot = pd.read_csv('/Users/김서영/Desktop/datacap/data/moviedata/moviePlot/dramaPlot.csv')\n",
"#fantasy_plot = pd.read_csv('/Users/김서영/Desktop/datacap/data/moviedata/moviePlot/fantasyPlot.csv')\n",
"#history_plot = pd.read_csv('/Users/김서영/Desktop/datacap/data/moviedata/moviePlot/historyPlot.csv')\n",
"#social_plot = pd.read_csv('/Users/김서영/Desktop/datacap/data/moviedata/moviePlot/socialPlot.csv')\n",
"\n",
"print(len(romance_plot)) #5699 ==> train 2500 test 2500\n",
"print(len(thriller_plot)) #9823 ==> train 2500 test 2500\n",
"print(len(drama_plot))\n",
"print(len(fantasy_plot))\n",
"print(len(history_plot))\n",
"print(len(social_plot))\n",
"\n",
"train_data_size = 1463\n",
"test_data_size = 1463\n",
"\n",
"#전처리(1) 전부 소문자로 변환\n",
"\n",
"\n",
"#romance_plot.줄거리 = romance_plot.줄거리.str.lower()\n",
"#thriller_plot.줄거리 = thriller_plot.줄거리.str.lower()\n",
"\n",
"#전처리(1-1) 데이터 csv 파일로 옮기기\n",
"#romance_plot 2899개 train_data로 to_csv || 2800개 test_data로 to_csv\n",
"#thriller_plot 2899개 train_data로 to_csv || 2800개 test_data로 to_csv\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"RM = [[] for _ in range(5699)]\n",
"for i in range(5699):\n",
" RM[i].append(''.join(romance_plot.줄거리[i]))\n",
" \n",
"TH = [[] for _ in range(9823)]\n",
"for i in range(9823):\n",
" TH[i].append(''.join(thriller_plot.줄거리[i]))\n",
"\n",
"FN = [[] for _ in range(2727)]\n",
"for i in range(2727):\n",
" FN[i].append(''.join(fantasy_plot.줄거리[i]))\n",
" \n",
"HS = [[] for _ in range(1498)]\n",
"for i in range(1498):\n",
" HS[i].append(''.join(history_plot.줄거리[i]))\n",
" \n",
"SC = [[] for _ in range(1464)]\n",
"for i in range(1464):\n",
" SC[i].append(''.join(social_plot.줄거리[i]))\n",
"\n",
"DR = [[] for _ in range(14019)]\n",
"for i in range(14019):\n",
" DR[i].append(''.join(drama_plot.줄거리[i]))\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"allplot = RM+TH+FN+HS+SC+DR #모든 드라마 줄거리"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"35230\n"
]
}
],
"source": [
"print(len(allplot))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 모든 장르 줄거리 "
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|███████████████████████████████████████████████████████████████████████████| 35230/35230 [04:06<00:00, 142.67it/s]\n"
]
}
],
"source": [
"# 토큰화+전처리(3) 전체 불용어 처리\n",
"# 전체 플롯\n",
"from tqdm import tqdm\n",
"all_vocab = {} \n",
"all_sentences = []\n",
"stop_words = set(stopwords.words('english'))\n",
"\n",
"for i in tqdm(allplot):\n",
" all_sentences = word_tokenize(str(i)) # 단어 토큰화를 수행합니다.\n",
" result = []\n",
" for word in all_sentences: \n",
" word = word.lower() # 모든 단어를 소문자화하여 단어의 개수를 줄입니다.\n",
" if word not in stop_words: # 단어 토큰화 된 결과에 대해서 불용어를 제거합니다.\n",
" if len(word) > 2: # 단어 길이가 2이하인 경우에 대하여 추가로 단어를 제거합니다.\n",
" result.append(word)\n",
" if word not in all_vocab:\n",
" all_vocab[word] = 0 \n",
" all_vocab[word] += 1\n",
" all_sentences.append(result) "
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"all_vocab_sorted = sorted(all_vocab.items(), key = lambda x:x[1], reverse = True)\n",
"\n",
"#전처리(4) 인덱스 부여\n",
"all_word_to_index = {}\n",
"i=0\n",
"for (word, frequency) in all_vocab_sorted :\n",
" if frequency > 1 : # 정제(Cleaning) 챕터에서 언급했듯이 빈도수가 적은 단어는 제외한다.\n",
" i=i+1\n",
" all_word_to_index[word] = i\n",
"#print(all_word_to_index)\n",
"\n",
"vocab_size = 15000 #상위 15000개 단어만 사용\n",
"words_frequency = [w for w,c in all_word_to_index.items() if c >= vocab_size + 1] # 인덱스가 200 초과인 단어 제거\n",
"for w in words_frequency:\n",
" del all_word_to_index[w] # 해당 단어에 대한 인덱스 정보를 삭제\n",
"\n",
" \n",
"all_word_to_index['OOV'] = len(all_word_to_index) + 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 로맨스"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|█████████████████████████████████████████████████████████████████████████████| 5699/5699 [00:36<00:00, 154.26it/s]\n"
]
}
],
"source": [
"# 토큰화+전처리(3) 불용어 처리\n",
"# 로맨스 플롯\n",
"\n",
"vocab_r = {} \n",
"RMsentences = []\n",
"RMstop_words = set(stopwords.words('english'))\n",
"\n",
"for i in tqdm(RM):\n",
" RMsentence = word_tokenize(str(i)) # 단어 토큰화를 수행합니다.\n",
" result = []\n",
"\n",
" for word in RMsentence: \n",
" word = word.lower() # 모든 단어를 소문자화하여 단어의 개수를 줄입니다.\n",
" if word not in RMstop_words: # 단어 토큰화 된 결과에 대해서 불용어를 제거합니다.\n",
" if len(word) > 2: # 단어 길이가 2이하인 경우에 대하여 추가로 단어를 제거합니다.\n",
" result.append(word)\n",
" if word not in vocab_r:\n",
" vocab_r[word] = 0 \n",
" vocab_r[word] += 1\n",
" RMsentences.append(result) \n",
"\n",
"R_encoded = []\n",
"for s in RMsentences:\n",
" temp = []\n",
" for w in s:\n",
" try:\n",
" temp.append(all_word_to_index[w])\n",
" except KeyError:\n",
" temp.append(all_word_to_index['OOV'])\n",
" R_encoded.append(temp)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 스릴러"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|█████████████████████████████████████████████████████████████████████████████| 9823/9823 [01:19<00:00, 124.04it/s]\n"
]
}
],
"source": [
"# 토큰화+전처리(3) 불용어 처리\n",
"# 스릴러 플롯\n",
"\n",
"vocab_th = {} \n",
"THsentences = []\n",
"THstop_words = set(stopwords.words('english'))\n",
"\n",
"for i in tqdm(TH):\n",
" \n",
" THsentence = word_tokenize(str(i)) # 단어 토큰화를 수행합니다.\n",
" result = []\n",
"\n",
" for word in THsentence: \n",
" word = word.lower() # 모든 단어를 소문자화하여 단어의 개수를 줄입니다.\n",
" if word not in THstop_words: # 단어 토큰화 된 결과에 대해서 불용어를 제거합니다.\n",
" if len(word) > 2: # 단어 길이가 2이하인 경우에 대하여 추가로 단어를 제거합니다.\n",
" result.append(word)\n",
" if word not in vocab_th:\n",
" vocab_th[word] = 0 \n",
" vocab_th[word] += 1\n",
" THsentences.append(result) \n",
"\n",
"TH_encoded = []\n",
"for s in THsentences:\n",
" temp = []\n",
" for w in s:\n",
" try:\n",
" temp.append(all_word_to_index[w])\n",
" except KeyError:\n",
" temp.append(all_word_to_index['OOV'])\n",
" TH_encoded.append(temp)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#전처리 방법에는 NLTK의 FreqDist, 케라스(Keras) 토크나이저도 사용 가능."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"로맨스 플롯, 스릴러 따로 토큰화 해서 x train에 넣을지... 고민중\n",
"\n",
"이번주 : 전처리 완료, \n",
"이번 달 목표 : 뮤지컬 장르 분류 << 다양한 모델 사용해보기.\n",
"\n",
"6월에 교차검증 및 장르 시각화 설계까지.\n",
"\n",
"다음주 : 2진분류(LSTM) 완료, RNN 분류기 만들어보기"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 학습데이터"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 영화 줄거리는 X_train에, 장르 정보는 y_train에 저장된다.\n",
"# 테스트용 줄거리 X_test에, 테스트용 줄거리의 장르 정보는 y_test에 저장된다.\n",
"#맞춰서 저장하기. (진행중)\n",
"\n",
"#X_train = train_sc_df.dropna().drop(‘trade_price_idx_value’, axis=1)"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [],
"source": [
"X_train = []\n",
"Y_train = [] #0 : romance, 1 : thriller \n",
"for i in range(train_data_size):\n",
" X_train.append(R_encoded[i])\n",
" Y_train.append(0)\n",
" X_train.append(TH_encoded[i])\n",
" Y_train.append(1)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"#print(X_train[2])\n",
"#print(Y_train[2])"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"줄거리 최대 길이 : 1974\n",
"줄거리 평균 길이 : 267.093984962406\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD4CAYAAAAD6PrjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAdIElEQVR4nO3df5DU9Z3n8eeLSYt7ahQUlR+yY3IkhaEuaii1SjbnrBtFsiXkKslJpZTVuSPZEy7WJhYYrko3ualyvY1WZPfYwx1K3DJjvDWWVNQlxHPPoyoa0RDFTFzREDNCgAQWUZQ4zPv++H76azP0DD3TPd3T7etR1dXf/vS3+/ue6Yb3fD+fz/f9UURgZmYGMKHRAZiZ2fjhpGBmZjknBTMzyzkpmJlZzknBzMxyH2p0AMdzxhlnRHt7e6PDsBb13HPP/TYiptT7uP5e21iq5ns97pNCe3s7W7ZsaXQY1qIk/aoRx/X32sZSNd9rdx+ZmVnOScHMzHLHTQqSzpH0pKReSS9J+mpqnyxpk6RX0v2k1C5Jd0vaLukFSReWvNeStP8rkpaM3Y9lZmajUcmZQj/wtYiYDVwC3CjpPGAl8EREzAKeSI8BrgJmpdtSYA1kSQS4FbgYuAi4tZhIzMxsfDhuUoiIXRHxfNo+CPQC04GFwPq023pgUdpeCNwXmaeB0yRNBa4ENkXEvojYD2wC5tf0pzEzs6qMaExBUjtwAfAMcFZE7IIscQBnpt2mA78ueVlfahuq3Wqgp6eHOXPm0NbWxpw5c+jp6Wl0SGbWhCqekirpZOAh4KaIeFPSkLuWaYth2ssdaylZ1xMzZ86sNMQPrJ6eHlatWkV3dzfz5s1j8+bNdHZ2ArB48eIGR2dmzaSiMwVJBbKEcH9EfD81707dQqT7Pam9Dzin5OUzgJ3DtB8jItZGxNyImDtlSt2vK2o6XV1ddHd309HRQaFQoKOjg+7ubrq6uhodmpk1mUpmHwnoBnoj4s6SpzYAxRlES4BHStqvS7OQLgEOpO6ljcAVkialAeYrUptVqbe3l3nz5h3VNm/ePHp7exsUkZk1q0q6jy4FrgVelLQ1tX0DuB14UFIn8DrwhfTcY8ACYDtwCLgeICL2SfoW8Gza75sRsa8mP8UH3OzZs9m8eTMdHR152+bNm5k9e3YDo7JaaF/5aL694/bPNjAS+6A4blKIiM2UHw8AuLzM/gHcOMR7rQPWjSRAO75Vq1bR2dl5zJiCu4/MbKTGfe0jO77iYPLy5cvp7e1l9uzZdHV1eZDZzEbMSaFFLF682EnAzKrm2kdmZpZzUjAzs5yTgtkgkk6U9BNJP0tFIP8ytZ8r6ZlU0PF7kk5I7RPT4+3p+fZGxm9WDScFs2MdBv44Ij4JnA/MT9fc/BVwVyoCuR/oTPt3Avsj4t8Cd6X9zJqSk4LZIKmY41vpYSHdAvhj4B9T++AikMXikP8IXK5h6sCYjWdOCmZlSGpLF2vuIavo+yrwrxHRn3YpLeiYF3tMzx8ATq9vxGa14aRgVkZEHImI88lqdF0ElLs8vFjQsaJij5KWStoiacvevXtrF6xZDTkpmA0jIv4V+GeyBaZOk1S8tqe0oGNe7DE9fypwTAkXF3q0ZuCkYDaIpCmSTkvbfwD8CdniUk8Cn0+7DS4CWSwO+Xng/6RyL2ZNx1c0mx1rKrBeUhvZH04PRsQPJP0ceEDSfwd+SlY9mHT/D5K2k50hXNOIoM1qwUnBbJCIeIFshcHB7a+RjS8Mbn+X96sEmzU1dx+ZmVnOScGsSbSvfPSo9RXMxoKTgpmZ5ZwUzMwsV8kazesk7ZG0raTte5K2ptuO4jKdktolvVPy3N+VvOZTkl5MRcPudhkAM7Pxp5LZR/cCfwPcV2yIiP9Y3Jb0bbLL+oteTVeCDrYGWAo8TbaO83zg8ZGHbGZmY+W4ZwoR8RRlrs4ESH/tfxHoGe49JE0FPhwRP04X9dzH+8XEzMxsnKh2TOGPgN0R8UpJ27mSfirp/0r6o9Q2nawUQFFpMbFjuEaMmVljVJsUFnP0WcIuYGZEXAD8BfBdSR+mwoJh+ROuEWNm1hCjvqI5Ff76D8Cnim0RcZhsgRIi4jlJrwIfIzszmFHy8tJiYmZmNk5Uc6bwJ8AvIiLvFkqFxNrS9keAWcBrEbELOCjpkjQOcR3vFxMzM7NxopIpqT3Aj4GPS+qTVFyC8BqOHWD+NPCCpJ+RrUD1lYgoDlL/OfD3wHayBUs888jMbJw5bvdRRCweov3PyrQ9BDw0xP5bgDkjjM/MzOrIVzSbmVnOScHMzHJOCmZmlnNSMDOznJOCmZnlnBTMzCznpGBmZjknhRbR09PDnDlzaGtrY86cOfT0DFu41sysrFHXPrLxo6enh1WrVtHd3c28efPYvHkznZ3ZheeLF5e99tDMrCyfKbSArq4uuru76ejooFAo0NHRQXd3N11dXY0OzcyajJNCC+jt7aWvr++o7qO+vj56e3sbHZqZNRknhRYwbdo0VqxYwerVq3n33XdZvXo1K1asYNq0aY0OrSlJOkfSk5J6Jb0k6aup/TZJb5SsQb6g5DW3pPXHX5Z0ZeOiN6uOxxRaRLbK6dCPbUT6ga9FxPOSTgGek7QpPXdXRPx16c6SziOrGvwJYBrwI0kfi4gjozl4+8pHqwjdrDo+U2gBO3fu5I477mD58uWceOKJLF++nDvuuIOdO72O0WhExK6IeD5tHwR6GWb5WGAh8EBEHI6IX5KVh79o7CM1qz0nhRYwe/ZsZsyYwbZt2zhy5Ajbtm1jxowZzJ49u9GhNT1J7cAFwDOpaZmkFyStkzQptU0Hfl3ysrJrkHvtcWsGTgotYNWqVXR2dvLkk0/y3nvv8eSTT9LZ2cmqVasaHVpTk3Qy2fogN0XEm8Aa4KPA+WTrkX+7uGuZlx/Tf+e1x60ZHHdMQdI64E+BPRExJ7XdBvxnoPjnzjci4rH03C1AJ3AE+K8RsTG1zwe+A7QBfx8Rt9f2R/ngKl6LsHz5cnp7e5k9ezZdXV2+RqEKkgpkCeH+iPg+QETsLnn+HuAH6WEfcE7Jy70GuTWtSgaa7wX+BrhvUHvFA27p6b8FPkP2D+hZSRsi4udVxG4lFi9e7CRQI2kd8W6gNyLuLGmfmtYbB/gcsC1tbwC+K+lOsu/9LOAndQzZrGYqWY7zqdSvWol8wA34paTSAbftEfEagKQH0r5OCjYeXQpcC7woaWtq+wawWNL5ZF1DO4AvA0TES5IeJPs+9wM3jnbmkVmjVTMldZmk64AtZNP39pMNrj1dsk/pgNvggbiLh3pjSUuBpQAzZ86sIkSzkYuIzZQfJ3hsmNd0Ab6E3JreaAeaRzrgVtFAXP6EB+TMzBpiVGcKoxxw80Ccmdk4N6ozBUlTSx4OHnC7RtJESefy/oDbs8AsSedKOoFsMHrD6MM2M7OxUMmU1B7gMuAMSX3ArcBlIx1wk7QM2Eg2JXVdRLxU85/GzMyqUsnso3LzHLuH2b/sgFu6jmHIgTozM2s8X9FsZmY5JwUzM8s5KZiZWc5JwczMck4KZmaWc1Iwa0HtKx/1Cm42Kk4KZmaWc1IwM7Ock4KZmeWqKZ1tZk2kdIxhx+2fbWAkNp75TMHMzHJOCmZmlnNSMDOznJOCmZnlnBTMzCznpGBmZrnjJgVJ6yTtkbStpO1/SPqFpBckPSzptNTeLukdSVvT7e9KXvMpSS9K2i7pbkkamx/JzMxGq5IzhXuB+YPaNgFzIuLfAf8C3FLy3KsRcX66faWkfQ2wlGzd5lll3tPMzBrsuEkhIp4C9g1q+2FE9KeHTwMzhnsPSVOBD0fEjyMigPuARaML2czMxkotrmi+AfheyeNzJf0UeBP4bxHx/4DpQF/JPn2prSxJS8nOKpg5c2YNQjSrnKRzyP5wORsYANZGxHckTSb7rrcDO4AvRsT+1BX6HWABcAj4s4h4vt5xuyqq1UJVA82SVgH9wP2paRcwMyIuAP4C+K6kDwPlxg9iqPeNiLURMTci5k6ZMqWaEM1Gox/4WkTMBi4BbpR0HrASeCIiZgFPpMcAV/F+t+hSsq5Ss6Y06qQgaQnwp8CXUpcQEXE4In6Xtp8DXgU+RnZmUNrFNAPYOdpjm42liNhV/Es/Ig4CvWRntguB9Wm39bzfBboQuC8yTwOnpS5Ts6Yzqu4jSfOBFcC/j4hDJe1TgH0RcUTSR8j+cnotIvZJOijpEuAZ4DpgdfXhm40tSe3ABWTf27MiYhdkiUPSmWm36cCvS15W7B7dNei96t4t6i4lG6njJgVJPcBlwBmS+oBbyWYbTQQ2pZmlT6eZRp8GvimpHzgCfCUiioPUf042k+kPgMfTzWzcknQy8BBwU0S8Ocws6oq6RyNiLbAWYO7cuUN2n9aDK6baUI6bFCJicZnm7iH2fYjsH1G557YAc0YUnVmDSCqQfZfvj4jvp+bdkqams4SpwJ7U3gecU/Jyd49a0/IVzWaDpNlE3UBvRNxZ8tQGYEnaXgI8UtJ+nTKXAAeK3UxmzcaL7Jgd61LgWuBFSVtT2zeA24EHJXUCrwNfSM89RjYddTvZlNTr6xWoxwys1pwUzAaJiM2UHycAuLzM/gHcOKZBmdWJu4/MzCznpGBmZjknBTMzyzkpmJlZzgPNZk3GM45sLPlMwczMck4KZmaWc1JoET09PcyZM4e2tjbmzJlDT09Po0MysybkMYUW0NPTw6pVq+ju7mbevHls3ryZzs5OABYvLle6yux9Lo5npXym0AK6urro7u6mo6ODQqFAR0cH3d3ddHV1NTo0M2syTgotoLe3l76+vqO6j/r6+ujt7W10aGbWZNx91AKmTZvGihUruP/++/Puoy996UtMmzat0aGZWZNxUmgRhw4d4oYbbuD1119n5syZHDp0iFNOOaXRYZlZk6mo+0jSOkl7JG0raZssaZOkV9L9pNQuSXdL2i7pBUkXlrxmSdr/lbTGs9XAG2+8wcDAQNl7M7ORqHRM4V5g/qC2lcATETELeCI9BriKbG3mWWTr0a6BLImQLeV5MXARcGsxkVh12traKBQKbNy4kd///vds3LiRQqFAW1tbo0MzsyZTUVKIiKeAfYOaFwLr0/Z6YFFJ+32ReRo4LS1deCWwKSL2RcR+YBPHJhobhf7+fgqFwlFthUKB/v7+BkVkZs2qmtlHZxWXHEz3Z6b26cCvS/brS21DtR9D0lJJWyRt2bt3bxUhfnBcf/31LF++nBNPPJHly5dz/fV1W/zLzFrIWExJLbdiVQzTfmxjxNqImBsRc6dMmVLT4FrRjBkzWLNmDW+//TYAb7/9NmvWrGHGjBkNjszMmk01SWF36hYi3e9J7X3AOSX7zQB2DtNuVVq0aBEHDhxgx44dDAwMsGPHDg4cOMCiRYuO/2IzsxLVJIUNQHEG0RLgkZL269IspEuAA6l7aSNwhaRJaYD5itRmVbrnnnsAmDBhwlH3xXYzs0pVdJ2CpB7gMuAMSX1ks4huBx6U1Am8Dnwh7f4YsADYDhwCrgeIiH2SvgU8m/b7ZkQMHry2UTh8+DATJkygra2NgYGBfNbR4cOHGxyZmTWbipJCRAxVVe3yMvsGcOMQ77MOWFdxdFaxgYEBzjzzTPbs2cPpp5/Ob37zm0aHZGZNyLWPWsjNN9/MwYMHufnmmxsdipk1KZe5aCG33norX//61znppJMaHYqZNSmfKbSQt956i4jgrbfeanQoTW2Isi63SXpD0tZ0W1Dy3C2prMvLkq5sTNRmteGk0AImTpwIwMknn3zUfbHdRuxeyl9tf1dEnJ9ujwFIOg+4BvhEes3/lOT6Ita0nBRawJEjRygUCvkZwltvvUWhUODIkSMNjqw5DVHWZSgLgQci4nBE/JJs1t1FYxac2RhzUmgB/f39nHrqqbS3tzNhwgTa29s59dRTXfuo9palyr/rSoo5unyLtRQnhRYgibPPPptdu3YxMDDArl27OPvss5HKVRaxUVoDfBQ4H9gFfDu1u3yLtRQnhRYQEWzbti2vlFooFNi2bRvZJSNWCxGxOyKORMQAcA/vdxG5fIu1FCeFFjFhwoSjxhSKpS6sNop1vpLPAcWZSRuAayRNlHQu2ToiP6l3fLXSvvJR2lc+2ugwrIF8nUKLGBgY4EMf+hD9/f35vY3OEGVdLpN0PlnX0A7gywAR8ZKkB4GfA/3AjRHhEX5rWk4KLaSYCJwQqjNEWZfuYfbvArrGLiKz+nEfQwu5+uqr2bt3L1dffXWjQzGzJuUzhRZRKBR4/PHHmTJlCoVCgUKhwHvvvdfosMysyfhMoUWccMIJTJ8+nQkTJjB9+nROOOGERodkZk3ISaEFTJgwgUOHDvHOO+8wMDDAO++8w6FDhzwDycxGzN1HTarchWm7d+8+6j4i8v18zYKZVWLUf0pK+nhJxcitkt6UdJOrSdZHRBx1W7ZsWV4Ab+LEiSxbtuyo583MKjHqM4WIeJnskn9SVcg3gIfJlt+8KyL+unT/QdUkpwE/kvQxz+mujdWrV7N69Wok8e677zY6HDNrUrXqdL4ceDUifjXMPq4maWY2ztUqKVwD9JQ8rqqapJmZNUbVSUHSCcDVwP9OTVVXk3SJYTOzxqjFmcJVwPMRsRtqU03SJYbNzBqjFklhMSVdRx+UapJmZq2oqusUJP0b4DOkipHJHa4maWbWnKpKChFxCDh9UNu1w+zvapJmZuOY6yCYmVnOScHMzHJOCmZmlnNSMDOznJOCmZnlnBTMzCznpGBmZjknBTMzyzkpmJlZzknBzMxyTgpmg6R1QPZI2lbSNlnSJkmvpPtJqV2S7k7LzL4g6cLGRW5WPScFs2PdC8wf1LYSeCIiZgFPpMeQlY6flW5LydYTMWtaTgpmg0TEU8C+Qc0LgfVpez2wqKT9vsg8DZw2qHy8WVNxUjCrzFkRsQsg3Z+Z2iteZtYrClozcFIwq07Fy8x6RUFrBk4KZpXZXewWSvd7UnvFy8yaNQMnBbPKbACWpO0lwCMl7delWUiXAAeK3UxmzajqpCBph6QXJW2VtCW1efqeNS1JPcCPgY9L6pPUCdwOfEbSK2RL0N6edn8MeA3YDtwD/JcGhGxWM1Utx1miIyJ+W/K4OH3vdkkr0+MVHD1972Ky6XsX1ygGs5qIiMVDPHV5mX0DuHFsIzKrn7HqPvL0PTOzJlSLpBDADyU9J2lpaqtq+p6n7pmZNUYtuo8ujYidks4ENkn6xTD7VjR9LyLWAmsB5s6dW3Z6n5mNnfaVj+bbO27/bAMjsXqr+kwhInam+z3Aw8BFePqemVlTqiopSDpJ0inFbeAKYBuevmdm1pSq7T46C3hYUvG9vhsR/yTpWeDBNJXvdeALaf/HgAVk0/cOAddXeXwzM6uhqpJCRLwGfLJM++/w9D0zs6bjK5rNzCznpGBmZjknBTMzyzkpmJlZzknBzMxyTgrj3OTJk5FU8Q2oeN/Jkyc3+Kczs/GmVlVSbYzs37+fbCZv7RWTiJlZkc8UzMws56RgZmY5JwUzM8s5KZiZWc4DzWY2LK+t8MHiMwUzM8s5KZiZWc5JwczMck4KZmaWG3VSkHSOpCcl9Up6SdJXU/ttkt6QtDXdFpS85hZJ2yW9LOnKWvwAZmZWO9XMPuoHvhYRz6d1mp+TtCk9d1dE/HXpzpLOA64BPgFMA34k6WMRcaSKGMzqStIO4CBwBOiPiLmSJgPfA9qBHcAXI2J/o2I0q8aozxQiYldEPJ+2DwK9wPRhXrIQeCAiDkfEL8nWab5otMc3a6COiDg/IuamxyuBJyJiFvBEemzWlGpynYKkduAC4BngUmCZpOuALWRnE/vJEsbTJS/rY4gkImkpsBRg5syZtQixacWtH4bbTh2797ZaWAhclrbXA/8MrGhUMGbVqDopSDoZeAi4KSLelLQG+BYQ6f7bwA1AuZKcZct/RsRaYC3A3Llzx6ZEaJPQX745plVS47YxeetWFsAPJQXwv9J39ayI2AXZGbSkM8u90H/sWDOoKilIKpAlhPsj4vsAEbG75Pl7gB+kh33AOSUvnwHsrOb4Zg1waUTsTP/xb5L0i0pf6D92rBlUM/tIQDfQGxF3lrRPLdntc8C2tL0BuEbSREnnArOAn4z2+GaNEBE70/0e4GGycbHdxe99ut/TuAjNqlPNmcKlwLXAi5K2prZvAIslnU92mr0D+DJARLwk6UHg52Qzl270zCNrJpJOAiZExMG0fQXwTbI/eJYAt6f7RxoX5dhyHaTWN+qkEBGbKT9O8Ngwr+kCukZ7TLMGOwt4OK1Y9yHguxHxT5KeBR6U1Am8DnyhgTGaVcVVUs0qFBGvAZ8s0/474PL6R2RWey5zYWZmOScFMzPLOSmY2ai0r3z0qIFnaw1OCmZmlnNSMDOznJOCmZnlnBTMzCzn6xSaQLpYquYmTZo0Ju9rZs3LSWGcG2mFVEljVlXVzFqfu4/MzCznpGBmZjknBTMzy3lMwcyq4nLarcVnCmZmlnNSMDOzXN2TgqT5kl6WtF3Synof38zMhlbXpCCpDfhb4CrgPLKlO8+rZwxmZja0eg80XwRsTytYIekBYCHZus02AsNd5VzuOV/QZvXgQefmV+/uo+nAr0se96W2o0haKmmLpC179+6tW3DNJCJGdDMzq0S9k0K5P2+P+R8rItZGxNyImDtlypQ6hGVmZlD/pNAHnFPyeAaws84xmJnZEOqdFJ4FZkk6V9IJwDXAhjrHYGZmQ6jrQHNE9EtaBmwE2oB1EfFSPWMws/orDkB78Hn8q3uZi4h4DHis3sc1s/oqnYlUrm2oBOEZTI3lK5rNasQXZlorcEE8sxoouTDzM2QTKp6VtCEifA3OEMqdSYx0H59J1J6Tgllt+MLMD6ihEtdYJayxHp/ReL+wSdJe4FeNjqOJnAH8ttFBNJE/jIiqL4aR9HlgfkT8p/T4WuDiiFhWss9SYGl6+HHg5TJvNd4+v/EUj2Mpr1wso/5ej/szhVr8g/0gkbQlIuY2Oo4PoONemBkRa4G1w77JOPv8xlM8jqW8WsfigWaz2vCFmdYSnBTMasMXZlpLGPfdRzZiw3ZP2Nio4YWZ4+3zG0/xOJbyahrLuB9oNjOz+nH3kZmZ5ZwUzMws56TQIiStk7RH0rZGx2Kj04gyGZJ2SHpR0lZJW1LbZEmbJL2S7ieldkm6O8X3gqQLqzz2Md/Z0Rxb0pK0/yuSltQ4ntskvZF+P1slLSh57pYUz8uSrixpr+pzlHSOpCcl9Up6SdJX6/q7GekKXr6NzxvwaeBCYFujY/FtVJ9fG/Aq8BHgBOBnwHl1OO4O4IxBbXcAK9P2SuCv0vYC4HGyazIuAZ6p8tjHfGdHemxgMvBaup+UtifVMJ7bgK+X2fe89BlNBM5Nn11bLT5HYCpwYdo+BfiXdLy6/G58ptAiIuIpYF+j47BRy8tkRMTvgWKZjEZYCKxP2+uBRSXt90XmaeA0SVNHe5AhvrMjPfaVwKaI2BcR+4FNwPwaxjOUhcADEXE4In4JbCf7DKv+HCNiV0Q8n7YPAr1kyxbX5XfjpGA2PlS0fvkYCOCHkp5LZTgAzoqIXZD9BwWcWccYR3rsesS0LHXLrCt22dQrHkntwAXAM9Tpd+OkYDY+VLR++Ri4NCIuBK4CbpT06WH2bVSMwx17rGNaA3wUOB/YBXy7XvFIOhl4CLgpIt4cbtdaxuKkYDY+NKRMRkTsTPd7gIfJuj92F7uF0v2eOsY40mOPaUwRsTsijkTEAHAP2e9nzOORVCBLCPdHxPdTc11+N04KZuND3ctkSDpJ0inFbeAKYFs6bnGmyhLgkbS9AbguzXa5BDhQ7M6ooZEeeyNwhaRJqWvnitRWE4PGTD5H9vspxnONpImSzgVmAT+hBp+jJAHdQG9E3FnyVH1+N9XMHvBt/NyAHrLT2/fI/kLobHRMvo34M1xANtPkVWBVHY73EbLZMT8DXioeEzgdeAJ4Jd1PTu0iW0joVeBFYG6Vxz/mOzuaYwM3kA30bgeur3E8/5CO90L6z3dqyf6rUjwvA1fV6nME5pF187wAbE23BfX63bjMhZmZ5dx9ZGZmOScFMzPLOSmYmVnOScHMzHJOCmZmlnNSMDOznJOCmZnl/j8lGxvkzw55lwAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 2 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"len_result = [len(s) for s in X_train]\n",
"print(\"줄거리 최대 길이 : \",max(len_result))\n",
"print(\"줄거리 평균 길이 : \",sum(len_result)/len(len_result))\n",
"\n",
"plt.subplot(1,2,1)\n",
"plt.boxplot(len_result)\n",
"plt.subplot(1,2,2)\n",
"plt.hist(len_result, bins=50)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 테스트 데이터\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"X_test = []\n",
"Y_test = [] #0 : romance, 1 : thriller \n",
"for i in range(test_data_size):\n",
" X_test.append(R_encoded[train_data_size+i])\n",
" Y_test.append(0)\n",
" X_test.append(TH_encoded[train_data_size+i])\n",
" Y_test.append(1)\n",
" \n",
" "
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"줄거리 최대 길이 : 1749\n",
"줄거리 평균 길이 : 197.71394395078605\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAbkUlEQVR4nO3db3Bc1Znn8e/Psk2ymRgsMIY1sCLBSZm4NoaoGAqUFIbB2GQqTraSXZut4M1o43jGuJIKWxszegGTLVGE3Qwz7DBQZuSK2cqIZIfx4AJniNc4Q7kqEOTYAYMgFsQBY0dWYkMCGf5IevZFn7ZbUuuP1a3uVt/fp+rWvf3c092P3K1H1+eee64iAjMzy4YZ1U7AzMwqx0XfzCxDXPTNzDLERd/MLENc9M3MMmRmtRMYz1lnnRVNTU3VTsPq1J49e34dEfMq/b7+XttUGut7XfNFv6mpia6urmqnYXVK0i+r8b7+XttUGut77e4dM7MMcdE3M8sQF30zswxx0TczyxAXfTOzDBm36EvaLOmopP0Fse9J2peWg5L2pXiTpH8t2HdfwXM+IelZST2S7pakqfmRsqmzs5PFixfT0NDA4sWL6ezsrHZKZlaDJjJk8zvA3wAP5AMR8Z/y25K+DbxR0P6liFhS5HXuBdYCTwLbgeXAD049ZRuus7OTtrY2Ojo6aGlpYffu3bS2tgKwevXqKmdnZrVk3CP9iHgCOFZsXzpa/4/AmIeVks4F5kTEjyM3l/MDwGdPPV0rpr29nY6ODpYuXcqsWbNYunQpHR0dtLe3Vzs1M6sxpfbpfxLojYgDBbELJe2V9C+SPpliC4BDBW0OpVhRktZK6pLU1dfXV2KK9a+7u5uWlpYhsZaWFrq7u6uUkZnVqlKvyF3N0KP8I8AFEfEbSZ8A/knSx4Bi/fej3r0lIjYBmwCam5t9l5dxLFq0iN27d7N06dITsd27d7No0aIqZmXl0LTx0RPbB+/4dBUzsXox6SN9STOB/wB8Lx+LiHci4jdpew/wEvARckf25xU8/Tzg8GTf24Zqa2ujtbWVXbt28d5777Fr1y5aW1tpa2urdmpmVmNKOdL/I+CFiDjRbSNpHnAsIgYkfQhYCLwcEcck/U7S5cBTwI3A/y4lcTspf7J2w4YNdHd3s2jRItrb230S18xGGLfoS+oErgLOknQIuDUiOoBVjDyB+yngm5L6gQFgXUTkTwL/KbmRQO8nN2rHI3fKaPXq1S7yZjaucYt+RBStJBHxX4rEHgIeGqV9F7D4FPMzM7My8hW5ZmYZ4qJvZpYhLvpmZhniom9mliEu+mZmGeKib1ZEmj322TRbbFeKNUraIelAWs9NcaWZY3skPSPp0upmbzY6F32z0S2NiCUR0ZwebwR2RsRCYGd6DLCC3IWIC8nNJHtvxTM1myAXfbOJWwlsSdtbODlT7Erggch5EjgjzSxrVnNc9M2KC+CHkvZIWpti8yPiCEBan53iC4BXC55bdBZZzx5rtaDUWTbN6tWVEXFY0tnADkkvjNF2QrPIevZYqwU+0jcrIiIOp/VRYCtwGdCb77ZJ66Op+SHg/IKnexZZq1ku+mbDSPqApA/mt4FlwH5gG7AmNVsDPJy2twE3plE8lwNv5LuBzGqNu3fMRpoPbM3dDZSZwN9HxD9Lehr4vqRW4BXgC6n9duB6oAf4PfClyqdsNjEu+mbDRMTLwMeLxH8DXFMkHsD6CqRmVjJ375iZZYiLvplZhrjom5lliIu+mVmGuOibmWXIuEVf0mZJRyXtL4jdJum1NAPhPknXF+y7Jc02+KKk6wriy1OsR9LG4e9jZmZTbyJH+t8BlheJ35VmIFwSEdsBJF0MrAI+lp7zt5IaJDUA95CbjfBiYHVqa2ZmFTTuOP2IeEJS0wRfbyXwYES8A/xCUg+5y9cBetL4ZyQ9mNo+f8oZm5nZpJXSp39TumHE5vzNJBh9tsEJzUJoZmZTa7JF/17gw8AS4Ajw7RQfbbbBCc1CmOcpaM3Mpsakin5E9EbEQEQMAvdzsgtntNkGT2kWwojYFBHNEdE8b968yaRoZmZFTKroD7sr0OfIzUAIudkGV0k6TdKF5G4f9xPgaWChpAslzSZ3snfb5NM2M7PJGPdErqRO4CrgLEmHgFuBqyQtIddFcxD4CkBEPCfp++RO0PYD6yNiIL3OTcBjQAOwOSKeK/tPY2ZmY5rI6J3VRcIdY7RvB9qLxLeTm4LWzMyqxFfkmplliIu+mVmGuOibmWWIi76ZWYa46JuZZYiLvplZhrjom5lliIu+mVmGuOibmWWIi76ZWYa46JuZZYiLvplZhrjom5lliIu+mVmGuOibmWWIi76ZWYa46JuNQlKDpL2SHkmPL5T0lKQDkr6Xbv1Juj3o9yT1pP1N1czbbCwu+maj+yrQXfD4W8BdEbEQOA60pngrcDwiLgLuSu3MapKLvlkRks4DPg38XXos4GrgH1KTLcBn0/bK9Ji0/5rU3qzmuOibFfdXwH8HBtPjM4HXI6I/PT4ELEjbC4BXAdL+N1L7ISStldQlqauvr28qczcb1bhFX9JmSUcl7S+I/U9JL0h6RtJWSWekeJOkf5W0Ly33FTznE5KeTf2ed/tIyGqVpD8GjkbEnsJwkaYxgX0nAxGbIqI5IprnzZtXhkzNTt1EjvS/AywfFtsBLI6Ifw/8HLilYN9LEbEkLesK4vcCa4GFaRn+mma14krgM5IOAg+S69b5K+AMSTNTm/OAw2n7EHA+QNp/OnCskgmbTdS4RT8inmDYFzgifljw39wnyf0CjErSucCciPhxRATwACf7Q81qSkTcEhHnRUQTsAp4PCL+M7AL+HxqtgZ4OG1vS49J+x9P33OzmlOOPv0/AX5Q8PjCNMztXyR9MsUWkDsayivsDx3BfZ9Wo74BfF1SD7k++44U7wDOTPGvAxurlJ/ZuGaO32R0ktqAfuC7KXQEuCAifiPpE8A/SfoYE+zzPLEjYhOwCaC5udlHTFY1EfEj4Edp+2XgsiJt3ga+UNHEzCZp0kVf0hrgj4Fr8v+VjYh3gHfS9h5JLwEfIXdkX9gFVNgfamYFmjY+Wu0UrI5NqntH0nJy/9X9TET8viA+T1JD2v4QuRO2L0fEEeB3ki5Po3Zu5GR/qJmZVci4R/qSOoGrgLMkHQJuJTda5zRgRxp5+WQaqfMp4JuS+oEBYF1E5E8C/ym5kUDvJ3cOoPA8gJmZVcC4RT8iVhcJdxSJEREPAQ+Nsq8LWHxK2ZmZWVn5ilwzswxx0TczyxAXfTOzDHHRrxOdnZ0sXryYhoYGFi9eTGdnZ7VTMrMaVNLFWVYbOjs7aWtro6Ojg5aWFnbv3k1ra26q99Wri52HN7Os8pF+HWhvb6ejo4OlS5cya9Ysli5dSkdHB+3t7dVOzcxqjIt+Heju7qalpWVIrKWlhe7u7lGeYWZZ5aJfBxYtWsTu3buHxHbv3s2iRYuqlJGZ1SoX/TrQ1tZGa2sru3bt4r333mPXrl20trbS1tZW7dTMrMb4RG4dyJ+s3bBhA93d3SxatIj29nafxDWzEVz068Tq1atd5M1sXO7eMTPLEBf9OuGLs8xsIty9Uwd8cZaZTZSP9OtAe3s7N9xwAxs2bOB973sfGzZs4IYbbvDFWWY2go/068Dzzz/PK6+8wttvv83g4CA///nPufvuu3nzzTernZqZ1Rgf6deBGTNm8NZbb3HHHXcMWc+Y4Y/XzIZyVagDAwMDnH766VxyySXMmjWLSy65hNNPP52BgYFqp2ZmNcbdO3WiqamJa665hohAEkuWLOH48ePVTsvMasyEjvQlbZZ0VNL+glijpB2SDqT13BSXpLsl9Uh6RtKlBc9Zk9ofkLSm/D9ONkli7969rFu3jtdff51169axd+9e0k3rrU40bXyUpo2PVjsNm+Ym2r3zHWD5sNhGYGdELAR2pscAK4CFaVkL3Au5PxLArcAfApcBt+b/UFhp8n33W7dupbGxka1btw6Jm5nlTagqRMQTwLFh4ZXAlrS9BfhsQfyByHkSOEPSucB1wI6IOBYRx4EdjPxDYpMwMDDA1VdfTW9vL4ODg/T29nL11Ve7T9/MRijlUHB+RBwBSOuzU3wB8GpBu0MpNlp8BElrJXVJ6urr6yshxWyYOXMme/fuZefOnbz77rvs3LmTvXv3MnOmT9mY2VBTURWKdSTHGPGRwYhNwCaA5ubmom3spDlz5nD8+HGuvfZaBgYGaGhoYHBwkLlz3XtmZkOVcqTfm7ptSOujKX4IOL+g3XnA4THiVqJjx4b3vI0dN7PsKqXobwPyI3DWAA8XxG9Mo3guB95I3T+PAcskzU0ncJelmJVIEuvWraO/v5+IoL+/n3Xr1nn0ziRJep+kn0j6maTnJP1Fil8o6ak0+ux7kman+GnpcU/a31TN/M3GMtEhm53Aj4GPSjokqRW4A7hW0gHg2vQYYDvwMtAD3A/8GUBEHAP+B/B0Wr6ZYlaiiGD79u1D7py1fft2ItwzNknvAFdHxMeBJcDydADzLeCuNGLtONCa2rcCxyPiIuCu1M6sJk2oTz8iRpuq8ZoibQNYP8rrbAY2Tzg7m5DTTjuNlpaWIXfOamlp4Ve/+lW1U5uW0nc4P3HRrLQEcDVwQ4pvAW4jNyR5ZdoG+AfgbyQp/FfXapCHd9SBL3/5y9xzzz3MmDGDwcFBXnjhBZ5//nnWry/6t9cmQFIDsAe4CLgHeAl4PSL6U5PC0WcnRqZFRL+kN4AzgV8Pe8215K5d4YILLpjqH8GsKF+9Uyci4sS4/IGBAXftlCgiBiJiCbkBB5cBi4o1S+sJjUyLiE0R0RwRzfPmzStfsmanwEW/Dtx33300Njby+OOP8+677/L444/T2NjIfffdV+3Upr2IeB34EXA5uQsN8/87Lhx9dmJkWtp/OiMvZjSrCS76daC/v58rrriCFStWMHv2bFasWMEVV1xBf3//+E+2ESTNk3RG2n4/8EdAN7AL+HxqNnzEWn4k2+eBx92fb7XKRb9OPPLII9x+++289dZb3H777TzyyCPVTmk6OxfYJekZciPNdkTEI8A3gK9L6iHXZ9+R2ncAZ6b41zk5D5VZzfGJ3Dpy5513cvPNNzN//vxqpzKtRcQzwCVF4i+T698fHn8b+EIFUjMrmYt+Hent7R2ytvpUOL3ywTs+XcVMbDpy904daWhoGLI2MxvORb+OzJkzhxkzZjBnzpxqp2JmNcpFv07Mnj2bN998k8HBQd58801mz55d7ZTMrAa56NeJGTNmsGDBgiFrM7PhfCK3Trz99tu88sorDA4OnlibmQ3nw8E6kD9xmy/0+bVP6JrZcC76dWC0e+H6HrlmNpyLfh3J3xPX98Y1s9G46NcJSZx55plD1mZmw7no14n8/F7D12ZmhVz068TMmTNP3Aj92LFj7uIxs6JcGepEf3//ibH5AwMDHrJpZkX5SL8O5Iv98CGbvkDLzIabdFWQ9FFJ+wqW30r6mqTbJL1WEL++4Dm3SOqR9KKk68rzI9jg4CCSOOecc5gxYwbnnHMOkny0b2YjTLroR8SLEbEk3Uf0E8Dvga1p9135fRGxHUDSxcAq4GPAcuBv082nrQwuuugient7GRwcpLe3l4suuqjaKZlZDSrX//+vAV6KiF+O0WYl8GBEvBMRvwB6KHJDCpucAwcODBm5c+DAgSpnZGa1qFxFfxXQWfD4JknPSNosaW6KLQBeLWhzKMVGkLRWUpekrr6+vjKlaGZmJRd9SbOBzwD/N4XuBT4MLAGOAN/ONy3y9KKDySNiU0Q0R0TzvHnzSk3RzMySchzprwB+GhG9ABHRGxEDETEI3M/JLpxDwPkFzzsPOFyG97dk/vz5SPI9cs1sVOUo+qsp6NqRdG7Bvs8B+9P2NmCVpNMkXQgsBH5Shve3Ar4S18zGUtLFWZL+DXAt8JWC8J2SlpDrujmY3xcRz0n6PvA80A+sjwhPA1lGvjG6mY2npKIfEb8HzhwW++IY7duB9lLe08zMJs+XbJqZZYiLvplZhrjom5lliIu+mVmGuOjXkfzdsnzXLDMbjYu+mVmGuOjXEd8q0czG46I/TUk6sUyknU2cpPMl7ZLULek5SV9N8UZJOyQdSOu5KS5Jd6d7RTwj6dLq/gRmo3PRn6Yi4sSybNky4OSdsvLrZcuWnWhjp6QfuDkiFgGXA+vT/SA2AjsjYiGwMz2G3PxTC9OyltykgxXRtPHRE4vZRLjo14HHHnvsRIEHTvwheOyxx6qc2fQUEUci4qdp+3dAN7lpwFcCW1KzLcBn0/ZK4IHIeRI4Y9gcVGY1wzdGrxP5Au/bJJaXpCbgEuApYH5EHIHcHwZJZ6dmo90r4siw11pL7n8CXHDBBVOat9lofKRvNgpJfwA8BHwtIn47VtMisRF9ar5PhNUCF32zIiTNIlfwvxsR/5jCvflum7Q+muK+V4RNGy76ZsMoN9ypA+iOiL8s2LUNWJO21wAPF8RvTKN4LgfeyHcDmdUa9+mbjXQl8EXgWUn7UuzPgTuA70tqBV4BvpD2bQeuB3qA3wNfqmy6ZhPnom82TETspng/PcA1RdoHsH5KkzIrE3fvmJlliIu+mVmGuOibmWVIyUVf0kFJz0raJ6krxTxHiZlZDSrXkf7SiFgSEc3pcc3NUWJmZlPXveM5SszMalA5in4AP5S0J80tAsPmKAHGm6NkCElrJXVJ6urr6ytDimZmBuUZp39lRBxOk0/tkPTCGG0nPEcJsAmgubnZ8wKbmZVJyUf6EXE4rY8CW4HL8BwlZmY1qaSiL+kDkj6Y3waWAfvxHCVmZjWp1O6d+cDWdDu+mcDfR8Q/S3oaz1FiZlZzSir6EfEy8PEi8d/gOUrMzGqOr8g1M8sQF30zswzx1MpmdaJp46Mntg/e8ekqZmK1zEf6ZnWoaeOjQ/4ImOW56JuZZYiLvplZhrhP36yOuZ/fhvORvplZhrjom5lliIu+mVmGuOibmWWIi76ZWYa46JuZZYiLvplZhnicvlnGeSx/trjom2WEi7uBu3fMzDLFR/pmGeQZOLPLR/o1rrGxEUkTXoAJt21sbKzyT1ebJG2WdFTS/oJYo6Qdkg6k9dwUl6S7JfVIekbSpdXL3Gx8ky76ks6XtEtSt6TnJH01xW+T9JqkfWm5vuA5t6RfjhclXVeOH6DeHT9+nIiYkuX48ePV/vFq1XeA5cNiG4GdEbEQ2JkeA6wAFqZlLXBvhXI0m5RSunf6gZsj4qeSPgjskbQj7bsrIv5XYWNJFwOrgI8B/xb4f5I+EhEDJeRgVnYR8YSkpmHhlcBVaXsL8CPgGyn+QEQE8KSkMySdGxFHKpNteeW7fXyit35N+kg/Io5ExE/T9u+AbmDBGE9ZCTwYEe9ExC+AHuCyyb6/WYXNzxfytD47xRcArxa0O8QovweS1krqktTV19c3pcmajaYsffrpqOgS4KkUuin1b27O933iXw6rTyoSi2INI2JTRDRHRPO8efOmOC2z4kou+pL+AHgI+FpE/JZcn+aHgSXAEeDb+aZFnu5fDpsueiWdC5DWR1P8EHB+QbvzgMMVzs1swkoq+pJmkSv4342IfwSIiN6IGIiIQeB+Tnbh+JfDprNtwJq0vQZ4uCB+YxrFcznwxnTtz7dsKGX0joAOoDsi/rIgfm5Bs88B+WFv24BVkk6TdCG50Q4/mez7m00VSZ3Aj4GPSjokqRW4A7hW0gHg2vQYYDvwMrlzVPcDf1aFlM0mrJTRO1cCXwSelbQvxf4cWC1pCbmum4PAVwAi4jlJ3weeJzfyZ71H7lgtiojVo+y6pkjbANZPbUZm5TPpoh8RuyneT799jOe0A+2TfU8zMyuNr8g1M8sQF30zswzxhGtmNoKnYa5fPtI3M8sQF30zswxx0TczyxAXfTOzDPGJ3BoXt86B206futc2s0xx0a9x+ovfkrvocwpeWyJum5KXNrMa5e4dM7MMcdE3M8sQd++Y2Zh8oVZ98ZG+mVmGuOibmWWIi76ZWYa46JuZZYhP5E4DuTtTlt/cuXOn5HXNrHa56Ne4U70wS9KUXcxlZtOfu3fMbFKaNj46ZDinTQ8+0jezCXORn/4qXvQlLQf+GmgA/i4i7qh0DmZWPr54a3qpaPeOpAbgHmAFcDGwWtLFlczBzCzLKt2nfxnQExEvR8S7wIPAygrnUBckFV1G22dmBpUv+guAVwseH0qxISStldQlqauvr69iyU0nEXFKi5kZVL5Pv9gh54iKFBGbgE0Azc3Nrlhm04T792tfpY/0DwHnFzw+Dzhc4RzMzDKr0kf6TwMLJV0IvAasAm6ocA5mVgE+6q9NFS36EdEv6SbgMXJDNjdHxHOVzMHMLMsqPk4/IrYD2yv9vmZm5ityzcrGFx6OrtiVvO7yqQ4XfbMyKLjw8FpyAxaelrQtIp6vbma1a7Q+//GmevAfi9K46JuVx4kLDwEk5S88dNGfgFOZ02ey8/8U+8MyHf+AlJq7av3CHUl9wC+rncc0chbw62onMY38u4iYV+qLSPo8sDwi/mt6/EXgDyPipoI2a4G16eFHgReLvFStf37OrzSVym/U73XNH+mX4xcySyR1RURztfPIoHEvPCy86HDUF6nxz8/5laYW8vN8+mbl4QsPbVpw0TcrjxMXHkqaTe7Cw21VzslshJrv3rFTNmb3gU2NMl54WOufn/MrTdXzq/kTuWZmVj7u3jEzyxAXfTOzDHHRrxOSNks6Kml/tXOxyZG0XNKLknokbaxiHgclPStpn6SuFGuUtEPSgbSem+KSdHfK+RlJl05BPiO+25PJR9Ka1P6ApDVTmNttkl5L/377JF1fsO+WlNuLkq4riFfusz/VOzB5qc0F+BRwKbC/2rl4mdTn1wC8BHwImA38DLi4SrkcBM4aFrsT2Ji2NwLfStvXAz8gd53C5cBTU5DPiO/2qeYDNAIvp/XctD13inK7DfhvRdpenD7X04AL0+fdUOnP3kf6dSIingCOVTsPm7Rav3/0SmBL2t4CfLYg/kDkPAmcIenccr7xKN/tU83nOmBHRByLiOPADmD5FOU2mpXAgxHxTkT8Augh97lX9LN30TerDRO6f3SFBPBDSXvS1BEA8yPiCEBan53i1cr7VPOpdJ43pe6lzfmup1rJzUXfrDZM6P7RFXJlRFwKrADWS/rUGG1rKW8YPZ9K5nkv8GFgCXAE+HaK10JuLvpmNaJmpnGIiMNpfRTYSq77oTffbZPWR1PzauV9qvlULM+I6I2IgYgYBO4n9+9XE7mBi75ZraiJaRwkfUDSB/PbwDJgf8olP+JlDfBw2t4G3JhGzVwOvJHvdplip5rPY8AySXNTd8uyFCu7Yec0Pkfu3y+f2ypJp6X7hC8EfkKlP/upOkPspbIL0Enuv5LvkTtyaK12Tl5O+TO8Hvg5uZEcbVXK4UPkRo/8DHgunwdwJrATOJDWjSkucjePeQl4FmiegpxGfLcnkw/wJ+ROnvYAX5rC3P5Peu9nyBXvcwvat6XcXgRWVOOz9zQMZmYZ4u4dM7MMcdE3M8sQF30zswxx0TczyxAXfTOzDHHRNzPLEBd9M7MM+f8TmXUVrZGt4wAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 432x288 with 2 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"len_result = [len(s) for s in X_test]\n",
"print(\"줄거리 최대 길이 : \",max(len_result))\n",
"print(\"줄거리 평균 길이 : \",sum(len_result)/len(len_result))\n",
"\n",
"plt.subplot(1,2,1)\n",
"plt.boxplot(len_result)\n",
"plt.subplot(1,2,2)\n",
"plt.hist(len_result, bins=50)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## LSTM 분류 \n"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
"from tensorflow.keras.models import Sequential\n",
"from tensorflow.keras.layers import Dense, LSTM, Embedding\n",
"from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
"import numpy as np\n",
"\n",
"max_len = 600\n",
"X_train = pad_sequences(X_train, maxlen=max_len)\n",
"X_test = pad_sequences(X_train, maxlen=max_len)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Train on 2926 samples, validate on 2926 samples\n",
"Epoch 1/3\n",
"2880/2926 [============================>.] - ETA: 1s - loss: 0.6089 - acc: 0.6573\n",
"Epoch 00001: val_acc improved from -inf to 0.80554, saving model to best_model.h5\n",
"2926/2926 [==============================] - 108s 37ms/sample - loss: 0.6094 - acc: 0.6582 - val_loss: 0.6007 - val_acc: 0.8055\n",
"Epoch 2/3\n",
"2880/2926 [============================>.] - ETA: 1s - loss: 0.4706 - acc: 0.8271\n",
"Epoch 00002: val_acc improved from 0.80554 to 0.93233, saving model to best_model.h5\n",
"2926/2926 [==============================] - 103s 35ms/sample - loss: 0.4674 - acc: 0.8284 - val_loss: 0.1962 - val_acc: 0.9323\n",
"Epoch 3/3\n",
"2048/2926 [===================>..........] - ETA: 25s - loss: 0.1774 - acc: 0.9404"
]
}
],
"source": [
"model = Sequential()\n",
"model.add(Embedding(15002, 120))\n",
"model.add(LSTM(128))\n",
"model.add(Dense(1, activation='sigmoid'))\n",
"\n",
"es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=4)\n",
"mc = ModelCheckpoint('best_model.h5', monitor='val_acc', mode='max', verbose=1, save_best_only=True)\n",
"\n",
"X_train = np.array(X_train)\n",
"Y_train = np.array(Y_train)\n",
"X_test = np.array(X_test)\n",
"Y_test = np.array(Y_test)\n",
"\n",
"model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['acc'])\n",
"model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=3, batch_size=64, callbacks=[es, mc])\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"ename": "NameError",
"evalue": "name 'load_model' is not defined",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-22-d9b5bb19b28c>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mloaded_model\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mload_model\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'best_model.h5'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 2\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"\\n 테스트 정확도: %.4f\"\u001b[0m \u001b[1;33m%\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mloaded_model\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mevaluate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_test\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mNameError\u001b[0m: name 'load_model' is not defined"
]
}
],
"source": [
"loaded_model = load_model('best_model.h5')\n",
"print(\"\\n 테스트 정확도: %.4f\" % (loaded_model.evaluate(X_test, Y_test)[1]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using TensorFlow backend.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
"11493376/11490434 [==============================] - 6s 1us/step\n",
"WARNING:tensorflow:From /Users/yangyoonji/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Colocations handled automatically by placer.\n",
"WARNING:tensorflow:From /Users/yangyoonji/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use tf.cast instead.\n",
"Train on 18000 samples, validate on 42000 samples\n",
"Epoch 1/5\n",
"18000/18000 [==============================] - 3s 181us/step - loss: 1.1402 - acc: 0.7294 - val_loss: 0.6392 - val_acc: 0.8463\n",
"Epoch 2/5\n",
"18000/18000 [==============================] - 3s 170us/step - loss: 0.5055 - acc: 0.8734 - val_loss: 0.4687 - val_acc: 0.8739\n",
"Epoch 3/5\n",
"18000/18000 [==============================] - 3s 153us/step - loss: 0.4080 - acc: 0.8901 - val_loss: 0.4105 - val_acc: 0.8872\n",
"Epoch 4/5\n",
"18000/18000 [==============================] - 3s 148us/step - loss: 0.3652 - acc: 0.8979 - val_loss: 0.3805 - val_acc: 0.8939\n",
"Epoch 5/5\n",
"18000/18000 [==============================] - 3s 156us/step - loss: 0.3390 - acc: 0.9049 - val_loss: 0.3642 - val_acc: 0.8976\n",
"10000/10000 [==============================] - 0s 28us/step\n",
"\n",
"loss_and_metrics : [0.33921422773599624, 0.9048]\n",
"True : 7, Predict : 7\n",
"True : 1, Predict : 1\n",
"True : 1, Predict : 1\n",
"True : 7, Predict : 7\n",
"True : 9, Predict : 9\n"
]
}
],
"source": [
"# 영화 리뷰 예제\n",
"# 0. 사용할 패키지 불러오기\n",
"from keras.utils import np_utils\n",
"from keras.datasets import mnist\n",
"from keras.models import Sequential\n",
"from keras.layers import Dense, Activation\n",
"import numpy as np\n",
"from numpy import argmax\n",
"\n",
"# 1. 데이터셋 생성하기\n",
"\n",
"# 훈련셋과 시험셋 불러오기\n",
"(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
"\n",
"# 데이터셋 전처리\n",
"x_train = x_train.reshape(60000, 784).astype('float32') / 255.0\n",
"x_test = x_test.reshape(10000, 784).astype('float32') / 255.0\n",
"\n",
"# 원핫인코딩 (one-hot encoding) 처리\n",
"y_train = np_utils.to_categorical(y_train)\n",
"y_test = np_utils.to_categorical(y_test)\n",
"\n",
"# 훈련셋과 검증셋 분리\n",
"x_val = x_train[:42000] # 훈련셋의 30%를 검증셋으로 사용\n",
"x_train = x_train[42000:]\n",
"y_val = y_train[:42000] # 훈련셋의 30%를 검증셋으로 사용\n",
"y_train = y_train[42000:]\n",
"\n",
"# 2. 모델 구성하기\n",
"model = Sequential()\n",
"model.add(Dense(units=64, input_dim=28*28, activation='relu'))\n",
"model.add(Dense(units=10, activation='softmax'))\n",
"\n",
"# 3. 모델 학습과정 설정하기\n",
"model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])\n",
"\n",
"# 4. 모델 학습시키기\n",
"model.fit(x_train, y_train, epochs=5, batch_size=32, validation_data=(x_val, y_val))\n",
"\n",
"# 5. 모델 평가하기\n",
"loss_and_metrics = model.evaluate(x_test, y_test, batch_size=32)\n",
"print('')\n",
"print('loss_and_metrics : ' + str(loss_and_metrics))\n",
"\n",
"# 6. 모델 사용하기\n",
"xhat_idx = np.random.choice(x_test.shape[0], 5)\n",
"xhat = x_test[xhat_idx]\n",
"yhat = model.predict_classes(xhat)\n",
"\n",
"for i in range(5):\n",
" print('True : ' + str(argmax(y_test[xhat_idx[i]])) + ', Predict : ' + str(yhat[i]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
No preview for this file type
No preview for this file type