2020-2-capstone-design2 / 2016103262

Go to a project
Toggle navigation Toggle navigation pinning
heeseon cheon
9c9844b6 1 parent 2564644d

fix run.sh and add readme

Inline Side-by-side
Showing 2 changed files with 83 additions and 20 deletions
1 +## 상관계수 정규화와 동적 필터 가지치기를 이용한 심층 신경망 압축
2 +Dynamic Filter Pruning with Decorrelation Regularization for Compression of Deep Neural Network
3 +> 2020 한국소프트웨어종합학술대회 (KSC2020) 학부생논문 경진대회 출품작
4 +
5 +### Prerequisites
6 +
7 +* Ubuntu 18.04
8 +* Python 3.7.4
9 +* Pytorch 1.6.0
10 +* numpy 1.18.1
11 +* GPU (cuda)
12 +
13 +### Build
14 +
15 +```
16 +$ python modeling.py --prune_type structured --prune_rate 0.6 --reg reg_cov --odecay 0.7 > result.txt
17 +```
18 +* `run.sh`에서 parameter 조절 후 `./run.sh`로 진행
19 +
20 +### Process
21 +**0. Data, Model & Parameters**
22 +- Data : CIFAR-10
23 +- Model : ResNet-50
24 +- Optimizer : Stochastic Gradient Descent
25 +- Learning Rate : 0.2
26 +- Epoch : 300
27 +- Batch size : 128
28 +- Loss Function : Cross Entropy
29 +- Metric : Accuracy, Sparsity
30 +
31 +**1. 동적 필터 가지치기 (Dynamic Filter Pruning)**
32 +
33 +L1 norm 크기를 기반으로 필터 마스크를 생성하여 가중치 학습 시 반영
34 +- 필터 마스크 : ![image](https://user-images.githubusercontent.com/41580746/102396051-41616d80-401f-11eb-9738-7b5df9aee0d4.png)
35 + - i : 층 위치
36 + - j : 필터 위치
37 + - t : epoch 값
38 + - W : 필터 가중치 행렬
39 + - η : 임계값 (전체 필터 개수 중 가지치기 필터 개수 비율 통해 계산)
40 +- 가중치 학습 : ![image](https://user-images.githubusercontent.com/41580746/102396101-51794d00-401f-11eb-9303-99dd712798ee.png)
41 + - g : 기울기
42 + - γ : learning rate
43 +
44 +**2. 상관계수 정규화 (Decorrleation Regularization)**
45 +
46 +기존 loss function에 상관계수 정규화 식을 더하여 최종 손실 함수 계산
47 +- loss function : ![image](https://user-images.githubusercontent.com/41580746/102396482-d82e2a00-401f-11eb-93b3-7db5fea8f8af.png)
48 + - α : 정규화 상수
49 + - ![image](https://user-images.githubusercontent.com/41580746/102396603-01e75100-4020-11eb-933e-f85305dd874d.png)
50 +
51 +### Result
52 +가지치기 비율 60%, 정규화 상수 0.7일 때의 모델별 Accuracy 및 Sparsity 비교 결과
53 +- ![image](https://user-images.githubusercontent.com/41580746/102395542-a072b280-401e-11eb-9e47-c3b52d859479.png)
54 +- ![image](https://user-images.githubusercontent.com/41580746/102396706-28a58780-4020-11eb-9ebb-6cc723b4fcbf.png)
55 +- 기존 동적 필터 가지치기 대비 Accuracy 1.47%, Sparsity 1.08% 증가
56 +
57 +### References
58 +- [1] Yann LeCun, Yoshua Bengio, Geoffrey Hinton. Deep learning. Nature 521, 436-444, 2015.
59 +- [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jain Sun. Deep Residual Learning for Image Recognition. 2015.
60 +- [3] 조인천, 배성호. 동적 필터 프루닝 기법을 이용한 심층 신경망 압축. 한국방송미디어공학회 하계학술대회, 2020.
61 +- [4] Benoit Jacob. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. 2017.
62 +- [5] Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, Martin Jaggi. Dynamic Model Pruning with Feedback. ICLR, 2020.
63 +- [6] Namhoon Lee, Thalaiyasingam Ajanthan, Philip HS Torr, SNIP: Single-shot network pruningbased on connection sensitivity. ICLR, 2019.
64 +- [7] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf. Pruning Filters For Effiecient ConvNets. ICLR, 2017.
65 +- [8] Jian-Hao Luo, Jianxin Wu, Weiyao Lin. ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression. ICCV, 2017.
66 +- [9] Song Han, Huizi Mao, William J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. ICLR, 2016.
67 +- [10] Xiaotian Zhu, Wengang Zhou, Houqiang Li. Improving Deep Neural Network Sparsity through Decorrelation Regularization. IJCAI, 2018.
1 #!/bin/bash 1 #!/bin/bash
2 -RESULT_DIR=result_201203 2 +RESULT_DIR=result
3 3
4 if [ ! -d $RESULT_DIR ]; then 4 if [ ! -d $RESULT_DIR ]; then
5 mkdir $RESULT_DIR 5 mkdir $RESULT_DIR
6 fi 6 fi
7 7
8 -#python modeling_default.py > $RESULT_DIR/default.txt #&
9 -#python modeling_pruning.py > $RESULT_DIR/pruning_prune90.txt &
10 -#python modeling_decorrelation.py > $RESULT_DIR/decorrelation_lambda1.txt &
11 -#python modeling_pruning+decorrelation.py > $RESULT_DIR/pruning+decorrelation_lambda1+prune90.txt
12 8
13 -#python modeling.py --prune_type structured --prune_rate 0.5 > $RESULT_DIR/prune_05.txt 9 +#python modeling.py --prune_type structured --prune_rate 0.5 > $RESULT_DIR/prune_50.txt
14 -#python modeling.py --prune_type structured --prune_rate 0.6 > $RESULT_DIR/prune_06.txt 10 +#python modeling.py --prune_type structured --prune_rate 0.6 > $RESULT_DIR/prune_60.txt
15 -#python modeling.py --prune_type structured --prune_rate 0.8 > $RESULT_DIR/prune_08.txt & 11 +#python modeling.py --prune_type structured --prune_rate 0.7 > $RESULT_DIR/prune_70.txt
16 -#python modeling.py --prune_type structured --prune_rate 0.7 > $RESULT_DIR/prune_07.txt 12 +#python modeling.py --prune_type structured --prune_rate 0.8 > $RESULT_DIR/prune_80.txt
17 13
18 -#python modeling.py --reg reg_cov --odecay 0.9 > $RESULT_DIR/reg_9.txt 14 +#python modeling.py --reg reg_cov --odecay 0.5 > $RESULT_DIR/reg_05.txt
19 -#python modeling.py --reg reg_cov --odecay 0.8 > $RESULT_DIR/reg_8.txt 15 +#python modeling.py --reg reg_cov --odecay 0.6 > $RESULT_DIR/reg_06.txt
20 -#python modeling.py --reg reg_cov --odecay 0.7 > $RESULT_DIR/reg_7.txt 16 +#python modeling.py --reg reg_cov --odecay 0.7 > $RESULT_DIR/reg_07.txt
21 -#python modeling.py --reg reg_cov --odecay 0.6 > $RESULT_DIR/reg_6.txt 17 +#python modeling.py --reg reg_cov --odecay 0.8 > $RESULT_DIR/reg_08.txt
22 -#python modeling.py --reg reg_cov --odecay 0.5 > $RESULT_DIR/reg_5.txt 18 +#python modeling.py --reg reg_cov --odecay 0.9 > $RESULT_DIR/reg_09.txt
23 19
24 -#python modeling.py --prune_type structured --prune_rate 0.5 --reg reg_cov --odecay 0.7 > $RESULT_DIR/prune_05_reg_07.txt &
25 -#python modeling.py --prune_type structured --prune_rate 0.5 --reg reg_cov --odecay 0.8 > $RESULT_DIR/prune_05_reg_08.txt &
26 -#python modeling.py --prune_type structured --prune_rate 0.5 --reg reg_cov --odecay 0.9 > $RESULT_DIR/prune_05_reg_09.txt &
27 -#python modeling.py --prune_type structured --prune_rate 0.6 --reg reg_cov --odecay 0.7 > $RESULT_DIR/prune_06_reg_07.txt &
28 -#python modeling.py --prune_type structured --prune_rate 0.6 --reg reg_cov --odecay 0.8 > $RESULT_DIR/prune_06_reg_08.txt &
29 -python modeling.py --prune_type structured --prune_rate 0.6 --reg reg_cov --odecay 0.9 > $RESULT_DIR/prune_06_reg_09.txt
30 20
21 +#python modeling.py --prune_type structured --prune_rate 0.5 --reg reg_cov --odecay 0.7 > $RESULT_DIR/prune_50_reg_07.txt
22 +#python modeling.py --prune_type structured --prune_rate 0.5 --reg reg_cov --odecay 0.8 > $RESULT_DIR/prune_50_reg_08.txt
23 +#python modeling.py --prune_type structured --prune_rate 0.5 --reg reg_cov --odecay 0.9 > $RESULT_DIR/prune_50_reg_09.txt
24 +python modeling.py --prune_type structured --prune_rate 0.6 --reg reg_cov --odecay 0.7 > $RESULT_DIR/prune_60_reg_07.txt
25 +#python modeling.py --prune_type structured --prune_rate 0.6 --reg reg_cov --odecay 0.8 > $RESULT_DIR/prune_60_reg_08.txt
26 +#python modeling.py --prune_type structured --prune_rate 0.6 --reg reg_cov --odecay 0.9 > $RESULT_DIR/prune_60_reg_09.txt
......