이동찬

add guideline about object detection

1 +# How to make your own object detector
2 +
3 +> ### with TF (TensorFlow) Object Detection API & GCP (Google Gloud Platform)
4 +
5 +EastFull: LEE DONGCHAN (huromech@gmail.com)
6 +
7 +
8 +
9 +## TF Object Detection API and GCP
10 +
11 +![1_1](./imgs/1_1.PNG)
12 +
13 +
14 +
15 + Object detection is the task of recognizing what class an objects is and where it is located. So, this may be more complex and difficult to implement than a simple classification model. [TensorFlow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection) uses pre-trained object detection model to make it easier to implement object detection functions. In addition, the model can be trained in the cloud using [Google Cloud Platform](https://cloud.google.com/gcp/?utm_source=google&utm_medium=cpc&utm_campaign=na-US-all-en-dr-skws-all-all-trial-b-dr-1008076&utm_content=text-ad-none-any-DEV_c-CRE_109860918967-ADGP_Hybrid+%7C+AW+SEM+%7C+SKWS+%7C+US+%7C+en+%7C+Multi+~+Cloud-KWID_43700009609890930-kwd-19383198255&utm_term=KW_%2Bcloud-ST_%2Bcloud&gclid=CjwKCAiA4Y7yBRB8EiwADV1haezYvLCzbz2c6lJvwoLzlRAcCk7YaVJemA3VasM2MR4axqgmG02MqRoCS48QAvD_BwE) without configuring the hardware environment for learning. Therefore, this article will talk about how to make your own object detector using TensorFlow Object Detection API and Google Cloud Platform :D
16 +
17 +
18 +
19 +## Tabel of Contents
20 +
21 +0. Setting up the Google Cloud Platform
22 +1. Installing TensorFlow and TensorFlow Object Detection API
23 +2. Making dataset
24 +3. Using the pre-trained SSD for Transfer Learning
25 +4. Training and Evaluation Jobs on Google Cloud ML Engine
26 +
27 +
28 +
29 +## 0. Setting up the Google Cloud Platform
30 +
31 + Before we begin, we will do the following to use Google Cloud ML engine and Storage.
32 +
33 +> 1. [Create a project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
34 +> 2. [Modify a project's billing settings](https://cloud.google.com/billing/docs/how-to/modify-project)
35 +> 3. [Install the Google Cloud SDK](https://cloud.google.com/sdk/install) to use Google Cloud Storage and ML Engine in your workplace (*In my case, I installed Ubuntu 18.04 LTS terminal in Windows 10. For the installaion of terminal, see [Terminal tutorial at fast.ai](https://course.fast.ai/terminal_tutorial.html) and the installation of Goolge Cloud CLI, see [here](https://course.fast.ai/start_gcp.html#step-2-install-google-cli) at course of fast.ai*)
36 +> 4. [Enable the ML Engine APIs](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component&_ga=2.43515109.-1978295503.1509743045)
37 +> 5. [Set up a Google Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) (*Or, you can use the following command in the Google Cloud CLI*:`gsutil mb gs://YOUR_UNIQUE_BUCKET_NAME`)
38 +
39 +
40 +
41 + Next, you need to add a TPU-specific service account to give the cloud TPU access to the project. Enter the following command to get the name of the account:
42 +
43 +```
44 +curl -H "Authorization: Bearer $(gcloud auth print-access-token) \
45 + https://ml.googleapis.com/v1/projects/YOUR_PROJECT_ID:getConfig
46 +```
47 +
48 +
49 +
50 +After command, copy the `tpuServiceAccount`(it looks like `service-account-12345@cloud-tpu.iam.gserviceaccount.com`) and save then as environment variable:
51 +
52 +```
53 +export TPU_ACCOUNT=your-service-account
54 +```
55 +
56 +
57 +
58 +And, give the role of ML serviceAgent to the TPU service account:
59 +
60 +```
61 +gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
62 + --member serviceAccount:$TPU_ACCOUNT --role roles/ml.serviceAgent
63 +```
64 +
65 +
66 +
67 +## 1. Installing TensorFlow and TensorFlow Object Detection API
68 +
69 + You can install the TensorFlow Object Detection API by following [this installaion](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md). Please refer to [here](https://www.tensorflow.org/install/) for more detailed procedures on TensorFlow. (*Notes: In my case, the version under tensorflow 1.13 and tensorflow 2.x did not work properly. I used the tensorflow 1.14.0 in a number of attempts :D - as of Feb, 2020*)
70 +
71 +
72 +
73 +## 2. Making dataset
74 +
75 +### 2.1. gathering data
76 +
77 +![1_2](./imgs/1_2.jpg)
78 +
79 + There is still one more important process left! We need data before we build our detection model. In my case, I have collected several data to find a badprint that frequently occurs in 3D printing. Collect data for your object detector :D
80 +
81 +
82 +
83 + ### 2.2. labeling data
84 +
85 +![1_3](./imgs/1_3.PNG)
86 +
87 + Now, we label the data that we have collected. I used [VoTT (Visual Object Tagging Tool)](https://github.com/microsoft/VoTT) to carry out labelling and saved it in the form of a `.csv` file.
88 +
89 +In fact, it doesn't matter which labelling tool you use. However, if yo want to follow this article to configure the dataset, the information (about `filename`, `xmin`, `ymin`, `xmax`, `ymax`, `class`) must be specified in the `.csv` file.
90 +
91 +
92 +
93 +### 2.3. generating TFRecords
94 +
95 + In order to train the object detector, we have to create [TFRecords](https://www.tensorflow.org/tutorials/load_data/tfrecord). To do this, we will use `generate_tfrecord.py` from [datitran's raccon_dataset](https://github.com/datitran/raccoon_dataset). Before make TFRecords we need to change a few lines in the `generate_tfrecord.py`.
96 +
97 +From:
98 +
99 +```python
100 +# TO-DO replace this with label map
101 +def class_text_to_int(row_label):
102 + if row_label == 'raccoon':
103 + return 1
104 + else:
105 + None
106 +```
107 +
108 +
109 +
110 +To:
111 +
112 +```python
113 +# TO-DO replace this with label map
114 +def class_text_to_int(row_label):
115 + if row_label == 'bad':
116 + return 1
117 + elif row_label == 'WHAT YOU WANT !':
118 + return 2
119 + #.
120 + #.
121 + #.
122 + else:
123 + None
124 +```
125 +
126 +If you are using a different dataset, you must replace the class name.
127 +
128 +
129 +
130 +Now our TFRecords can be generated by:
131 +
132 +```
133 +python generate_tfrecord.py \
134 + --csv_input=YOUR_PATH/TRAIN_CSV_NAME.csv \
135 + --output_path=YOUR_PATH/train.record \
136 + --image_dir=YOUR_PATH/IMAGE_DIR
137 +
138 +python generate_tfrecord.py \
139 + --csv_input=YOUR_PATH/VALID_CSV_NAME.csv \
140 + --output_path=YOUR_PATH/val.record \
141 + --image_dir=YOUR_PATH/IMAGE_DIR
142 +```
143 +
144 +
145 +
146 +Next, we will place the `.tfrecord` on our Google Cloud Storage bucket.
147 +
148 +```
149 +gsutil -m cp -r YOUR_PATH/train.record gs://YOUR_UNIQUE_BUCKET_NAME/data/
150 +gsutil -m cp -r YOUR_PATH/val.record gs://YOUR_UNIQUE_BUCKET_NAME/data/
151 +```
152 +
153 +
154 +
155 +### 2.4. creating a label map
156 +
157 + The label map maps an id to a name. This file is in `.pbtxt` format. In my case, I made it using the vim editor in terminal.
158 +
159 +```
160 +item {
161 + id: 1
162 + name: 'bad'
163 +}
164 +
165 +item {
166 + id: 2
167 + name: 'WHAT YOU WANT !'
168 +}
169 +
170 + .
171 + .
172 + .
173 +```
174 +
175 +
176 +
177 +After creating a label map, we will place the `.pbtxt` on our Google Cloud Storage bucket.
178 +
179 +```
180 +gsutil -m cp -r YOUR_PATH/LABEL_MAP.pbtxt gs://YOUR_UNIQUE_BUCKET_NAME/data/
181 +```
182 +
183 +
184 +
185 +## 3. Using the pre-trained SSD for Transfer Learning
186 +
187 +### 3.1. downloading model
188 +
189 + There are many pre-trained models we can use. We will use the checkpoints from these pre-trained models and apply to our task. What makes this possible is Transfer Learning. This is a strategy that enables pretty good performance with less time and data. Anyway, please check out the [Object Detection Models Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md).
190 +
191 +```
192 +wget http://storage.googleapis.com/download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03.tar.gz
193 +
194 +tar xzf ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03.tar.gz
195 +```
196 +
197 +So here we're going to use the ssd model based on the mobilenet. If you want a different model, try it! After downloading the model, upload the checkpoints of the model to Google Cloud Storage:
198 +
199 +```
200 +gsutil cp ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03/model.ckpt.* gs://YOUR_UNIQUE_BUCKET_NAME/data/
201 +```
202 +
203 +
204 +
205 +### 3.2. configuring the object detection pipeline
206 +
207 + In the TensorFlow Object Detection API, the model parameters, training and eval parameters are defined by a `.config` file. More details in [here](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/configuring_jobs.md). Anyway, we'll need to configure some paths in order for the template to work. It will be `PATH_TO_BE_CONFIGURED` in the code. Copy and move the config file corresponding to the model received for modification.
208 +
209 +```
210 +cp YOUR_PATH/models/research/object_detection/samples/configs/ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync.config .
211 +```
212 +
213 +
214 +
215 +First, modify the `PATH_TO_BE_CONFIGURED` of the copied file to our Google Cloud Storage bucket.
216 +
217 +```
218 +sed -i "s|PATH_TO_BE_CONFIGURED|"gs://YOUR_UNIQUE_BUCKET_NAME"/data|g" ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync.config
219 +```
220 +
221 +
222 +
223 +In addition, modify the other things (`input_path`, `output_path`,`batch_size`, `num_steps`, `total_steps`, and so on).
224 +
225 +From:
226 +
227 +```
228 + .
229 + .
230 +train_config: {
231 + fine_tune_checkpoint: "gs://YOUR_UNIQUE_BUCKET_NAME/data/model.ckpt"
232 + batch_size: 64 #
233 + sync_replicas: true
234 + startup_delay_steps: 0
235 + replicas_to_aggregate: 8
236 + num_steps: 25000 #
237 + data_augmentation_options {
238 + random_horizontal_flip {
239 + }
240 + }
241 + .
242 + .
243 +train_input_reader: {
244 + tf_record_input_reader {
245 + input_path: "gs://YOUR_UNIQUE_BUCKET_NAME/data/mscoco_train.record-00000-of-00100" #
246 + }
247 + label_map_path: "gs://YOUR_UNIQUE_BUCKET_NAME/data/mscoco_label_map.pbtxt" #
248 +}
249 + .
250 + .
251 +```
252 +
253 +
254 +
255 +To:
256 +
257 +```
258 + .
259 + .
260 +train_config: {
261 + fine_tune_checkpoint: "gs://YOUR_UNIQUE_BUCKET_NAME/data/model.ckpt"
262 + batch_size: 16 #
263 + sync_replicas: true
264 + startup_delay_steps: 0
265 + replicas_to_aggregate: 8
266 + num_steps: 2000 #
267 + data_augmentation_options {
268 + random_horizontal_flip {
269 + }
270 + }
271 + .
272 + .
273 +train_input_reader: {
274 + tf_record_input_reader {
275 + input_path: "gs://YOUR_UNIQUE_BUCKET_NAME/data/train.record" #
276 + }
277 + label_map_path: "gs://YOUR_UNIQUE_BUCKET_NAME/data/LABEL_MAP..pbtxt" #
278 +}
279 + .
280 + .
281 +```
282 +
283 +This is just an example. Therefore, you should modify the `.config` as you intended.
284 +
285 +
286 +
287 +Just like any other files, upload the `.config` to your Google Cloud Storage bucket.
288 +
289 +```
290 +gsutil cp ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync.config gs://YOUR_UNIQUE_BUCKET_NAME/data/pipeline.config
291 +```
292 +
293 +We're ready now! Let's train.
294 +
295 +
296 +
297 +## 4. Training and Evaluation Jobs on Google Cloud ML Engine
298 +
299 + Before we start our training job in Google Cloud ML Engine, we need to package the API, pycocotools and TF slim.
300 +
301 +```
302 +# From YOUR_PATH/models/research/
303 +bash object_detection/dataset_tools/create_pycocotools_package.sh /tmp/pycocotools
304 +python setup.py sdist
305 +(cd slim && python setup.py sdist)
306 +```
307 +
308 +
309 +
310 +And start training! Run following gcloud command:
311 +
312 +```
313 +gcloud ml-engine jobs submit training `whoami`_object_detection_`date +%s` \
314 + --job-dir=gs://YOUR_UNIQUE_BUCKET_NAME/train \
315 + --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz \
316 + --module-name object_detection.model_tpu_main \
317 + --runtime-version 1.14 \
318 + --scale-tier BASIC_TPU \
319 + --region us-central1 \
320 + -- \
321 + --model_dir=gs://YOUR_UNIQUE_BUCKET_NAME/train \
322 + --tpu_zone us-central1 \
323 + --pipeline_config_path=gs://YOUR_UNIQUE_BUCKET_NAME/data/pipeline.config
324 +```
325 +
326 +
327 +
328 +After we start our training , run following command:
329 +
330 +```
331 +gcloud ml-engine jobs submit training `whoami`_object_detection_eval_validation_`date +%s` \
332 + --job-dir=gs://YOUR_UNIQUE_BUCKET_NAME/train \
333 + --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz \
334 + --module-name object_detection.model_main \
335 + --runtime-version 1.14 \
336 + --scale-tier BASIC_GPU \
337 + --region us-central1 \
338 + -- \
339 + --model_dir=gs://YOUR_UNIQUE_BUCKET_NAME/train \
340 + --pipeline_config_path=gs://YOUR_UNIQUE_BUCKET_NAME/data/pipeline.config \
341 + --checkpoint_dir=gs://YOUR_UNIQUE_BUCKET_NAME/train
342 +```
343 +
344 +
345 +
346 +That's all! You can use TensorBoard to see the accuracy of your model.
347 +
348 +```
349 +# This command needs to be run once to allow your local machine to access your
350 +# GCS bucket.
351 +gcloud auth application-default login
352 +
353 +tensorboard --logdir=gs://YOUR_UNIQUE_BUCKET_NAME/model_dir
354 +```
355 +
356 +![1_4](./imgs/1_4.PNG)
357 +
358 +
359 +