License Plate Detection Application

Overview

LicensePlate_Project ๐Ÿš— ๐Ÿš™

[Project] 2021.02 ~ 2021.09 License Plate Detection Application

Overview


1. ๋ฐ์ดํ„ฐ ์ˆ˜์ง‘ ๋ฐ ๋ผ๋ฒจ๋ง

์ฐจ๋Ÿ‰ ๋ฒˆํ˜ธํŒ ์ด๋ฏธ์ง€๋ฅผ ์ง์ ‘ ์ˆ˜์ง‘ํ•˜์—ฌ ๊ฐ ์ด๋ฏธ์ง€์— ๋Œ€ํ•ด '๋ฒˆํ˜ธํŒ ๊ธ€์ž'์™€ '๋ฒˆํ˜ธํŒ ๋„ค ๊ผญ์ง“์ ์˜ x,y ์ขŒํ‘œ'๋ฅผ ๋ผ๋ฒจ๋ง ํ•œ๋‹ค.

๋ฒˆํ˜ธํŒ ์ด๋ฏธ์ง€
๋ผ๋ฒจ๋ง 20210210_222919.jpg 1481 2773 2043 2689 2043 2794 1486 2883 36์กฐ 2428

ํ…์ŠคํŠธ ํŒŒ์ผ๋กœ ์ €์žฅ๋œ ๋ผ๋ฒจ๋ง ์ •๋ณด๋Š” ๋ฒˆํ˜ธํŒ ๋„ค ๊ผญ์ง“์ ์˜ ์ ˆ๋Œ€ ์ขŒํ‘œ์™€ ๋ฒˆํ˜ธํŒ ๊ธ€์ž๋ฅผ ํฌํ•จํ•˜๊ณ  ์žˆ๋‹ค. ํ•™์Šต ๋ฐ์ดํ„ฐ์˜ 20%๋ฅผ ๊ฒ€์ฆ ๋ฐ์ดํ„ฐ๋กœ ๋‚˜๋ˆ„์–ด ๋ฐ์ดํ„ฐ์…‹ ์ค€๋น„๋ฅผ ๋งˆ์นœ๋‹ค. ์ตœ์ข… ๋ฐ์ดํ„ฐ์…‹ ๊ตฌ์„ฑ์€ ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

ํ•™์Šต ๋ฐ์ดํ„ฐ ๊ฒ€์ฆ ๋ฐ์ดํ„ฐ
1635์žฅ 409์žฅ

2. YOLOv5 ํ•™์Šต (Pytorch-YOLOv5)

  • ์ฐธ๊ณ : https://github.com/ultralytics/yolov5

  • ์ธํ’‹ ๋ฐ์ดํ„ฐ ์ค€๋น„
    ์›๋ณธ ์ด๋ฏธ์ง€๋Š” ๋ฒˆํ˜ธํŒ ์˜์—ญ์„ ํƒ์ง€ํ•˜๊ธฐ ์œ„ํ•ด ๊ณง์žฅ YOLO์˜ ์ž…๋ ฅ์œผ๋กœ ์‚ฌ์šฉ๋˜๊ธฐ ๋•Œ๋ฌธ์—, YOLO์˜ ์ž…๋ ฅ ํ˜•์‹์— ๋งž์ถ”๊ธฐ ์œ„ํ•ด ๊ฐ ์ด๋ฏธ์ง€ ๋งˆ๋‹ค ์ด๋ฏธ์ง€ ํŒŒ์ผ๋ช…๊ณผ ๋™์ผํ•œ ์ด๋ฆ„์˜ ํ…์ŠคํŠธ ํŒŒ์ผ์„ ๋งŒ๋“ค์–ด bounding box์˜ ์ขŒํ‘œ ์ •๋ณด๋ฅผ class, x_center, y_center, width, height์˜ ํฌ๋งท์˜ ๋ฌธ์ž์—ด๋กœ ์ €์žฅํ•œ๋‹ค. ์ด ๋•Œ, class๋ฅผ ์ œ์™ธํ•œ ๋‚˜๋จธ์ง€ ๊ฐ’์€ ๋ชจ๋‘ 0-1 ์‚ฌ์ด์˜ ์ƒ๋Œ€ ์ขŒํ‘œ๋กœ ๋ณ€ํ™˜ํ•œ๋‹ค.

โ”œโ”€โ”€ Yolo_input
    โ”œโ”€โ”€ train
    โ”‚   โ”œโ”€โ”€ images
    โ”‚   โ”‚   โ”œโ”€โ”€ 1.jpg
    โ”‚ 	โ”‚   โ”œโ”€โ”€ 2.jpg
    โ”‚ 	โ”‚  	โ”‚     :
    โ”‚ 	โ”‚  		  
    โ”‚   โ”œโ”€โ”€ labels
    โ”‚	    โ”œโ”€โ”€ 1.txt
    โ”‚	    โ”œโ”€โ”€ 2.txt
    โ”‚	   	โ”‚     :
    โ”‚	
    โ””โ”€โ”€ val
 	    โ”œโ”€โ”€ images
 	    โ”œโ”€โ”€ labels
  • dataset.yaml ์ค€๋น„
    Custom ๋ฐ์ดํ„ฐ์…‹์— YOLOv5 ํ•™์Šต ์ฝ”๋“œ๋ฅผ ๊ทธ๋Œ€๋กœ ์“ธ ๊ฒƒ์ด๊ธฐ ๋•Œ๋ฌธ์—, ๋ฐ์ดํ„ฐ์…‹ ์„ธํŒ… ๋ถ€๋ถ„๋งŒ ์ˆ˜์ •ํ•œ๋‹ค. dataset.yaml ํŒŒ์ผ์— ํ•™์Šต, ๊ฒ€์ฆ ๋ฐ์ดํ„ฐ ๊ฒฝ๋กœ์™€ ๊ฐ์ฒด ํด๋ž˜์Šค ์ •๋ณด๋ฅผ ๊ธฐ์ž…ํ•œ๋‹ค. ์šฐ๋ฆฌ ํ”„๋กœ์ ํŠธ์˜ ๊ฒฝ์šฐ ํƒ์ง€ํ•˜๋Š” ๊ฐ์ฒด๊ฐ€ ์ฐจ๋Ÿ‰ ๋ฒˆํ˜ธํŒ ํ•˜๋‚˜์ด๋ฏ€๋กœ ํด๋ž˜์Šค ๋ผ๋ฒจ์„ 0์œผ๋กœ, ์ด๋ฆ„์„ 'plate' ๋กœ ํ•œ๋‹ค.

  • YOLO ๋ชจ๋ธ ์„ ํƒ
    ๋ณธ ํ”„๋กœ์ ํŠธ๋ฅผ ์œ„ํ•ด ๊ฐ€์žฅ ์ž‘๊ณ  ๋น ๋ฅธ ๋ชจ๋ธ์ธ YOLOv5s๋ฅผ ์‚ฌ์šฉํ•˜์˜€๋‹ค.


3. ๊ผญ์ง“์  ์˜ˆ์ธก ๋ชจ๋ธ ํ•™์Šต

  • ์‚ฌ์šฉํ•œ ๋ชจ๋ธ : timm์œผ๋กœ ์‚ฌ์ „ํ•™์Šต๋œ Resnet18 ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜์˜€๋‹ค

  • ์ฒซ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•

    1. ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ : ๋„ค ๊ผญ์ง“์  ์ขŒํ‘œ๊ฐ’์„ ์ด์šฉํ•˜์—ฌ ๋งŒ๋“  ๋ฐ”์šด๋”ฉ ๋ฐ•์Šค์—์„œ ๊ฐ ์ถ•์œผ๋กœ 1%์”ฉ ๋Š˜์ธ ์ด๋ฏธ์ง€

    2. ๋ฐ์ดํ„ฐ ์ฆ๊ฐ•: ์ „๋‹จ ๋ณ€ํ™˜(shear transformation), ์‚ฌ์ง„ํ•ฉ์„ฑ, ๋ฐ๊ธฐ์กฐ์ ˆ, ๋ฆฌ์‚ฌ์ด์ฆˆ
      ์ž…๋ ฅ ์ด๋ฏธ์ง€๋ฅผ ์ „๋‹จ ๋ณ€ํ™˜ ๊ธฐ๋ฒ•์„ ์ด์šฉํ•ด x, y์ถ•์œผ๋กœ ๋žœ๋คํ•˜๊ฒŒ ๋ณ€ํ™˜ํ•˜๋ฉด ๊ฒ€์€์ƒ‰ ์—ฌ๋ฐฑ ๋ถ€๋ถ„์ด ์ƒ๊ฒจ, ์ด ๋ถ€๋ถ„์„ ๋‹ค๋ฅธ ์ด๋ฏธ์ง€์—์„œ ๋žœ๋คํ•˜๊ฒŒ ๊ฐ€์ ธ์™€ ํ•ฉ์„ฑ์‹œ์ผฐ๋‹ค. ์ด ์ด๋ฏธ์ง€์— ๋žœ๋ค์œผ๋กœ ๋ฐ๊ธฐ์กฐ์ ˆ์„ ์ถ”๊ฐ€ํ•˜์—ฌ, 128x128 ์ด๋ฏธ์ง€๋กœ ๋ฆฌ์‚ฌ์ด์ฆˆํ•œ ์ด๋ฏธ์ง€๋ฅผ ๋ชจ๋ธ์— ์ž…๋ ฅ์œผ๋กœ ๋„ฃ์—ˆ๋‹ค.

    3. ๋ฌธ์ œ์  : ๊ฒ€์€์ƒ‰ ๋ถ€๋ถ„์„ ๋‹ค๋ฅธ ์‚ฌ์ง„์œผ๋กœ ํ•ฉ์„ฑ์‹œ์ผฐ๋”๋‹ˆ ์‹ค์„ธ๊ณ„ ๋ฐ์ดํ„ฐ์™€ ๊ดด๋ฆฌ๊ฐ์ด ์ƒ๊ฒจ ์„ฑ๋Šฅ ์ €ํ•˜ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•˜์˜€๋‹ค.

    ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ ๋ฐ์ดํ„ฐ์ฆ๊ฐ•1 ๋ฐ์ดํ„ฐ์ฆ๊ฐ•2
    ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ ๋ฐ์ดํ„ฐ์ฆ๊ฐ•1 ๋ฐ์ดํ„ฐ์ฆ๊ฐ•2
  • ๋‘ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•

    1. ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ : ์›๋ณธ ์ด๋ฏธ์ง€

    2. ๋ฐ์ดํ„ฐ ์ฆ๊ฐ•: ์ „๋‹จ ๋ณ€ํ™˜, ๋ฐ๊ธฐ์กฐ์ ˆ, ๋ฆฌ์‚ฌ์ด์ฆˆ ์ž…๋ ฅ ์ด๋ฏธ์ง€์™€ ๋ผ๋ฒจ๋ง์„ ํ†ตํ•ด ์•Œ๋ ค์ง„ ๋ฒˆํ˜ธํŒ ๊ผญ์ง“์ ์˜ ์ขŒํ‘œ๋“ค์„ ์ „๋‹จ ๋ณ€ํ™˜ ๊ธฐ๋ฒ•์„ ์ด์šฉํ•ด ๋žœ๋ค ๊ฐ’์œผ๋กœ ๋ณ€ํ™˜ํ•œ๋‹ค. ์ด ์ด๋ฏธ์ง€์—์„œ ๋ฒˆํ˜ธํŒ์˜ ์ขŒํ‘œ๋ฅผ ๊ธฐ์ค€์œผ๋กœ margin์„ ์ฃผ๊ณ , ๊ทธ ์ง€์ ์œผ๋กœ๋ถ€ํ„ฐ ๋žœ๋คํ•˜๊ฒŒ ์ขŒํ‘œ๋ฅผ ์ฐ์–ด ์ด๋ฏธ์ง€๋ฅผ ์ž๋ฅธ ๊ฒƒ์„ ์‚ฌ์šฉ. ์ด๋ ‡๊ฒŒ ํ•˜๋ฉด ์ฒซ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•์—์„œ ๋‚˜์™”๋˜ ๊ฒ€์€ ์—ฌ๋ฐฑ ๋ถ€๋ถ„์ด ๋‚˜์˜ค์ง€ ์•Š์œผ๋ฏ€๋กœ ์‹ค์„ธ๊ณ„ ๋ฐ์ดํ„ฐ์™€ ๋” ๊ทผ์ ‘ํ•˜๋‹ค. ์ด ์ด๋ฏธ์ง€์— ๋žœ๋ค์œผ๋กœ ๋ฐ๊ธฐ์กฐ์ ˆ์„ ์ถ”๊ฐ€ํ•˜์—ฌ, 128x128 ์ด๋ฏธ์ง€๋กœ ๋ฆฌ์‚ฌ์ด์ฆˆํ•œ ์ด๋ฏธ์ง€๋ฅผ ๋ชจ๋ธ์— ์ž…๋ ฅ์œผ๋กœ ๋„ฃ์—ˆ๋‹ค.

    ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ ๋ฐ์ดํ„ฐ์ฆ๊ฐ•1 ๋ฐ์ดํ„ฐ์ฆ๊ฐ•2
    ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ ๋ฐ์ดํ„ฐ์ฆ๊ฐ•1 ๋ฐ์ดํ„ฐ์ฆ๊ฐ•2
  • Output : ์ƒํ•˜์ขŒ์šฐ ๋„ค ๊ผญ์ง“์ ์— ๋Œ€ํ•œ X,Y ์ƒ๋Œ€์ขŒํ‘œ


4. ๊ธ€์ž ์˜ˆ์ธก ๋ชจ๋ธ ํ•™์Šต

  • ์‚ฌ์šฉํ•œ ๋ชจ๋ธ : timm์œผ๋กœ ์‚ฌ์ „ํ•™์Šต๋œ Resnet18 ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜์˜€๋‹ค.

  • ์ฒซ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•

    1. ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ : ์›๋ณธ ์ด๋ฏธ์ง€์˜ ๋„ค ๊ผญ์ง“์  ์ขŒํ‘œ์— ๋Œ€ํ•œ ground truth๋ฅผ ์ด์šฉํ•˜์—ฌ (128, 256)์˜ ํฌ๊ธฐ๋กœ ํˆฌ์˜๋ณ€ํ™˜ํ•œ ์ด๋ฏธ์ง€

    2. ๋ฐ์ดํ„ฐ ์ฆ๊ฐ•: Salt & Pepper ๋…ธ์ด์ฆˆ ์‹ค์ œ ์ฐจ๋Ÿ‰์˜ ๋ฒˆํ˜ธํŒ์€ ๋จผ์ง€ ๋ฐ ๋ฒŒ๋ ˆ์™€ ๊ฐ™์€ ์ด๋ฌผ์งˆ ๋•Œ๋ฌธ์— ์–ผ๋ฃฉ๋œ๋ฃฉํ•œ ๊ฒฝ์šฐ๊ฐ€ ๋งŽ๋‹ค. ๋”ฐ๋ผ์„œ ์ž…๋ ฅ ๋ฐ์ดํ„ฐ์— ๋žœ๋คํ•œ ๋…ธ์ด์ฆˆ๋ฅผ ์ถ”๊ฐ€ํ•˜์—ฌ ์ผ๋ฐ˜์ ์ธ ์ƒํ™ฉ๊นŒ์ง€ ์ปค๋ฒ„ํ•  ์ˆ˜ ์žˆ๋„๋ก ํ•˜์˜€๋‹ค.

    3. ๋ฌธ์ œ์  : ์‹ค์ œ ์ถ”๋ก  ๊ณผ์ •์—์„œ๋Š” ๊ผญ์ง“์  ์˜ˆ์ธก ๋ชจ๋ธ๋กœ๋ถ€ํ„ฐ ์˜ˆ์ธก๋œ ๊ผญ์ง“์  ๊ฐ’์„ ๊ธฐ๋ฐ˜์œผ๋กœ ์ •๋ ฌ๋œ ๋ฒˆํ˜ธํŒ ์ด๋ฏธ์ง€๊ฐ€ ์ž…๋ ฅ์œผ๋กœ ์‚ฌ์šฉ๋˜๋ฏ€๋กœ, ๊ธ€์ž ์˜ˆ์ธก ๋ชจ๋ธ์˜ ์„ฑ๋Šฅ์ด ๊ผญ์ง“์  ์˜ˆ์ธก ๋ชจ๋ธ์˜ ์„ฑ๋Šฅ์— ํฐ ์˜ํ–ฅ์„ ๋ฐ›์Œ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์—ˆ๋‹ค.

  • ๋‘ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•

    1. ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ : ์›๋ณธ ์ด๋ฏธ์ง€์˜ ๋„ค ๊ผญ์ง“์  ์ขŒํ‘œ๋ฅผ x,y ๋ฐฉํ–ฅ์œผ๋กœ ๊ฐ๊ฐ ๋žœ๋คํ•˜๊ฒŒ ์ด๋™์‹œํ‚จ ํ›„ (128, 256)์˜ ํฌ๊ธฐ๋กœ ํˆฌ์˜๋ณ€ํ™˜ํ•œ ์ด๋ฏธ์ง€

    2. ๋ฐ์ดํ„ฐ ์ฆ๊ฐ•: Salt & Pepper ๋…ธ์ด์ฆˆ, ๋ฐ๊ธฐ ์กฐ์ ˆ(์ „์ฒด ๋ฐ๊ฒŒ, ์ „์ฒด ์–ด๋‘ก๊ฒŒ, ๊ทธ๋ฆผ์ž) ์ˆ˜์ง‘๋œ ๋ฐ์ดํ„ฐ์…‹์€ ๋Œ€๋ถ€๋ถ„ ๋‚ฎ์— ์ฐ์€ ๋ฒˆํ˜ธํŒ ์ด๋ฏธ์ง€์˜€๊ธฐ ๋•Œ๋ฌธ์—, ํ…Œ์ŠคํŠธ ๋ฆฌํฌํŒ… ์‹œ ์•ผ๊ฐ„ ์ด๋ฏธ์ง€์— ๋Œ€ํ•ด์„œ๋Š” ์„ฑ๋Šฅ์ด ๋‚ฎ์•„์ง€๋Š” ๊ฒƒ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์—ˆ๋‹ค. ๋”ฐ๋ผ์„œ ๋ฐ๊ธฐ ์กฐ์ ˆ ๋ฐ ๊ทธ๋ฆผ์ž ์ถ”๊ฐ€ ์ฆ๊ฐ• ๊ธฐ๋ฒ•์„ ์ถ”๊ฐ€ํ•˜์—ฌ ์—ฌ๋Ÿฌ ํ™˜๊ฒฝ์˜ ์ด๋ฏธ์ง€์— ๋Œ€ํ•ด ๊ฐ•๊ฑดํ•œ ์„ฑ๋Šฅ์„ ๋ณด์ด๋„๋ก ํ•˜์˜€๋‹ค.

    ์ถ”๋ก  ์‹œ ์‹ค์ œ ์ž…๋ ฅ๋˜๋Š” ์ด๋ฏธ์ง€ ์ฒซ ๋ฒˆ์งธ ๋ฐฉ๋ฒ• ๋‘ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•

    ๋ฐ์ดํ„ฐ ์ฆ๊ฐ•์˜ ์˜ˆ์‹œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

    ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ ๋ฐ์ดํ„ฐ์ฆ๊ฐ•1 ๋ฐ์ดํ„ฐ์ฆ๊ฐ•2
  • Output : (๋ฐฐ์น˜์‚ฌ์ด์ฆˆ, 7, 45, 1) ๋ชจ์–‘์˜ ํ…์„œ
    7 -> 7๊ธ€์ž 45 -> 45๊ฐœ์˜ ๊ฐ€๋Šฅํ•œ ๋ฌธ์ž (['๊ฐ€', '๋‚˜', '๋‹ค', '๋ผ', '๋งˆ', '๊ฑฐ', '๋„ˆ', '๋”', '๋Ÿฌ', '๋จธ', '๋ฒ„', '์„œ', '์–ด', '์ €', '๊ณ ', '๋…ธ', '๋„', '๋กœ', '๋ชจ', '๋ณด', '์†Œ', '์˜ค', '์กฐ', '๊ตฌ', '๋ˆ„', '๋‘', '๋ฃจ', '๋ฌด', '๋ถ€', '์ˆ˜', '์šฐ', '์ฃผ', 'ํ—ˆ', 'ํ•˜', 'ํ˜ธ', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9'])


5. pt >> onnx >> pb >> tflite ๋ณ€ํ™˜

  • YOLOv5
    ์ œ๊ณตํ•ด์ฃผ๋Š” export.py๋ฅผ ์‚ฌ์šฉํ•ด TensorFlow Lite ํŒŒ์ผ๋กœ ๋ณ€ํ™˜ํ•œ๋‹ค. ์ด ๋•Œ, Non Max Suppression ๋ถ€๋ถ„์€ TensorFlow Lite๋กœ ๋ณ€ํ™˜๋˜์ง€ ์•Š์•„ ์•ˆ๋“œ๋กœ์ด๋“œ ์ŠคํŠœ๋””์˜ค ์ฝ”๋“œ๋ฅผ ์งค ๋•Œ ๋”ฐ๋กœ ์ถ”๊ฐ€ํ•˜์˜€๋‹ค. YOLO์˜ ์ถœ๋ ฅ์œผ๋กœ ๋‚˜์˜ค๋Š” (1, 3024, 6)์˜ ํ…์„œ๋Š” 3024๊ฐœ์˜ ๊ฐ€๋Šฅํ•œ ๋ฐ”์šด๋”ฉ ๋ฐ•์Šค์™€, ๊ฐ ๋ฐ”์šด๋”ฉ ๋ฐ•์Šค์˜ x_center, y_center, width, height, confidence, ๊ฐ์ฒด ํด๋ž˜์Šค ์ •๋ณด๋ฅผ ํฌํ•จํ•˜๊ณ  ์žˆ๋‹ค. ์•„๋ž˜ ์ฝ”๋“œ๋Š” ๊ฐ€๋Šฅํ•œ 3024๊ฐœ์˜ ๋ฐ”์šด๋”ฉ ๋ฐ•์Šค ์ค‘ ๊ฐ€์žฅ ํฐ confidence ๊ฐ’์„ ๊ฐ€์ง€๋Š” ํ•˜๋‚˜์˜ ๋ฐ”์šด๋”ฉ ๋ฐ•์Šค๋งŒ์„ ์ถ”๋ก ์˜ ๊ฒฐ๊ณผ๋กœ ๋งŒ๋“œ๋Š” ์ฝ”๋“œ์ด๋‹ค (Non Max Suppression).
float max_conf = detectionResult[0][0][4];
        int idx = 0;
        for(int i = 0; i<3024; i++){
            if(max_conf < detectionResult[0][i][4]){
                max_conf = detectionResult[0][i][4];
                idx = i;
            }
        }
  • ๊ผญ์ง“์  ์˜ˆ์ธก ๋ชจ๋ธ & ๊ธ€์ž ์˜ˆ์ธก ๋ชจ๋ธ
    ๋ชจ๋ธ ํ•™์Šต ์‹œ, ๊ฒ€์ฆ ๋ฐ์ดํ„ฐ์…‹์— ๋Œ€ํ•ด ๊ฐ€์žฅ ๋†’์€ ์ •ํ™•๋„๋ฅผ ๊ฐ€์ง€๋Š” ๋ชจ๋ธ์˜ ๊ฐ€์ค‘์น˜๋ฅผ onnx ํŒŒ์ผ๋กœ ์ €์žฅํ•˜๊ณ , tflite_converter.py๋ฅผ ํ†ตํ•ด ์ตœ์ข…์ ์œผ๋กœ ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜ ์ƒ์—์„œ ๋ชจ๋ธ์„ ๋กœ๋“œํ•  ๋•Œ ์“ฐ์ด๋Š” TensorFlow Lite ํŒŒ์ผ๋กœ ๋ณ€ํ™˜ํ•œ๋‹ค.

6. ์•ˆ๋“œ๋กœ์ด๋“œ ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜ ์ œ์ž‘

์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜์— ์•ž์„œ ๋งŒ๋“  ํ•™์Šต๋œ ๋ชจ๋ธ๋“ค์„ ์‚ฌ์šฉํ•˜๊ธฐ ์œ„ํ•ด์„œ๋Š” ๊ฐ ๋ชจ๋ธ์— ๋Œ€ํ•œ ์ถ”๋ก  ์ฝ”๋“œ๋ฅผ ๋งŒ๋“ค๊ณ , ์ด๋ฅผ ์•ˆ๋“œ๋กœ์ด๋“œ ์ŠคํŠœ๋””์˜ค์˜ MainActivity์— ๋ถˆ๋Ÿฌ์™€์„œ ์‚ฌ์šฉํ•ด์•ผ ํ•œ๋‹ค. ์šฐ๋ฆฌ๋Š” YOLOv5(DHDetectionModel.java), ๊ผญ์ง“์  ์˜ˆ์ธก(AlignmentModel.java), ๊ธ€์ž์˜ˆ์ธก(CharModel.java) ์ด ์„ธ ๊ฐ€์ง€ ๋ชจ๋ธ์— ๋Œ€ํ•œ ์ถ”๋ก  ์ฝ”๋“œ๋ฅผ ๋งŒ๋“ค์—ˆ๋‹ค. ์ถ”๋ก  ์ฝ”๋“œ์— ์‚ฌ์šฉ๋œ ๋ฉ”์†Œ๋“œ๋“ค์€ ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค:

  • ์ƒ์„ฑ์ž

    DHDetectionModel(Activity activity, Interpreter.Options options)
    AlignmentModel(Activity activity, Interpreter.Options options)
    CharModel(Activity activity, Interpreter.Options options)

    --> ๊ฐ ์ถ”๋ก  ์ธ์Šคํ„ด์Šค๋ฅผ ์ƒ์„ฑํ•  ๋•Œ, ๋ชจ๋ธ ์ธํ„ฐํ”„๋ฆฌํ„ฐ(mInterpreter)์™€ ๋ชจ๋ธ์— ๋“ค์–ด๊ฐ€๋Š” ์ž…๋ ฅ(mImageData)์— ๋Œ€ํ•ด์„œ ์ •์˜ํ•œ๋‹ค.

  • ๊ณตํ†ต์ ์œผ๋กœ ์‚ฌ์šฉ๋œ ๋ฉ”์†Œ๋“œ

    MappedByteBuffer loadModelFile(Activity activity)

    --> tflite ํŒŒ์ผ์„ ๋ถˆ๋Ÿฌ์˜ค๋Š” ๋ฉ”์†Œ๋“œ๋กœ ์ธํ„ฐํ”„๋ฆฌํ„ฐ ์ƒ์„ฑ์‹œ์— ์‚ฌ์šฉ๋œ๋‹ค.

    void convertBitmapToByteBuffer(Bitmap bitmap)

    --> ์ถ”๋ก ํ• ๋•Œ ์ด๋ฏธ์ง€๋ฅผ ๋ชจ๋ธ์— ๋“ค์–ด๊ฐ€๋Š” ์ž…๋ ฅ ํ˜•์‹์ธ ByteBuffer์˜ ํ˜•ํƒœ๋กœ ๋ฐ”๊พธ์–ด์ฃผ๋Š” ๋ฉ”์†Œ๋“œ์ด๋‹ค.

  • ์ถ”๋ก  ๋ฉ”์†Œ๋“œ

    • DHDetectionModel

      float[][] getProposal(Bitmap bm, Mat input)

      --> ์ด๋ฏธ์ง€๊ฐ€ ์ž…๋ ฅ์œผ๋กœ ๋“ค์–ด๊ฐ€๋ฉด float[2][5] ํ˜•ํƒœ์˜ ์ •๋ณด๋ฅผ ์ถœ๋ ฅํ•œ๋‹ค. ์ถœ๋ ฅ๊ฐ’์—๋Š” ๋ชจ๋ธ์ด ํƒ์ง€ํ•œ bounding box์˜ x, y, w, h, confidence์— ๋Œ€ํ•œ ์ •๋ณด๋ฅผ ๋‹ด๊ณ  ์žˆ๋‹ค. Yolov5์— nms๊ฐ€ tflite ํ˜•ํƒœ๋กœ ๋ณ€ํ™˜๋˜์ง€ ์•Š๊ธฐ ๋•Œ๋ฌธ์— ๋”ฐ๋กœ nms ์ฝ”๋“œ๋ฅผ ์ถ”๊ฐ€ํ•˜์˜€๋‹ค.

    • AlignmentModel

      float[] getCoordinate(Bitmap bitmap)

      --> DHDetectionModel์—์„œ ๋‚˜์˜จ ์ถœ๋ ฅ์„ ์ด์šฉํ•ด bounding box์˜ ํฌ๊ธฐ๋กœ ์ž๋ฅธ ์ด๋ฏธ์ง€๊ฐ€ ์ž…๋ ฅ์œผ๋กœ ๋“ค์–ด๊ฐ€๋ฉด, float[8] ํ˜•ํƒœ์˜ ์ •๋ณด๋ฅผ ์ถœ๋ ฅํ•œ๋‹ค. ์ถœ๋ ฅ๊ฐ’์—๋Š” ๋ชจ๋ธ์ด ์˜ˆ์ธกํ•œ ๊ผญ์ง“์ ์˜ ๋„ค ์ขŒํ‘œ์˜ (x,y)๊ฐ’์„ ๋‹ด๊ณ ์žˆ๋‹ค.

    • CharModel

      String getString(Bitmap bm)

      --> AlignmentModel์—์„œ ๋‚˜์˜จ ์ถœ๋ ฅ์„ ์ด์šฉํ•ด ๋ฒˆํ˜ธํŒ ํฌ๊ธฐ๋กœ ์ด๋ฏธ์ง€๋ฅผ ์ž๋ฅธ ํ›„ ์ „๋‹จ๋ณ€ํ™˜์„ ์ด์šฉํ•ด ์ •๋ฉด์œผ๋กœ ๊ณง๊ฒŒ ํŽธ ์ด๋ฏธ์ง€๊ฐ€ ์ž…๋ ฅ์œผ๋กœ ๋“ค์–ด๊ฐ€๋ฉด, String ํ˜•ํƒœ์˜ ์ •๋ณด๋ฅผ ์ถœ๋ ฅํ•œ๋‹ค. ์ถœ๋ ฅ๊ฐ’์—๋Š” ๋ชจ๋ธ์ด ์˜ˆ์ธกํ•œ ๋ฒˆํ˜ธํŒ์˜ ๊ธ€์ž ์ •๋ณด๋ฅผ ๋‹ด๊ณ ์žˆ๋‹ค.

  • ์ถ”๋ก  ์†๋„(FPS) ๋ฌธ์ œ ๊ฐœ์„ 
    ์ดˆ๊ธฐ์— ๋ชจ๋“  ๋ชจ๋ธ๋“ค์„ ์•ฑ์— ์ ์šฉํ•˜์˜€์„ ๋•Œ, ํ•œ ์ด๋ฏธ์ง€๋ฅผ ์ฒ˜๋ฆฌํ•˜๋Š” ์‹œ๊ฐ„์ด ๋„ˆ๋ฌด ์˜ค๋ž˜๊ฑธ๋ ค์„œ ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๋ฐฉ๋ฒ•์œผ๋กœ ์‹ค์‹œ๊ฐ„ ์ถ”๋ก ์„ ๊ฐ€๋Šฅํ•˜๊ฒŒ ํ•˜์˜€๋‹ค.

    1. YOLO ์ž…๋ ฅ ์ด๋ฏธ์ง€ ํฌ๊ธฐ ๊ฐ์†Œ (640, 480) -> (256,192)
    2. GPU ๋Œ€๋ฆฌ์ž ์‚ฌ์šฉ
    3. ๋ฉ€ํ‹ฐ์Šค๋ ˆ๋”ฉ
  • ์ตœ์ข… ๋ชจ๋ธ๋ณ„ & ์ „์ฒด ์ถ”๋ก ์‹œ๊ฐ„

    ๋ชจ๋ธ ์ถ”๋ก ์‹œ๊ฐ„(millisecond)
    ๋ฒˆํ˜ธํŒ ํƒ์ง€ ๋ชจ๋ธ 45
    ๊ผญ์ง“์  ์˜ˆ์ธก ๋ชจ๋ธ 82
    ๊ธ€์ž ๋ชจ๋ธ 86
  • ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜ ์˜ˆ

    ์˜ˆ์‹œ1 ์˜ˆ์‹œ2
    ์˜ˆ์‹œ1 ์˜ˆ์‹œ2

7. Google Play์— ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋“ฑ๋ก

๋‹ค์šด๋กœ๋“œ:

์„ค์น˜ ์ „ ์„ค์น˜ ํ›„
์˜ˆ์‹œ ์˜ˆ์‹œ2
[TIP 2020] Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion

Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion Code for Multi-Temporal Scene Classification and Scene Ch

Lixiang Ru 33 Dec 12, 2022
Lane assist for ETS2, built with the ultra-fast-lane-detection model.

Euro-Truck-Simulator-2-Lane-Assist Lane assist for ETS2, built with the ultra-fast-lane-detection model. This project was made possible by the amazing

36 Jan 05, 2023
A PyTorch implementation of the paper "Semantic Image Synthesis via Adversarial Learning" in ICCV 2017

Semantic Image Synthesis via Adversarial Learning This is a PyTorch implementation of the paper Semantic Image Synthesis via Adversarial Learning. Req

Seonghyeon Nam 146 Nov 25, 2022
Experimental Python implementation of OpenVINO Inference Engine (very slow, limited functionality). All codes are written in Python. Easy to read and modify.

PyOpenVINO - An Experimental Python Implementation of OpenVINO Inference Engine (minimum-set) Description The PyOpenVINO is a spin-off product from my

Yasunori Shimura 7 Oct 31, 2022
Uses Open AI Gym environment to create autonomous cryptocurrency bot to trade cryptocurrencies.

Crypto_Bot Uses Open AI Gym environment to create autonomous cryptocurrency bot to trade cryptocurrencies. Steps to get started using the bot: Sign up

21 Oct 03, 2022
Cooperative Driving Dataset: a dataset for multi-agent driving scenarios

Cooperative Driving Dataset (CODD) The Cooperative Driving dataset is a synthetic dataset generated using CARLA that contains lidar data from multiple

Eduardo Henrique Arnold 124 Dec 28, 2022
Code for the paper Learning the Predictability of the Future

Learning the Predictability of the Future Code from the paper Learning the Predictability of the Future. Website of the project in hyperfuture.cs.colu

Computer Vision Lab at Columbia University 139 Nov 18, 2022
Code for paper 'Hand-Object Contact Consistency Reasoning for Human Grasps Generation' at ICCV 2021

GraspTTA Hand-Object Contact Consistency Reasoning for Human Grasps Generation (ICCV 2021). Project Page with Videos Demo Quick Results Visualization

Hanwen Jiang 47 Dec 09, 2022
Pytorch Lightning Implementation of SC-Depth Methods.

SC_Depth_pl: This is a pytorch lightning implementation of SC-Depth (V1, V2) for self-supervised learning of monocular depth from video. In the V1 (IJ

JiaWang Bian 216 Dec 30, 2022
Rohit Ingole 2 Mar 24, 2022
PyTorch implementation of EigenGAN

PyTorch Implementation of EigenGAN Train python train.py [image_folder_path] --name [experiment name] Test python test.py [ckpt path] --traverse FFH

62 Nov 12, 2022
๐Ÿฅˆ78th place in Riiid Solution๐Ÿฅˆ

Riiid Answer Correctness Prediction Introduction This repository is the code that placed 78th in Riiid Answer Correctness Prediction competition. Requ

ds wook 14 Apr 26, 2022
This is a Deep Leaning API for classifying emotions from human face and human audios.

Emotion AI This is a Deep Leaning API for classifying emotions from human face and human audios. Starting the server To start the server first you nee

crispengari 5 Oct 02, 2022
SAGE: Sensitivity-guided Adaptive Learning Rate for Transformers

SAGE: Sensitivity-guided Adaptive Learning Rate for Transformers This repo contains our codes for the paper "No Parameters Left Behind: Sensitivity Gu

Chen Liang 23 Nov 07, 2022
City Surfaces: City-scale Semantic Segmentation of Sidewalk Surfaces

City Surfaces: City-scale Semantic Segmentation of Sidewalk Surfaces Paper Temporary GitHub page for City Surfaces paper. More soon! While designing s

14 Nov 10, 2022
The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation

PointNav-VO The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation Project Page | Paper Table of Contents Setup

Xiaoming Zhao 41 Dec 15, 2022
Tom-the-AI - A compound artificial intelligence software for Linux systems.

Tom the AI (version 0.82) WARNING: This software is not yet ready to use, I'm still setting up the GitHub repository. Should be ready in a few days. T

2 Apr 28, 2022
Repositorio oficial del curso IIC2233 Programaciรณn Avanzada ๐Ÿš€โœจ

IIC2233 - Programaciรณn Avanzada Evaluaciรณn Las evaluaciones serรกn efectuadas por medio de actividades prรกcticas en clases y tareas. Se calcularรก la no

IIC2233 @ UC 0 Dec 15, 2022
Model search is a framework that implements AutoML algorithms for model architecture search at scale

Model search (MS) is a framework that implements AutoML algorithms for model architecture search at scale. It aims to help researchers speed up their exploration process for finding the right model a

Google 3.2k Dec 31, 2022
Consensus Learning from Heterogeneous Objectives for One-Class Collaborative Filtering

Consensus Learning from Heterogeneous Objectives for One-Class Collaborative Filtering This repository provides the source code of "Consensus Learning

SeongKu-Kang 6 Apr 29, 2022