License Plate Detection Application

Overview

LicensePlate_Project ๐Ÿš— ๐Ÿš™

[Project] 2021.02 ~ 2021.09 License Plate Detection Application

Overview


1. ๋ฐ์ดํ„ฐ ์ˆ˜์ง‘ ๋ฐ ๋ผ๋ฒจ๋ง

์ฐจ๋Ÿ‰ ๋ฒˆํ˜ธํŒ ์ด๋ฏธ์ง€๋ฅผ ์ง์ ‘ ์ˆ˜์ง‘ํ•˜์—ฌ ๊ฐ ์ด๋ฏธ์ง€์— ๋Œ€ํ•ด '๋ฒˆํ˜ธํŒ ๊ธ€์ž'์™€ '๋ฒˆํ˜ธํŒ ๋„ค ๊ผญ์ง“์ ์˜ x,y ์ขŒํ‘œ'๋ฅผ ๋ผ๋ฒจ๋ง ํ•œ๋‹ค.

๋ฒˆํ˜ธํŒ ์ด๋ฏธ์ง€
๋ผ๋ฒจ๋ง 20210210_222919.jpg 1481 2773 2043 2689 2043 2794 1486 2883 36์กฐ 2428

ํ…์ŠคํŠธ ํŒŒ์ผ๋กœ ์ €์žฅ๋œ ๋ผ๋ฒจ๋ง ์ •๋ณด๋Š” ๋ฒˆํ˜ธํŒ ๋„ค ๊ผญ์ง“์ ์˜ ์ ˆ๋Œ€ ์ขŒํ‘œ์™€ ๋ฒˆํ˜ธํŒ ๊ธ€์ž๋ฅผ ํฌํ•จํ•˜๊ณ  ์žˆ๋‹ค. ํ•™์Šต ๋ฐ์ดํ„ฐ์˜ 20%๋ฅผ ๊ฒ€์ฆ ๋ฐ์ดํ„ฐ๋กœ ๋‚˜๋ˆ„์–ด ๋ฐ์ดํ„ฐ์…‹ ์ค€๋น„๋ฅผ ๋งˆ์นœ๋‹ค. ์ตœ์ข… ๋ฐ์ดํ„ฐ์…‹ ๊ตฌ์„ฑ์€ ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

ํ•™์Šต ๋ฐ์ดํ„ฐ ๊ฒ€์ฆ ๋ฐ์ดํ„ฐ
1635์žฅ 409์žฅ

2. YOLOv5 ํ•™์Šต (Pytorch-YOLOv5)

  • ์ฐธ๊ณ : https://github.com/ultralytics/yolov5

  • ์ธํ’‹ ๋ฐ์ดํ„ฐ ์ค€๋น„
    ์›๋ณธ ์ด๋ฏธ์ง€๋Š” ๋ฒˆํ˜ธํŒ ์˜์—ญ์„ ํƒ์ง€ํ•˜๊ธฐ ์œ„ํ•ด ๊ณง์žฅ YOLO์˜ ์ž…๋ ฅ์œผ๋กœ ์‚ฌ์šฉ๋˜๊ธฐ ๋•Œ๋ฌธ์—, YOLO์˜ ์ž…๋ ฅ ํ˜•์‹์— ๋งž์ถ”๊ธฐ ์œ„ํ•ด ๊ฐ ์ด๋ฏธ์ง€ ๋งˆ๋‹ค ์ด๋ฏธ์ง€ ํŒŒ์ผ๋ช…๊ณผ ๋™์ผํ•œ ์ด๋ฆ„์˜ ํ…์ŠคํŠธ ํŒŒ์ผ์„ ๋งŒ๋“ค์–ด bounding box์˜ ์ขŒํ‘œ ์ •๋ณด๋ฅผ class, x_center, y_center, width, height์˜ ํฌ๋งท์˜ ๋ฌธ์ž์—ด๋กœ ์ €์žฅํ•œ๋‹ค. ์ด ๋•Œ, class๋ฅผ ์ œ์™ธํ•œ ๋‚˜๋จธ์ง€ ๊ฐ’์€ ๋ชจ๋‘ 0-1 ์‚ฌ์ด์˜ ์ƒ๋Œ€ ์ขŒํ‘œ๋กœ ๋ณ€ํ™˜ํ•œ๋‹ค.

โ”œโ”€โ”€ Yolo_input
    โ”œโ”€โ”€ train
    โ”‚   โ”œโ”€โ”€ images
    โ”‚   โ”‚   โ”œโ”€โ”€ 1.jpg
    โ”‚ 	โ”‚   โ”œโ”€โ”€ 2.jpg
    โ”‚ 	โ”‚  	โ”‚     :
    โ”‚ 	โ”‚  		  
    โ”‚   โ”œโ”€โ”€ labels
    โ”‚	    โ”œโ”€โ”€ 1.txt
    โ”‚	    โ”œโ”€โ”€ 2.txt
    โ”‚	   	โ”‚     :
    โ”‚	
    โ””โ”€โ”€ val
 	    โ”œโ”€โ”€ images
 	    โ”œโ”€โ”€ labels
  • dataset.yaml ์ค€๋น„
    Custom ๋ฐ์ดํ„ฐ์…‹์— YOLOv5 ํ•™์Šต ์ฝ”๋“œ๋ฅผ ๊ทธ๋Œ€๋กœ ์“ธ ๊ฒƒ์ด๊ธฐ ๋•Œ๋ฌธ์—, ๋ฐ์ดํ„ฐ์…‹ ์„ธํŒ… ๋ถ€๋ถ„๋งŒ ์ˆ˜์ •ํ•œ๋‹ค. dataset.yaml ํŒŒ์ผ์— ํ•™์Šต, ๊ฒ€์ฆ ๋ฐ์ดํ„ฐ ๊ฒฝ๋กœ์™€ ๊ฐ์ฒด ํด๋ž˜์Šค ์ •๋ณด๋ฅผ ๊ธฐ์ž…ํ•œ๋‹ค. ์šฐ๋ฆฌ ํ”„๋กœ์ ํŠธ์˜ ๊ฒฝ์šฐ ํƒ์ง€ํ•˜๋Š” ๊ฐ์ฒด๊ฐ€ ์ฐจ๋Ÿ‰ ๋ฒˆํ˜ธํŒ ํ•˜๋‚˜์ด๋ฏ€๋กœ ํด๋ž˜์Šค ๋ผ๋ฒจ์„ 0์œผ๋กœ, ์ด๋ฆ„์„ 'plate' ๋กœ ํ•œ๋‹ค.

  • YOLO ๋ชจ๋ธ ์„ ํƒ
    ๋ณธ ํ”„๋กœ์ ํŠธ๋ฅผ ์œ„ํ•ด ๊ฐ€์žฅ ์ž‘๊ณ  ๋น ๋ฅธ ๋ชจ๋ธ์ธ YOLOv5s๋ฅผ ์‚ฌ์šฉํ•˜์˜€๋‹ค.


3. ๊ผญ์ง“์  ์˜ˆ์ธก ๋ชจ๋ธ ํ•™์Šต

  • ์‚ฌ์šฉํ•œ ๋ชจ๋ธ : timm์œผ๋กœ ์‚ฌ์ „ํ•™์Šต๋œ Resnet18 ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜์˜€๋‹ค

  • ์ฒซ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•

    1. ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ : ๋„ค ๊ผญ์ง“์  ์ขŒํ‘œ๊ฐ’์„ ์ด์šฉํ•˜์—ฌ ๋งŒ๋“  ๋ฐ”์šด๋”ฉ ๋ฐ•์Šค์—์„œ ๊ฐ ์ถ•์œผ๋กœ 1%์”ฉ ๋Š˜์ธ ์ด๋ฏธ์ง€

    2. ๋ฐ์ดํ„ฐ ์ฆ๊ฐ•: ์ „๋‹จ ๋ณ€ํ™˜(shear transformation), ์‚ฌ์ง„ํ•ฉ์„ฑ, ๋ฐ๊ธฐ์กฐ์ ˆ, ๋ฆฌ์‚ฌ์ด์ฆˆ
      ์ž…๋ ฅ ์ด๋ฏธ์ง€๋ฅผ ์ „๋‹จ ๋ณ€ํ™˜ ๊ธฐ๋ฒ•์„ ์ด์šฉํ•ด x, y์ถ•์œผ๋กœ ๋žœ๋คํ•˜๊ฒŒ ๋ณ€ํ™˜ํ•˜๋ฉด ๊ฒ€์€์ƒ‰ ์—ฌ๋ฐฑ ๋ถ€๋ถ„์ด ์ƒ๊ฒจ, ์ด ๋ถ€๋ถ„์„ ๋‹ค๋ฅธ ์ด๋ฏธ์ง€์—์„œ ๋žœ๋คํ•˜๊ฒŒ ๊ฐ€์ ธ์™€ ํ•ฉ์„ฑ์‹œ์ผฐ๋‹ค. ์ด ์ด๋ฏธ์ง€์— ๋žœ๋ค์œผ๋กœ ๋ฐ๊ธฐ์กฐ์ ˆ์„ ์ถ”๊ฐ€ํ•˜์—ฌ, 128x128 ์ด๋ฏธ์ง€๋กœ ๋ฆฌ์‚ฌ์ด์ฆˆํ•œ ์ด๋ฏธ์ง€๋ฅผ ๋ชจ๋ธ์— ์ž…๋ ฅ์œผ๋กœ ๋„ฃ์—ˆ๋‹ค.

    3. ๋ฌธ์ œ์  : ๊ฒ€์€์ƒ‰ ๋ถ€๋ถ„์„ ๋‹ค๋ฅธ ์‚ฌ์ง„์œผ๋กœ ํ•ฉ์„ฑ์‹œ์ผฐ๋”๋‹ˆ ์‹ค์„ธ๊ณ„ ๋ฐ์ดํ„ฐ์™€ ๊ดด๋ฆฌ๊ฐ์ด ์ƒ๊ฒจ ์„ฑ๋Šฅ ์ €ํ•˜ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•˜์˜€๋‹ค.

    ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ ๋ฐ์ดํ„ฐ์ฆ๊ฐ•1 ๋ฐ์ดํ„ฐ์ฆ๊ฐ•2
    ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ ๋ฐ์ดํ„ฐ์ฆ๊ฐ•1 ๋ฐ์ดํ„ฐ์ฆ๊ฐ•2
  • ๋‘ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•

    1. ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ : ์›๋ณธ ์ด๋ฏธ์ง€

    2. ๋ฐ์ดํ„ฐ ์ฆ๊ฐ•: ์ „๋‹จ ๋ณ€ํ™˜, ๋ฐ๊ธฐ์กฐ์ ˆ, ๋ฆฌ์‚ฌ์ด์ฆˆ ์ž…๋ ฅ ์ด๋ฏธ์ง€์™€ ๋ผ๋ฒจ๋ง์„ ํ†ตํ•ด ์•Œ๋ ค์ง„ ๋ฒˆํ˜ธํŒ ๊ผญ์ง“์ ์˜ ์ขŒํ‘œ๋“ค์„ ์ „๋‹จ ๋ณ€ํ™˜ ๊ธฐ๋ฒ•์„ ์ด์šฉํ•ด ๋žœ๋ค ๊ฐ’์œผ๋กœ ๋ณ€ํ™˜ํ•œ๋‹ค. ์ด ์ด๋ฏธ์ง€์—์„œ ๋ฒˆํ˜ธํŒ์˜ ์ขŒํ‘œ๋ฅผ ๊ธฐ์ค€์œผ๋กœ margin์„ ์ฃผ๊ณ , ๊ทธ ์ง€์ ์œผ๋กœ๋ถ€ํ„ฐ ๋žœ๋คํ•˜๊ฒŒ ์ขŒํ‘œ๋ฅผ ์ฐ์–ด ์ด๋ฏธ์ง€๋ฅผ ์ž๋ฅธ ๊ฒƒ์„ ์‚ฌ์šฉ. ์ด๋ ‡๊ฒŒ ํ•˜๋ฉด ์ฒซ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•์—์„œ ๋‚˜์™”๋˜ ๊ฒ€์€ ์—ฌ๋ฐฑ ๋ถ€๋ถ„์ด ๋‚˜์˜ค์ง€ ์•Š์œผ๋ฏ€๋กœ ์‹ค์„ธ๊ณ„ ๋ฐ์ดํ„ฐ์™€ ๋” ๊ทผ์ ‘ํ•˜๋‹ค. ์ด ์ด๋ฏธ์ง€์— ๋žœ๋ค์œผ๋กœ ๋ฐ๊ธฐ์กฐ์ ˆ์„ ์ถ”๊ฐ€ํ•˜์—ฌ, 128x128 ์ด๋ฏธ์ง€๋กœ ๋ฆฌ์‚ฌ์ด์ฆˆํ•œ ์ด๋ฏธ์ง€๋ฅผ ๋ชจ๋ธ์— ์ž…๋ ฅ์œผ๋กœ ๋„ฃ์—ˆ๋‹ค.

    ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ ๋ฐ์ดํ„ฐ์ฆ๊ฐ•1 ๋ฐ์ดํ„ฐ์ฆ๊ฐ•2
    ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ ๋ฐ์ดํ„ฐ์ฆ๊ฐ•1 ๋ฐ์ดํ„ฐ์ฆ๊ฐ•2
  • Output : ์ƒํ•˜์ขŒ์šฐ ๋„ค ๊ผญ์ง“์ ์— ๋Œ€ํ•œ X,Y ์ƒ๋Œ€์ขŒํ‘œ


4. ๊ธ€์ž ์˜ˆ์ธก ๋ชจ๋ธ ํ•™์Šต

  • ์‚ฌ์šฉํ•œ ๋ชจ๋ธ : timm์œผ๋กœ ์‚ฌ์ „ํ•™์Šต๋œ Resnet18 ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜์˜€๋‹ค.

  • ์ฒซ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•

    1. ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ : ์›๋ณธ ์ด๋ฏธ์ง€์˜ ๋„ค ๊ผญ์ง“์  ์ขŒํ‘œ์— ๋Œ€ํ•œ ground truth๋ฅผ ์ด์šฉํ•˜์—ฌ (128, 256)์˜ ํฌ๊ธฐ๋กœ ํˆฌ์˜๋ณ€ํ™˜ํ•œ ์ด๋ฏธ์ง€

    2. ๋ฐ์ดํ„ฐ ์ฆ๊ฐ•: Salt & Pepper ๋…ธ์ด์ฆˆ ์‹ค์ œ ์ฐจ๋Ÿ‰์˜ ๋ฒˆํ˜ธํŒ์€ ๋จผ์ง€ ๋ฐ ๋ฒŒ๋ ˆ์™€ ๊ฐ™์€ ์ด๋ฌผ์งˆ ๋•Œ๋ฌธ์— ์–ผ๋ฃฉ๋œ๋ฃฉํ•œ ๊ฒฝ์šฐ๊ฐ€ ๋งŽ๋‹ค. ๋”ฐ๋ผ์„œ ์ž…๋ ฅ ๋ฐ์ดํ„ฐ์— ๋žœ๋คํ•œ ๋…ธ์ด์ฆˆ๋ฅผ ์ถ”๊ฐ€ํ•˜์—ฌ ์ผ๋ฐ˜์ ์ธ ์ƒํ™ฉ๊นŒ์ง€ ์ปค๋ฒ„ํ•  ์ˆ˜ ์žˆ๋„๋ก ํ•˜์˜€๋‹ค.

    3. ๋ฌธ์ œ์  : ์‹ค์ œ ์ถ”๋ก  ๊ณผ์ •์—์„œ๋Š” ๊ผญ์ง“์  ์˜ˆ์ธก ๋ชจ๋ธ๋กœ๋ถ€ํ„ฐ ์˜ˆ์ธก๋œ ๊ผญ์ง“์  ๊ฐ’์„ ๊ธฐ๋ฐ˜์œผ๋กœ ์ •๋ ฌ๋œ ๋ฒˆํ˜ธํŒ ์ด๋ฏธ์ง€๊ฐ€ ์ž…๋ ฅ์œผ๋กœ ์‚ฌ์šฉ๋˜๋ฏ€๋กœ, ๊ธ€์ž ์˜ˆ์ธก ๋ชจ๋ธ์˜ ์„ฑ๋Šฅ์ด ๊ผญ์ง“์  ์˜ˆ์ธก ๋ชจ๋ธ์˜ ์„ฑ๋Šฅ์— ํฐ ์˜ํ–ฅ์„ ๋ฐ›์Œ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์—ˆ๋‹ค.

  • ๋‘ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•

    1. ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ : ์›๋ณธ ์ด๋ฏธ์ง€์˜ ๋„ค ๊ผญ์ง“์  ์ขŒํ‘œ๋ฅผ x,y ๋ฐฉํ–ฅ์œผ๋กœ ๊ฐ๊ฐ ๋žœ๋คํ•˜๊ฒŒ ์ด๋™์‹œํ‚จ ํ›„ (128, 256)์˜ ํฌ๊ธฐ๋กœ ํˆฌ์˜๋ณ€ํ™˜ํ•œ ์ด๋ฏธ์ง€

    2. ๋ฐ์ดํ„ฐ ์ฆ๊ฐ•: Salt & Pepper ๋…ธ์ด์ฆˆ, ๋ฐ๊ธฐ ์กฐ์ ˆ(์ „์ฒด ๋ฐ๊ฒŒ, ์ „์ฒด ์–ด๋‘ก๊ฒŒ, ๊ทธ๋ฆผ์ž) ์ˆ˜์ง‘๋œ ๋ฐ์ดํ„ฐ์…‹์€ ๋Œ€๋ถ€๋ถ„ ๋‚ฎ์— ์ฐ์€ ๋ฒˆํ˜ธํŒ ์ด๋ฏธ์ง€์˜€๊ธฐ ๋•Œ๋ฌธ์—, ํ…Œ์ŠคํŠธ ๋ฆฌํฌํŒ… ์‹œ ์•ผ๊ฐ„ ์ด๋ฏธ์ง€์— ๋Œ€ํ•ด์„œ๋Š” ์„ฑ๋Šฅ์ด ๋‚ฎ์•„์ง€๋Š” ๊ฒƒ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์—ˆ๋‹ค. ๋”ฐ๋ผ์„œ ๋ฐ๊ธฐ ์กฐ์ ˆ ๋ฐ ๊ทธ๋ฆผ์ž ์ถ”๊ฐ€ ์ฆ๊ฐ• ๊ธฐ๋ฒ•์„ ์ถ”๊ฐ€ํ•˜์—ฌ ์—ฌ๋Ÿฌ ํ™˜๊ฒฝ์˜ ์ด๋ฏธ์ง€์— ๋Œ€ํ•ด ๊ฐ•๊ฑดํ•œ ์„ฑ๋Šฅ์„ ๋ณด์ด๋„๋ก ํ•˜์˜€๋‹ค.

    ์ถ”๋ก  ์‹œ ์‹ค์ œ ์ž…๋ ฅ๋˜๋Š” ์ด๋ฏธ์ง€ ์ฒซ ๋ฒˆ์งธ ๋ฐฉ๋ฒ• ๋‘ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•

    ๋ฐ์ดํ„ฐ ์ฆ๊ฐ•์˜ ์˜ˆ์‹œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

    ์‚ฌ์šฉ๋œ ์ด๋ฏธ์ง€ ๋ฐ์ดํ„ฐ์ฆ๊ฐ•1 ๋ฐ์ดํ„ฐ์ฆ๊ฐ•2
  • Output : (๋ฐฐ์น˜์‚ฌ์ด์ฆˆ, 7, 45, 1) ๋ชจ์–‘์˜ ํ…์„œ
    7 -> 7๊ธ€์ž 45 -> 45๊ฐœ์˜ ๊ฐ€๋Šฅํ•œ ๋ฌธ์ž (['๊ฐ€', '๋‚˜', '๋‹ค', '๋ผ', '๋งˆ', '๊ฑฐ', '๋„ˆ', '๋”', '๋Ÿฌ', '๋จธ', '๋ฒ„', '์„œ', '์–ด', '์ €', '๊ณ ', '๋…ธ', '๋„', '๋กœ', '๋ชจ', '๋ณด', '์†Œ', '์˜ค', '์กฐ', '๊ตฌ', '๋ˆ„', '๋‘', '๋ฃจ', '๋ฌด', '๋ถ€', '์ˆ˜', '์šฐ', '์ฃผ', 'ํ—ˆ', 'ํ•˜', 'ํ˜ธ', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9'])


5. pt >> onnx >> pb >> tflite ๋ณ€ํ™˜

  • YOLOv5
    ์ œ๊ณตํ•ด์ฃผ๋Š” export.py๋ฅผ ์‚ฌ์šฉํ•ด TensorFlow Lite ํŒŒ์ผ๋กœ ๋ณ€ํ™˜ํ•œ๋‹ค. ์ด ๋•Œ, Non Max Suppression ๋ถ€๋ถ„์€ TensorFlow Lite๋กœ ๋ณ€ํ™˜๋˜์ง€ ์•Š์•„ ์•ˆ๋“œ๋กœ์ด๋“œ ์ŠคํŠœ๋””์˜ค ์ฝ”๋“œ๋ฅผ ์งค ๋•Œ ๋”ฐ๋กœ ์ถ”๊ฐ€ํ•˜์˜€๋‹ค. YOLO์˜ ์ถœ๋ ฅ์œผ๋กœ ๋‚˜์˜ค๋Š” (1, 3024, 6)์˜ ํ…์„œ๋Š” 3024๊ฐœ์˜ ๊ฐ€๋Šฅํ•œ ๋ฐ”์šด๋”ฉ ๋ฐ•์Šค์™€, ๊ฐ ๋ฐ”์šด๋”ฉ ๋ฐ•์Šค์˜ x_center, y_center, width, height, confidence, ๊ฐ์ฒด ํด๋ž˜์Šค ์ •๋ณด๋ฅผ ํฌํ•จํ•˜๊ณ  ์žˆ๋‹ค. ์•„๋ž˜ ์ฝ”๋“œ๋Š” ๊ฐ€๋Šฅํ•œ 3024๊ฐœ์˜ ๋ฐ”์šด๋”ฉ ๋ฐ•์Šค ์ค‘ ๊ฐ€์žฅ ํฐ confidence ๊ฐ’์„ ๊ฐ€์ง€๋Š” ํ•˜๋‚˜์˜ ๋ฐ”์šด๋”ฉ ๋ฐ•์Šค๋งŒ์„ ์ถ”๋ก ์˜ ๊ฒฐ๊ณผ๋กœ ๋งŒ๋“œ๋Š” ์ฝ”๋“œ์ด๋‹ค (Non Max Suppression).
float max_conf = detectionResult[0][0][4];
        int idx = 0;
        for(int i = 0; i<3024; i++){
            if(max_conf < detectionResult[0][i][4]){
                max_conf = detectionResult[0][i][4];
                idx = i;
            }
        }
  • ๊ผญ์ง“์  ์˜ˆ์ธก ๋ชจ๋ธ & ๊ธ€์ž ์˜ˆ์ธก ๋ชจ๋ธ
    ๋ชจ๋ธ ํ•™์Šต ์‹œ, ๊ฒ€์ฆ ๋ฐ์ดํ„ฐ์…‹์— ๋Œ€ํ•ด ๊ฐ€์žฅ ๋†’์€ ์ •ํ™•๋„๋ฅผ ๊ฐ€์ง€๋Š” ๋ชจ๋ธ์˜ ๊ฐ€์ค‘์น˜๋ฅผ onnx ํŒŒ์ผ๋กœ ์ €์žฅํ•˜๊ณ , tflite_converter.py๋ฅผ ํ†ตํ•ด ์ตœ์ข…์ ์œผ๋กœ ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜ ์ƒ์—์„œ ๋ชจ๋ธ์„ ๋กœ๋“œํ•  ๋•Œ ์“ฐ์ด๋Š” TensorFlow Lite ํŒŒ์ผ๋กœ ๋ณ€ํ™˜ํ•œ๋‹ค.

6. ์•ˆ๋“œ๋กœ์ด๋“œ ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜ ์ œ์ž‘

์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜์— ์•ž์„œ ๋งŒ๋“  ํ•™์Šต๋œ ๋ชจ๋ธ๋“ค์„ ์‚ฌ์šฉํ•˜๊ธฐ ์œ„ํ•ด์„œ๋Š” ๊ฐ ๋ชจ๋ธ์— ๋Œ€ํ•œ ์ถ”๋ก  ์ฝ”๋“œ๋ฅผ ๋งŒ๋“ค๊ณ , ์ด๋ฅผ ์•ˆ๋“œ๋กœ์ด๋“œ ์ŠคํŠœ๋””์˜ค์˜ MainActivity์— ๋ถˆ๋Ÿฌ์™€์„œ ์‚ฌ์šฉํ•ด์•ผ ํ•œ๋‹ค. ์šฐ๋ฆฌ๋Š” YOLOv5(DHDetectionModel.java), ๊ผญ์ง“์  ์˜ˆ์ธก(AlignmentModel.java), ๊ธ€์ž์˜ˆ์ธก(CharModel.java) ์ด ์„ธ ๊ฐ€์ง€ ๋ชจ๋ธ์— ๋Œ€ํ•œ ์ถ”๋ก  ์ฝ”๋“œ๋ฅผ ๋งŒ๋“ค์—ˆ๋‹ค. ์ถ”๋ก  ์ฝ”๋“œ์— ์‚ฌ์šฉ๋œ ๋ฉ”์†Œ๋“œ๋“ค์€ ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค:

  • ์ƒ์„ฑ์ž

    DHDetectionModel(Activity activity, Interpreter.Options options)
    AlignmentModel(Activity activity, Interpreter.Options options)
    CharModel(Activity activity, Interpreter.Options options)

    --> ๊ฐ ์ถ”๋ก  ์ธ์Šคํ„ด์Šค๋ฅผ ์ƒ์„ฑํ•  ๋•Œ, ๋ชจ๋ธ ์ธํ„ฐํ”„๋ฆฌํ„ฐ(mInterpreter)์™€ ๋ชจ๋ธ์— ๋“ค์–ด๊ฐ€๋Š” ์ž…๋ ฅ(mImageData)์— ๋Œ€ํ•ด์„œ ์ •์˜ํ•œ๋‹ค.

  • ๊ณตํ†ต์ ์œผ๋กœ ์‚ฌ์šฉ๋œ ๋ฉ”์†Œ๋“œ

    MappedByteBuffer loadModelFile(Activity activity)

    --> tflite ํŒŒ์ผ์„ ๋ถˆ๋Ÿฌ์˜ค๋Š” ๋ฉ”์†Œ๋“œ๋กœ ์ธํ„ฐํ”„๋ฆฌํ„ฐ ์ƒ์„ฑ์‹œ์— ์‚ฌ์šฉ๋œ๋‹ค.

    void convertBitmapToByteBuffer(Bitmap bitmap)

    --> ์ถ”๋ก ํ• ๋•Œ ์ด๋ฏธ์ง€๋ฅผ ๋ชจ๋ธ์— ๋“ค์–ด๊ฐ€๋Š” ์ž…๋ ฅ ํ˜•์‹์ธ ByteBuffer์˜ ํ˜•ํƒœ๋กœ ๋ฐ”๊พธ์–ด์ฃผ๋Š” ๋ฉ”์†Œ๋“œ์ด๋‹ค.

  • ์ถ”๋ก  ๋ฉ”์†Œ๋“œ

    • DHDetectionModel

      float[][] getProposal(Bitmap bm, Mat input)

      --> ์ด๋ฏธ์ง€๊ฐ€ ์ž…๋ ฅ์œผ๋กœ ๋“ค์–ด๊ฐ€๋ฉด float[2][5] ํ˜•ํƒœ์˜ ์ •๋ณด๋ฅผ ์ถœ๋ ฅํ•œ๋‹ค. ์ถœ๋ ฅ๊ฐ’์—๋Š” ๋ชจ๋ธ์ด ํƒ์ง€ํ•œ bounding box์˜ x, y, w, h, confidence์— ๋Œ€ํ•œ ์ •๋ณด๋ฅผ ๋‹ด๊ณ  ์žˆ๋‹ค. Yolov5์— nms๊ฐ€ tflite ํ˜•ํƒœ๋กœ ๋ณ€ํ™˜๋˜์ง€ ์•Š๊ธฐ ๋•Œ๋ฌธ์— ๋”ฐ๋กœ nms ์ฝ”๋“œ๋ฅผ ์ถ”๊ฐ€ํ•˜์˜€๋‹ค.

    • AlignmentModel

      float[] getCoordinate(Bitmap bitmap)

      --> DHDetectionModel์—์„œ ๋‚˜์˜จ ์ถœ๋ ฅ์„ ์ด์šฉํ•ด bounding box์˜ ํฌ๊ธฐ๋กœ ์ž๋ฅธ ์ด๋ฏธ์ง€๊ฐ€ ์ž…๋ ฅ์œผ๋กœ ๋“ค์–ด๊ฐ€๋ฉด, float[8] ํ˜•ํƒœ์˜ ์ •๋ณด๋ฅผ ์ถœ๋ ฅํ•œ๋‹ค. ์ถœ๋ ฅ๊ฐ’์—๋Š” ๋ชจ๋ธ์ด ์˜ˆ์ธกํ•œ ๊ผญ์ง“์ ์˜ ๋„ค ์ขŒํ‘œ์˜ (x,y)๊ฐ’์„ ๋‹ด๊ณ ์žˆ๋‹ค.

    • CharModel

      String getString(Bitmap bm)

      --> AlignmentModel์—์„œ ๋‚˜์˜จ ์ถœ๋ ฅ์„ ์ด์šฉํ•ด ๋ฒˆํ˜ธํŒ ํฌ๊ธฐ๋กœ ์ด๋ฏธ์ง€๋ฅผ ์ž๋ฅธ ํ›„ ์ „๋‹จ๋ณ€ํ™˜์„ ์ด์šฉํ•ด ์ •๋ฉด์œผ๋กœ ๊ณง๊ฒŒ ํŽธ ์ด๋ฏธ์ง€๊ฐ€ ์ž…๋ ฅ์œผ๋กœ ๋“ค์–ด๊ฐ€๋ฉด, String ํ˜•ํƒœ์˜ ์ •๋ณด๋ฅผ ์ถœ๋ ฅํ•œ๋‹ค. ์ถœ๋ ฅ๊ฐ’์—๋Š” ๋ชจ๋ธ์ด ์˜ˆ์ธกํ•œ ๋ฒˆํ˜ธํŒ์˜ ๊ธ€์ž ์ •๋ณด๋ฅผ ๋‹ด๊ณ ์žˆ๋‹ค.

  • ์ถ”๋ก  ์†๋„(FPS) ๋ฌธ์ œ ๊ฐœ์„ 
    ์ดˆ๊ธฐ์— ๋ชจ๋“  ๋ชจ๋ธ๋“ค์„ ์•ฑ์— ์ ์šฉํ•˜์˜€์„ ๋•Œ, ํ•œ ์ด๋ฏธ์ง€๋ฅผ ์ฒ˜๋ฆฌํ•˜๋Š” ์‹œ๊ฐ„์ด ๋„ˆ๋ฌด ์˜ค๋ž˜๊ฑธ๋ ค์„œ ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๋ฐฉ๋ฒ•์œผ๋กœ ์‹ค์‹œ๊ฐ„ ์ถ”๋ก ์„ ๊ฐ€๋Šฅํ•˜๊ฒŒ ํ•˜์˜€๋‹ค.

    1. YOLO ์ž…๋ ฅ ์ด๋ฏธ์ง€ ํฌ๊ธฐ ๊ฐ์†Œ (640, 480) -> (256,192)
    2. GPU ๋Œ€๋ฆฌ์ž ์‚ฌ์šฉ
    3. ๋ฉ€ํ‹ฐ์Šค๋ ˆ๋”ฉ
  • ์ตœ์ข… ๋ชจ๋ธ๋ณ„ & ์ „์ฒด ์ถ”๋ก ์‹œ๊ฐ„

    ๋ชจ๋ธ ์ถ”๋ก ์‹œ๊ฐ„(millisecond)
    ๋ฒˆํ˜ธํŒ ํƒ์ง€ ๋ชจ๋ธ 45
    ๊ผญ์ง“์  ์˜ˆ์ธก ๋ชจ๋ธ 82
    ๊ธ€์ž ๋ชจ๋ธ 86
  • ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜ ์˜ˆ

    ์˜ˆ์‹œ1 ์˜ˆ์‹œ2
    ์˜ˆ์‹œ1 ์˜ˆ์‹œ2

7. Google Play์— ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋“ฑ๋ก

๋‹ค์šด๋กœ๋“œ:

์„ค์น˜ ์ „ ์„ค์น˜ ํ›„
์˜ˆ์‹œ ์˜ˆ์‹œ2
This is the official PyTorch implementation of the paper "TransFG: A Transformer Architecture for Fine-grained Recognition" (Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang, Alan Yuille).

TransFG: A Transformer Architecture for Fine-grained Recognition Official PyTorch code for the paper: TransFG: A Transformer Architecture for Fine-gra

Ju He 307 Jan 03, 2023
OBG-FCN - implementation of 'Object Boundary Guided Semantic Segmentation'

OBG-FCN This repository is to reproduce the implementation of 'Object Boundary Guided Semantic Segmentation' in http://arxiv.org/abs/1603.09742 Object

Jiu XU 3 Mar 11, 2019
Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

HamasKhan 3 Jul 08, 2022
SlideGraph+: Whole Slide Image Level Graphs to Predict HER2 Status in Breast Cancer

SlideGraph+: Whole Slide Image Level Graphs to Predict HER2 Status in Breast Cancer A novel graph neural network (GNN) based model (termed SlideGraph+

28 Dec 24, 2022
๐Ÿฅˆ78th place in Riiid Answer Correctness Prediction competition

Riiid Answer Correctness Prediction Introduction This repository is the code that placed 78th in Riiid Answer Correctness Prediction competition. Requ

Jungwoo Park 10 Jul 14, 2022
Source code for "FastBERT: a Self-distilling BERT with Adaptive Inference Time".

FastBERT Source code for "FastBERT: a Self-distilling BERT with Adaptive Inference Time". Good News 2021/10/29 - Code: Code of FastPLM is released on

Weijie Liu 584 Jan 02, 2023
[Link]mareteutral - pars tradg wth M []

pairs-trading-with-ML Jonathan Larkin, August 2017 One popular strategy classification is Pairs Trading. Though this category of strategies can exhibi

Jonathan Larkin 134 Jan 06, 2023
Official Repository for "Robust On-Policy Data Collection for Data Efficient Policy Evaluation" (NeurIPS 2021 Workshop on OfflineRL).

Robust On-Policy Data Collection for Data-Efficient Policy Evaluation Source code of Robust On-Policy Data Collection for Data-Efficient Policy Evalua

Autonomous Agents Research Group (University of Edinburgh) 2 Oct 09, 2022
Code for How To Create A Fully Automated AI Based Trading System Withย Python

AI Based Trading System This code works as a boilerplate for an AI based trading system with yfinance as data source and RobinHood or Alpaca as broker

Rubรฉn 196 Jan 05, 2023
This code reproduces the results of the paper, "Measuring Data Leakage in Machine-Learning Models with Fisher Information"

Fisher Information Loss This repository contains code that can be used to reproduce the experimental results presented in the paper: Awni Hannun, Chua

Facebook Research 43 Dec 30, 2022
Neural Magic Eye: Learning to See and Understand the Scene Behind an Autostereogram, arXiv:2012.15692.

Neural Magic Eye Preprint | Project Page | Colab Runtime Official PyTorch implementation of the preprint paper "NeuralMagicEye: Learning to See and Un

Zhengxia Zou 56 Jul 15, 2022
Leibniz is a python package which provide facilities to express learnable partial differential equations with PyTorch

Leibniz is a python package which provide facilities to express learnable partial differential equations with PyTorch

Beijing ColorfulClouds Technology Co.,Ltd. 16 Aug 07, 2022
Code and datasets for the paper "KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction"

KnowPrompt Code and datasets for our paper "KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction" Requireme

ZJUNLP 137 Dec 31, 2022
Implementation of Shape and Electrostatic similarity metric in deepFMPO.

DeepFMPO v3D Code accompanying the paper "On the value of using 3D-shape and electrostatic similarities in deep generative methods". The paper can be

34 Nov 28, 2022
๐Ÿงฎ Matrix Factorization for Collaborative Filtering is just Solving an Adjoint Latent Dirichlet Allocation Model after All

Accompanying source code to the paper "Matrix Factorization for Collaborative Filtering is just Solving an Adjoint Latent Dirichlet Allocation Model A

Florian Wilhelm 39 Dec 03, 2022
Implementation of CVPR'2022:Surface Reconstruction from Point Clouds by Learning Predictive Context Priors

Surface Reconstruction from Point Clouds by Learning Predictive Context Priors (CVPR 2022) Personal Web Pages | Paper | Project Page This repository c

136 Dec 12, 2022
The code used for the free [email protected] Webinar series on Reinforcement Learning in Finance

Reinforcement Learning in Finance [email protected] Webinar This repository provides the code f

Yves Hilpisch 62 Dec 22, 2022
RuDOLPH: One Hyper-Modal Transformer can be creative as DALL-E and smart as CLIP

[Paper] [ะฅะฐะฑั€] [Model Card] [Colab] [Kaggle] RuDOLPH ๐ŸฆŒ ๐ŸŽ„ โ˜ƒ๏ธ One Hyper-Modal Transformer can be creative as DALL-E and smart as CLIP Russian Diffusio

AI Forever 232 Jan 04, 2023
Keyword spotting on Arm Cortex-M Microcontrollers

Keyword spotting for Microcontrollers This repository consists of the tensorflow models and training scripts used in the paper: Hello Edge: Keyword sp

Arm Software 1k Dec 30, 2022
Numba-accelerated Pythonic implementation of MPDATA with examples in Python, Julia and Matlab

PyMPDATA PyMPDATA is a high-performance Numba-accelerated Pythonic implementation of the MPDATA algorithm of Smolarkiewicz et al. used in geophysical

Atmospheric Cloud Simulation Group @ Jagiellonian University 15 Nov 23, 2022