๋ฐ์—”์œผ๋กœ ์„ฑ์žฅ์ค‘ ๐ŸŒฑ

๊ฐœ๋ฐœ ์ผ์ง€/[LOL Data Pipeline ๊ตฌ์ถ•ํ•˜๊ธฐ] 4

4. ์ž ์‹œ ๋ณด๋ฅ˜ํ•ฉ๋‹ˆ๋‹ค.

์ „๋‚  ๋ฉด์ ‘ ์ œ์˜๊ฐ€ ์™€์„œ, ์–ด์ œ ์ง„ํ–‰ํ•œ ์ดํ›„๋กœ ์Šคํ†ฑ ์ƒํƒœ์ผ ์˜ˆ์ •์ž…๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ์ตœ๊ทผ์— ๋กค ๋‹‰๋„ค์ž„ ๊ด€ํ•ด์„œ ์ •์ฑ…์ด ๋ฐ”๋€๋‹ค๊ณ  ํ–ˆ๋Š”๋ฐ, ํ˜น์‹œ ์ด์— ๋”ฐ๋ผ์„œ API ์ •์ฑ…๋„ ๋ฐ”๋€” ์ˆ˜๋„ ์žˆ์–ด์„œ ๊ฒธ์‚ฌ๊ฒธ์‚ฌ ๊ธฐ๋‹ค๋ ค๋ณด๋ ค ํ•ฉ๋‹ˆ๋‹ค. ์†Œํ™˜์‚ฌ ์ด๋ฆ„์—์„œ Riot ID๋กœ ์ „ํ™˜ ํ˜„์žฌ ๋ฆฌ๊ทธ ์˜ค๋ธŒ ๋ ˆ์ „๋“œ์˜ ์„ธ๊ณ„๋ฅผ ๋” ์ž˜ ๋ฐ˜์˜ํ•˜๊ณ  ๋ผ์ด์—‡์˜ ๊ฒŒ์ž„ ๊ฐ„ ์—ฐ๊ณ„์„ฑ์„ ๊ฐœ์„ ํ•˜๋Š” ๋™์‹œ์— ๊ธฐ์ˆ  ์ฐจ์›์—์„œ ์ด๋ฆ„์˜ ๊ด€๋ฆฌ๋ฅผ ๊ฐ„์†Œํ™”ํ•˜๊ณ ์ž ํ”Œ๋ ˆ์ด์–ด ์ด๋ฆ„์„ ์ง€์นญํ•˜๋Š” ์šฉ์–ด๋ฅผ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค. www.riotgames.com +) ์ œ๊ฐ€ ๋ฉด์ ‘ ๊ธฐ๊ฐ„ ์ค‘์— ์ •๋ง ๊ฐ์‚ฌํ•˜๊ฒŒ๋„ ํ•œ ํšŒ์‚ฌ์— ํ•ฉ๊ฒฉํ•˜๊ฒŒ ๋˜์–ด ์ž…์‚ฌํ•˜๊ฒŒ ๋˜์—ˆ์Šต๋‹ˆ๋‹ค...! ์ถ”ํ›„ ํฌ์ŠคํŒ…์ด ์•„๋งˆ ์–ด๋ ค์šธ ๊ฒƒ ๊ฐ™์ง€๋งŒ ํ‹ˆํ‹ˆ์ด ์ค€๋น„ํ•ด์„œ ๊ตฌ์ƒ‰์ด ์–ด์ƒ‰ํ•˜๋”๋ผ๋„ ๊ผญ ๋งˆ๋ฌด๋ฆฌ ์ง“๊ณ  ์‹ถ๋„ค์š”! (์ ์–ด๋„ Spark๊นŒ์ง€ ์ ์šฉํ•ด๋ณด๋Š” ๊ฒƒ์„ ๋ชฉํ‘œ๋กœ...โœจ)

3. kafka์™€ ๋ชจ๋‹ˆํ„ฐ๋ง์„ ์œ„ํ•œ grafana ์„ธํŒ…

์ด์ „ ํฌ์ŠคํŒ…์—์„œ ๊ฐ„๋‹จํ•œ API ๊ตฌ์ถ• ํ…Œ์ŠคํŠธ๋ฅผ ์™„๋ฃŒํ–ˆ๋‹ค. 0. Kafka ๊ฐœ๋… ๊ณต๋ถ€ํ•˜๊ธฐ 1. docker-compose๋กœ Kafka ํด๋Ÿฌ์Šคํ„ฐ ๊ตฌ์ถ•ํ•˜๊ธฐ ์ด์ œ ์‹ค์‹œ๊ฐ„ ๋ฐ์ดํ„ฐ๋ฅผ ๋ฐ›์•„์˜ค๊ณ  ์ €์žฅํ•  ์ˆ˜ ์žˆ๋„๋ก Kafka๋ฅผ docker-compose๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๊ตฌ์ถ•ํ•ด๋ณด์•˜๋‹ค. kafka broker 3๊ฐœ, zookeeper 3๊ฐœ๋กœ ์ธ์Šคํ„ด์Šค๋ฅผ ์˜ฌ๋ ค ์ง„ํ–‰ํ–ˆ๋‹ค. | docker-compose.yml version: '2' services: zookeeper-1: image: confluentinc/cp-zookeeper:latest ports: - "12181:2181" environment: ZOOKEEPER_SERVER_ID: 1 ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 20..

2. ์†Œํ™˜์‚ฌ๋ช… ๊ฒ€์ƒ‰์œผ๋กœ ์ •๋ณด ๊ฐ€์ ธ์˜ค๋Š” API ๊ตฌํ˜„ํ•˜๊ธฐ

์•ž์˜ ํฌ์ŠคํŒ…์—์„œ ์Šคํ”„๋ง ๋ถ€ํŠธ ํ”„๋กœ์ ํŠธ๋ฅผ ์ƒ์„ฑํ•˜๊ณ  ์„ธํŒ…์„ ์™„๋ฃŒํ–ˆ์œผ๋‹ˆ, ์Šคํ”„๋ง ํ”„๋กœ์ ํŠธ์—์„œ API๋ฅผ ๊ตฌํ˜„ํ•ด๋ณด์ž. 0. ๋””์ž์ธ ํŒจํ„ด... ๋‚ด๊ฐ€ ์•„๋Š” ๊ฒƒ์ด๋ผ๊ณ ๋Š” MVC ๋ฟ์ด๋ผ ใ…‹ใ…‹ใ…‹ ์–ด๋–ป๊ฒŒ ๋ชจ๋“ˆํ™”ํ•˜๊ณ  ๊ตฌ์กฐ๋ฅผ ์งœ์•ผํ• ์ง€ ์–ด๋ ค์šด ๊ฒƒ ๊ฐ™๋‹ค. ๊ทธ๋ž˜์„œ ์ผ๋‹จ service์™€ controller๋ฅผ ๊ตฌํ˜„ํ•ด์„œ ์ง„ํ–‰ํ–ˆ๋‹ค. | SummonerService.java package sungmin.LOLDataPipeline.service; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Service; import org.springframework.web.client.RestTemplate; import sun..

1. Riot API ์‹ ์ฒญ & Spring Boot ํ”„๋กœ์ ํŠธ ์ƒ์„ฑ

0. ํ”„๋กœ์ ํŠธ ์ฃผ์ œ ๋ญํ•˜์ง€? ์‹ค์‹œ๊ฐ„์œผ๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ๋ฌด๋ฃŒ๋กœ ๋ฐ›์„ ์ˆ˜ ์žˆ๊ณ , ๋ฐ์ดํ„ฐ ์–‘๋„ ๋งŽ์€ ๊ณณ์„ ๊ฝค ์ฐพ๊ธฐ ์–ด๋ ค์› ๋‹ค. ์‹ค์‹œ๊ฐ„ ์ง€์ง„ ๋ฐ์ดํ„ฐ, ํ•ญ๊ณต ๋ฐ ์„ ๋ฐ• ๋ฐ์ดํ„ฐ ๊ฐ™์ด ์žฌ๋ฏธ๋„ ์žˆ์„ ๊ฒƒ ๊ฐ™๊ณ  ์‹ค์‹œ๊ฐ„์œผ๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ๋‹ค๋ค„์„œ ์ด๊ฒƒ ์ €๊ฒƒ ๋‹ค์–‘ํ•œ ๊ฒฐ๊ณผ๋„ ๋งŒ๋“ค์–ด ๋ณผ ์ˆ˜ ์žˆ์„ ๊ฒƒ ๊ฐ™์•„์„œ ๊ตฌํ•  ์ˆ˜ ์žˆ๋Š”์ง€ ์ฐพ์•„๋ณด๋‹ค ํ•ด์™ธ ์‚ฌ์ดํŠธ๊นŒ์ง€ ๊ฐ€์ž…ํ•ด์„œ ์‹ ์ฒญ๋„ ํ•ด๋ดค์ง€๋งŒ ๊ฒฐ๊ตญ Fail... ์–ด๋–ค ํ”„๋กœ์ ํŠธ ์ฃผ์ œ๋ฅผ ํ•ด๋ณผ์ง€ ์ด๊ฒƒ์ €๊ฒƒ ์ฐพ์•„๋ณด๊ณ  ์‹œ๋„ํ•˜๋‹ค๊ฐ€, ์ฆ๊ฒจํ•˜๋Š” ๊ฒŒ์ž„์ธ ๋กค(League of Legends, LOL) ๋ฐ์ดํ„ฐ๋กœ ๊ฒฐ์ •ํ–ˆ๋‹ค. ๋งˆ์นจ ๊ด€๋ จ ํ”„๋กœ์ ํŠธ๋ฅผ ์ด๋ฏธ ์ง„ํ–‰ํ•˜์‹  ์„ ๋ก€์˜ ๋ธ”๋กœ๊ทธ๊ฐ€ ์žˆ์–ด, ๋งŽ์ด ์ฐธ๊ณ ํ•˜๊ณ  ์‘์šฉํ•ด๋ณด๋Š” ๊ฒƒ์„ ๋ชฉํ‘œ๋กœ ์ง„ํ–‰ํ•˜๊ธฐ๋กœ ํ–ˆ๋‹ค ใ…Žใ…Ž ๊ฑฐ์ฐฝํ•˜๊ฒŒ ๋งŒ๋“ค ์ˆ˜๋Š” ์—†๊ฒ ์ง€๋งŒ, ๊ทธ๋™์•ˆ ๋‹ค๋ค„๋ณด๊ณ  ์‹ถ์—ˆ๋˜ ๊ธฐ์ˆ  ์Šคํƒ๋“ค(ํŠนํžˆ Kafka๋ž‘ Spark ๋“ฑ๋“ฑ)..