[Spark] Join Strategies 과 Shuffle
shuffle join, broadcast join / shuffle sort merge join, broadcast hash join / join hint
Posted by
Wonyong Jang
on April 20, 2024 ·
14 mins read
[Spark] Data Skew 해결을 위한 Salting 기법
데이터를 Even 하게 분산시키기 / Data Skew 발생시 문제 / AQE 으로 Skew 문제를 해결할 수 없는 경우
Posted by
Wonyong Jang
on April 10, 2024 ·
8 mins read
[Spark] On Kubernetes 에서 Apache YuniKorn 으로 스케줄링하기
guaranteed, limit, preemption / gang scheduler / preemption, burst, guaranteed
Posted by
Wonyong Jang
on March 12, 2024 ·
10 mins read
[Scala] Guice의 Singleton 사용
Scala 언어를 이용하여 Dependency Injection 구조 만들기
Posted by
Wonyong Jang
on March 09, 2024 ·
2 mins read
[Spark] External Shuffle Service 와 Remote Shuffle Service
ESS 와 RSS 동작방식 / Celeborn
Posted by
Wonyong Jang
on March 03, 2024 ·
6 mins read
[Spark] On Kubernetes
EMR, CDP on-premise, K8s 환경에 따른 차이 / Spark on Yarn 과 Spark on K8s 비교 / ESS 와 RSS
Posted by
Wonyong Jang
on March 03, 2024 ·
10 mins read
[Spark] Memory 관리 및 튜닝
Spark 실행시 적절한 Driver와 Executor 개수 / on-heap, off-heap, overHead memory /PySpark에서의 Memory
Posted by
Wonyong Jang
on February 13, 2024 ·
13 mins read