[Spark] External Shuffle Service 와 Remote Shuffle Service
ESS 와 RSS 동작방식 / Celeborn
Posted by
Wonyong Jang
on March 03, 2024 ·
6 mins read
[Spark] On Kubernetes
EMR, CDP on-premise, K8s 환경에 따른 차이 / Spark on Yarn 과 Spark on K8s 비교 / ESS 와 RSS
Posted by
Wonyong Jang
on March 03, 2024 ·
7 mins read
[Spark] Memory 관리 및 튜닝
Spark 실행시 적절한 Driver와 Executor 개수 / on-heap, off-heap, overHead memory /PySpark에서의 Memory
Posted by
Wonyong Jang
on February 13, 2024 ·
13 mins read
[AWS] S3 버킷 수명 주기 구성
DeletingObjectsfromVersioningSuspendedBuckets, Versioning Suspended Bucket Lifecycle
Posted by
Wonyong Jang
on January 11, 2024 ·
4 mins read
[Scala] is 로 시작하는 Boolean 타입 필드 사용시 이슈
java, kotlin 그리고 scala 언어에서의 차이 / jackson을 이용한 serialize 할 때 주의사항
Posted by
Wonyong Jang
on November 25, 2023 ·
10 mins read
[Spark] Log4j를 이용한 Log Rolling(RollingFileAppender)
Custom Log4j 사용하기 / Long Running Spark Streaming 에서 Log Rolling
Posted by
Wonyong Jang
on November 19, 2023 ·
3 mins read
[AWS] Event Bridge
Event bridge dead letter queue, CloudWatch Log group, Monitoring
Posted by
Wonyong Jang
on October 22, 2023 ·
4 mins read