Section4

Queue

queue의 역할

• A Hook is a high-level interface to an external platform that lets you quickly and easily talk t o them without having t o write low-level code that hits their API or uses special libraries. They’re also often t h e building blocks t h a t O p e r a t o r s a r e b u i l t o u t o f .

Perfomance Tuning

• core.paralleism : scheduler당 task인스턴스의 동시 수행 개수, 워커의 수와 상관없음
• core.max_active_runs_per_dag : 데이터 파이프라인당 동시성
• core.max_active_runs_per_dag :
• celery.worker_concurrency : 워커당 동시성
• celery.worker_prefetch_multiplier : 워커당 동시성 곱하기 값

Queue Setting 예시