Re: [DISCUSS] AIP-5: Dynamic Resource Allocation for Optimizer

[Jiwon Park <ba...@gmail.com>]

Hi all,



All phases of AIP-5 (Dynamic Resource Allocation for Optimizer) have now been merged, so I’d like to close out this thread.



AIP-5: https://cwiki.apache.org/confluence/display/AMORO/AIP-5%3A+Dynamic+Resource+Allocation+for+Optimizer Issue: https://github.com/apache/amoro/issues/4191



#4254 (AMORO-4253) — Phase 1: configuration and validation

#4272 (AMORO-4271) — Phase 2: demand-driven scale-up

#4296 (AMORO-4295) — Phase 3+4: idle-driven scale-down with graceful drain

#4314 (AMORO-4313) — Phase 5: observability metrics



As proposed, DRA is opt-in: with dynamic-allocation.enabled=false (the default), behavior is unchanged and no migration is needed. The wiki page has been updated to Implemented, with the PR list recorded there. User documentation shipped alongside the corresponding PRs.



Many thanks to Jinsong Zhou, ConradJam, and Xu Bai for the reviews and design feedback throughout, and to everyone who commented on the original proposal.



Feedback from production use is welcome — please open an issue if you hit anything unexpected.



Thanks, Jiwon Park


> 2026. 6. 5. 오후 9:43, Jiwon Park <ba...@gmail.com> 작성:
> 
> Hi all,
> 
> I’d like to open discussion on AIP-5, which proposes opt-in Dynamic Resource Allocation (DRA) for optimizer resource groups.
> 
> AIP-5: https://cwiki.apache.org/confluence/display/AMORO/AIP-5%3A+Dynamic+Resource+Allocation+for+Optimizer
> Issue: https://github.com/apache/amoro/issues/4191
> 
> Optimizer groups today run at a fixed parallelism: min-parallelism scales up to a floor but never down, so pods sit idle during low load and operators must manually rescale for spikes. AIP-5 adds an opt-in DRA mode (modeled on Spark’s SPARK-3174) that scales optimizers up and down with demand, with graceful drain, all inside AMS. With dynamic-allocation.enabled=false (default), behavior is unchanged and no migration is needed.
> 
> The wiki page has the full design — algorithm, config properties, metrics, diagrams, phased plan, and rejected alternatives.
> 
> Feedback and questions welcome. I’d love to hear your thoughts on this direction.
> 
> Thanks,
> Jiwon Park