[DISCUSS] Proposal to Add Bolt as a New Backend

[顾海燕 <gu...@bytedance.com>]

Hi Gluten community,
I would like to restart the discussion on adding Bolt as a standalone
backend to Apache Gluten.
The previous vote was withdrawn after concerns were raised about the
integration boundaries, long-term maintenance, CI resource usage, and
licensing and IP considerations. Thank you to everyone who provided
feedback.
Since then, we have revised the integration and completed additional
preparation work. The licensing and IP question was discussed with Apache
Legal in LEGAL-729 <https://issues.apache.org/jira/browse/LEGAL-729>, which
is now closed.
This thread is for discussion only. We would like to understand whether the
revised proposal addresses the previous concerns and identify any remaining
issues.

*1. Proposal and value*
Adding Bolt would provide Gluten users with a production-proven native
execution option. Bolt has been deployed at large scale across multiple
ByteDance product lines and validated on more than one million Spark jobs
per day. This production experience has driven capabilities including
adaptive task parallelism, shuffle offload, native memory management,
operator fusion, JIT optimization, and support for data lake workloads.
We propose adding Bolt as an optional standalone backend under
backends-bolt/. It will not replace or change the behavior of any existing
backend. The required shared-module changes were split into five
backend-neutral PRs and have been merged into Gluten main, as tracked in
issue #12456 <https://github.com/apache/gluten/issues/12456>. The current
PR focuses on the Bolt-specific implementation and its build, test,
documentation, and CI integration.
Tracking issue:
https://github.com/apache/gluten/issues/12456
Current Bolt backend PR:
https://github.com/apache/gluten/pull/12454
Bolt repository and technical documentation:
https://github.com/bytedance/bolt
https://bytedance.github.io/bolt/blog/

*2. Avoiding additional Bolt-specific maintenance burden*
The revised proposal is structured so that day-to-day Bolt-specific
maintenance and CI responsibilities remain with the current Bolt
contributors rather than being transferred to existing Gluten maintainers.
*Current integration status.* The Bolt backend branch has had no code
changes for more than three weeks. Against the latest Gluten main, the only
merge conflict is in package/pom.xml; no source-code conflicts were found.
*Maintenance commitment.* The current Bolt contributors will take primary
responsibility for the day-to-day maintenance of Bolt-specific code,
including compatibility updates, documentation, testing, and issue
resolution. All changes, including those under backends-bolt/, will
continue to follow the normal Gluten contribution and review process.
*CI commitment.* Bolt contributors will provide and maintain the CI
resources required for the Bolt backend and will investigate Bolt-specific
CI failures. After each PR is merged into the bytedance/bolt repository,
Bolt CI will check out the latest Gluten main branch and run the Bolt
backend unit tests against it to detect compatibility issues early.

*3. Feedback requested*
We would appreciate feedback on whether any concerns remain with the
proposed integration, maintenance commitment, or CI commitment. If so,
please identify the specific concern and, if possible, the change needed to
address it.
After the discussion has converged, we plan to start a separate *[VOTE]*
 thread.
Original Bolt backend discussion:
https://github.com/apache/gluten/discussions/10929
Previous vote thread (withdrawn):
https://lists.apache.org/thread/lh5gos0hrg79pnhmjcpkv83fk6pwp9lq
Thank you for reviewing the revised proposal.

Best regards,
Haiyan Gu