[D] March 13, 2026: Weekly Status Update in Gluten [gluten]

["GlutenPerfBot (via GitHub)" <gi...@apache.org>]

GitHub user GlutenPerfBot created a discussion: March 13, 2026: Weekly Status Update in Gluten

*This weekly update is generated by LLMs. You're welcome to join our [Github](https://github.com/apache/incubator-gluten/discussions) for in-depth discussions.*

## Overall Activity Summary
The Gluten project celebrated its graduation as an Apache Top-Level Project (TLP) this week, with extensive cleanup efforts to remove "incubator" references across the codebase. Development activity remained strong with 47 merged PRs and 21 open PRs, focusing heavily on Velox backend improvements, Spark 4.x compatibility, and memory management optimizations.

## Key Ongoing Projects
- **TLP Graduation Cleanup**: Major effort by @weiting-chen and team to remove all incubator references from source code, CI workflows, Dockerfiles, and documentation (#11735, #11737, #11738, #11739, #11741, #11742)
- **Spark 4.x Test Suite Enablement**: @baibaichen continues enabling disabled test suites, with significant progress on variant handling and XML functions (#11726, #11727, #11725)
- **Velox Backend Optimization**: Multiple performance improvements including hash join optimizations (#8931 by @JkSelf), dynamic filter pushdown (#11657 by @acvictor), and memory management enhancements
- **Parquet Type Widening Support**: @baibaichen leading effort to enable GlutenParquetTypeWideningSuite with 84 tests for Spark 4.x compatibility (#11683, #11684, #11689, #11719)

## Priority Items
- **Memory Management Issues**: Critical OOM issues reported (#11747 by @FelixYBW) requiring immediate attention for production deployments
- **Dynamic Partition Pruning Regression**: Hive scan DPP failures (#11692 by @manikumararyas) affecting Spark 4.x users
- **Columnar Shuffle Buffer Issues**: DirectByteBuffer availability problems (#11716 by @ijbgreen) blocking columnar shuffle adoption
- **Complex Type Support**: Native write validation incorrectly rejecting supported ArrayType/MapType/StructType (#11746 by @VvanFalleaves)

## Notable Discussions
- #11585: Comprehensive tracking of useful Velox PRs not yet merged upstream, providing visibility into potential performance improvements
- #8429: New ASF Slack channel setup for community communication and support
- #11550: Spark 4.x disabled test suite tracking showing 51 unique suites need attention across both Spark versions

## Emerging Trends
1. **Graduation Momentum**: Strong community response to TLP status with increased contribution activity
2. **Spark 4.x Focus**: Major push to achieve full Spark 4.0/4.1 compatibility with 79% of disabled test suites now enabled
3. **Memory Optimization**: Multiple PRs addressing memory management, off-heap allocation, and OOM prevention
4. **Velox Integration Deepening**: Daily Velox version updates and close collaboration on upstream features

## Good First Issues
- #11622: Support TIMESTAMP_NTZ Type - Well-defined scope with clear implementation steps for Velox backend
- #11703: Map iceberg configuration with Velox configuration - Straightforward configuration mapping task with clear requirements
- #10134: Add ANSI mode support - Large but well-organized tracking issue with multiple sub-tasks suitable for new contributors
- #11699: S3 IMDS configuration support - Simple configuration addition task with clear implementation path

GitHub link: https://github.com/apache/gluten/discussions/11761

----
This is an automatically sent email for dev@gluten.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gluten.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@gluten.apache.org
For additional commands, e-mail: dev-help@gluten.apache.org