You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "KevinyhZou (via GitHub)" <gi...@apache.org> on 2026/02/09 08:24:09 UTC
[PR] [FLINK][GLUTEN-11588]Support rocksdb state for window operator [incubator-gluten]
KevinyhZou opened a new pull request, #11589:
URL: https://github.com/apache/incubator-gluten/pull/11589
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/incubator-gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the corresponding issue
when the PR is merged.
-->
support rocksdb state for window operator
this pr relies on: https://github.com/bigo-sg/velox4j/pull/21 https://github.com/bigo-sg/velox/pull/21
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
manual tests
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11588][FLINK]Support rocksdb state for window operator [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on code in PR #11589:
URL: https://github.com/apache/incubator-gluten/pull/11589#discussion_r2867118094
##########
gluten-flink/runtime/src/main/java/org/apache/gluten/client/OffloadedJobGraphGenerator.java:
##########
@@ -248,6 +249,21 @@ private void createOffloadedOneInputOperator(
inClass,
outClass,
sourceOperator.getDescription());
+ if (sourceOperator instanceof WindowAggOperator) {
Review Comment:
should use if-else
```java
if (sourceOperator instanceof WindowAggOperator) {
// create new windowAggOperator
offloadedOpConfig.setStreamOperator(newOneInputOp);
} else {
// the default handle
offloadedOpConfig.setStreamOperator(newOneInputOp);
}
```
##########
gluten-flink/runtime/src/main/java/org/apache/gluten/client/OffloadedJobGraphGenerator.java:
##########
@@ -248,6 +249,21 @@ private void createOffloadedOneInputOperator(
inClass,
outClass,
sourceOperator.getDescription());
+ if (sourceOperator instanceof WindowAggOperator) {
Review Comment:
Let's make the code simpler
1. Add a new method in `GlutenOneInputOperator`
```java
public <NIN, NOUT> GlutenOneInputOperator<NIN, NOUT> cloneWith(
Class<NIN> newInClass, Class<NOUT> newOutClass) {
return new GlutenOneInputOperator<>(
this.glutenPlan,
this.id,
this.inputType,
this.outputTypes,
newInClass,
newOutClass,
this.description);
}
```
2. Override `cloneWith` in `WindowAggOperator`
```java
@Override
public <NIN, NOUT> WindowAggOperator<NIN, NOUT, W> cloneWith(
Class<NIN> newInClass, Class<NOUT> newOutClass) {
return new WindowAggOperator<>(
getPlanNode(),
getId(),
getInputType(),
getOutputTypes(),
newInClass,
newOutClass,
getDescription(),
keyType,
accNames,
accTypes);
}
```
3. Then only need to make following in `OffloadedJobGraphGenerator`
```java
GlutenOneInputOperator<?, ?> newOneInputOp = sourceOperator.cloneWith(inClass, outClass);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11588][FLINK]Support rocksdb state for window operator [incubator-gluten]
Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on PR #11589:
URL: https://github.com/apache/incubator-gluten/pull/11589#issuecomment-4001430762
@PHILO-HE can you help to review this pr? thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11588][FLINK] Support rocksdb state for window operator [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE merged PR #11589:
URL: https://github.com/apache/incubator-gluten/pull/11589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [FLINK][GLUTEN-11588]Support rocksdb state for window operator [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on code in PR #11589:
URL: https://github.com/apache/incubator-gluten/pull/11589#discussion_r2785351324
##########
gluten-flink/docs/Flink.md:
##########
@@ -140,6 +140,30 @@ bin/sql-client.sh -f data-generator.sql
TODO
+### RocksDB State
+
+**Get & compile RocksDB**
+```bash
+wget https://github.com/ververica/frocksdb/archive/refs/heads/FRocksDB-6.20.3.zip
Review Comment:
add the unzip command
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11588][FLINK]Support rocksdb state for window operator [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on code in PR #11589:
URL: https://github.com/apache/incubator-gluten/pull/11589#discussion_r2867153496
##########
gluten-flink/runtime/src/main/java/org/apache/gluten/client/OffloadedJobGraphGenerator.java:
##########
@@ -248,6 +249,21 @@ private void createOffloadedOneInputOperator(
inClass,
outClass,
sourceOperator.getDescription());
+ if (sourceOperator instanceof WindowAggOperator) {
Review Comment:
Let's make the code simpler
1. Add a new method in `GlutenOneInputOperator`
```java
public <NIN, NOUT> GlutenOneInputOperator<NIN, NOUT> cloneWithInputOutputClasses(
Class<NIN> newInClass, Class<NOUT> newOutClass) {
return new GlutenOneInputOperator<>(
this.glutenPlan,
this.id,
this.inputType,
this.outputTypes,
newInClass,
newOutClass,
this.description);
}
```
2. Override `cloneWithInputOutputClasses` in `WindowAggOperator`
```java
@Override
public <NIN, NOUT> WindowAggOperator<NIN, NOUT, W> cloneWithInputOutputClasses(
Class<NIN> newInClass, Class<NOUT> newOutClass) {
return new WindowAggOperator<>(
getPlanNode(),
getId(),
getInputType(),
getOutputTypes(),
newInClass,
newOutClass,
getDescription(),
keyType,
accNames,
accTypes);
}
```
3. Then only need to make following in `OffloadedJobGraphGenerator`
```java
GlutenOneInputOperator<?, ?> newOneInputOp = sourceOperator.cloneWithInputOutputClasses(inClass, outClass);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11588][FLINK] Support rocksdb state for window operator [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on code in PR #11589:
URL: https://github.com/apache/incubator-gluten/pull/11589#discussion_r2888481941
##########
.github/workflows/util/install-flink-resources.sh:
##########
@@ -84,18 +91,25 @@ function run_and_time {
}
function install_librdkafka {
- wget_and_untar https://github.com/confluentinc/librdkafka/archive/refs/tags/${LIBRDKAFKA_VERSION}.tar.gz librdkafka
+ wget_and_untar https://github.com/confluentinc/librdkafka/archive/refs/tags/${LIBRDKAFKA_VERSION}.tar.gz librdkafka tar.gz
cmake_install_dir librdkafka -DBUILD_TESTS=OFF
}
function install_cppkafka {
- wget_and_untar https://github.com/mfontanini/cppkafka/archive/refs/tags/${CPPKAFKA_VERSION}.tar.gz cppkafka
+ wget_and_untar https://github.com/mfontanini/cppkafka/archive/refs/tags/${CPPKAFKA_VERSION}.tar.gz cppkafka tar.gz
cmake_install_dir cppkafka -DBUILD_TESTS=OFF
}
+function install_rocksdb {
+ wget_and_untar https://github.com/ververica/frocksdb/archive/refs/heads/FRocksDB-${FROCKSDB_VERSION}.zip "" zip
Review Comment:
For downloading github repo code, maybe, better to add and use github_checkout function. Then, keep wget_and_untar unchanged.
https://github.com/facebookincubator/velox/blob/7bc1be16af78637be17ee4950a68b43250c3d44d/scripts/setup-helper-functions.sh#L53
##########
.github/workflows/flink.yml:
##########
@@ -61,8 +61,8 @@ jobs:
sudo dnf install -y patchelf
sudo yum install https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/tzdata-2025a-1.el9.noarch.rpm -y
sudo .github/workflows/util/install-flink-resources.sh
- git clone -b gluten-0530 https://github.com/bigo-sg/velox4j.git
- cd velox4j && git reset --hard 288d181a1b05c47f1f17339eb498dd6375f7aec8
+ git clone -b fix_state_init https://github.com/KevinyhZou/velox4j.git
Review Comment:
Recommend to use organization repo. And please also make it consistent with the commit ID provided in Flink.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11588][FLINK] Support rocksdb state for window operator [incubator-gluten]
Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on PR #11589:
URL: https://github.com/apache/incubator-gluten/pull/11589#issuecomment-4010095451
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org