You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2023/10/09 04:02:48 UTC

[PR] [flink] Introduce zorder sort for dynamic bucket table [incubator-paimon]

leaves12138 opened a new pull request, #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100

   
   ### Purpose
   
   Speed up query performance with dynamic table.
   
   We rewrite the dynamic table after z-order sorting the records, with the bucket filter function, we can speed up query. (The whole bucket may be filtered)
   
   ### Tests
   
   End to end test in my pc.
   
   SQL:
   `DROP TABLE IF EXISTS Orders1;
   CREATE TABLE IF NOT EXISTS Orders1 (
       pk        INT,
       f0        INT,
       f1        INT,
       v         STRING,
       PRIMARY KEY(pk) NOT ENFORCED
   )
   WITH (
   'bucket' = '-1',
   'dynamic-bucket.target-row-num' = '10000'
   );
   
   
   CREATE TEMPORARY TABLE Orders_in (
       pk        INT,
       f0        INT,
       f1        INT,
       v         STRING
   ) WITH (
       'connector' = 'datagen',
       'fields.f0.min'   = '0',
       'fields.f0.max'   = '10000',
       'fields.f1.min'   = '0',
       'fields.f1.max'   = '10000',
       'number-of-rows' = '1000000'
   );
   
   SET 'parallelism.default' = '2';
   SET 'execution.checkpointing.interval' = '20 s';
   
   SET 'execution.runtime-mode' = 'streaming';
   SET 'sql-client.execution.result-mode' = 'TABLEAU';
   
   INSERT INTO Orders1 SELECT * FROM Orders_in;`
   
   Query sql:
   the query sql `SELECT * FROM Orders1 where f1 < 200 and f1 > 100;`
   
   Before sort:
   take 2.5 s to get result, no bucket was filtered by value push down
   ![image](https://github.com/apache/incubator-paimon/assets/41894543/836e2196-d784-4814-a417-6fb68dbab06e)
   
   
   After sort:
   take 0.5s to get result, only 17% data was passed, 83% data was filtered by value push down.
   ![image](https://github.com/apache/incubator-paimon/assets/41894543/0d83b6ef-3c84-4bf8-832f-9dea4a8016c0)
   
   
   ### API and Format
   
   Usage example:
   flink run ./paimon-flink-action-0.6-SNAPSHOT.jar compact \
   --warehouse \
   /Users/paimontest/DynamicBucket \
   --database \
   my_db \
   --table \
   Orders1 \
   --order-strategy zorder \
   --order-by f0,f1
   
   No other influence to user.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100#discussion_r1349873623


##########
paimon-core/src/main/java/org/apache/paimon/index/SimpleBucketAssigner.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.index;
+
+import org.apache.paimon.data.BinaryRow;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/** Only for batch sort compact bucket assign. */
+public class SimpleBucketAssigner implements BucketAssigner {

Review Comment:
   For hash collision, you can not just use a new simple bucket assigner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100#discussion_r1349926688


##########
paimon-core/src/main/java/org/apache/paimon/index/SimpleHashBucketAssigner.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.index;
+
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.utils.Int2ShortHashMap;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/** Only for batch sort compact bucket assign. */

Review Comment:
   overwrite also can use this one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100#discussion_r1350133279


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/BucketAssignerOperator.java:
##########
@@ -32,7 +34,7 @@
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 
 /** Assign bucket for the input record, output record with bucket. */
-public class HashBucketAssignerOperator<T> extends AbstractStreamOperator<Tuple2<T, Integer>>
+public class BucketAssignerOperator<T> extends AbstractStreamOperator<Tuple2<T, Integer>>

Review Comment:
   revert this name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100#discussion_r1350095057


##########
paimon-core/src/main/java/org/apache/paimon/index/SimpleHashBucketAssigner.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.index;
+
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.utils.Int2ShortHashMap;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/** Only for batch sort compact bucket assign. */

Review Comment:
   done
   



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSinkBuilder.java:
##########
@@ -44,6 +44,7 @@ public class FlinkSinkBuilder {
     @Nullable private Map<String, String> overwritePartition;
     @Nullable private LogSinkFunction logSinkFunction;
     @Nullable private Integer parallelism;
+    private boolean compactSink = false;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100#discussion_r1351200088


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSinkBuilder.java:
##########
@@ -108,9 +108,13 @@ public DataStreamSink<?> build() {
     private DataStreamSink<?> buildDynamicBucketSink(
             DataStream<InternalRow> input, boolean globalIndex) {
         checkArgument(logSinkFunction == null, "Dynamic bucket mode can not work with log system.");
-        return globalIndex
-                ? new GlobalDynamicBucketSink(table, overwritePartition).build(input, parallelism)
-                : new RowDynamicBucketSink(table, overwritePartition).build(input, parallelism);
+        return overwritePartition != null

Review Comment:
   Added a todo here for `Global Index Sort Compact`, because Global Index table sort compact will need extra tests and   
   and end-to-end tests to guarantee it works.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100#discussion_r1350095518


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/DynamicBucketCompactSink.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSink;
+
+import javax.annotation.Nullable;
+
+import java.util.Map;
+import java.util.UUID;
+
+/** This class is only used for generate compact sink topology for dynamic bucket table. */
+public class DynamicBucketCompactSink extends RowDynamicBucketSink {
+
+    public DynamicBucketCompactSink(
+            FileStoreTable table, @Nullable Map<String, String> overwritePartition) {
+        super(table, overwritePartition);
+    }
+
+    @Override
+    public DataStreamSink<?> build(DataStream<InternalRow> input, @Nullable Integer parallelism) {
+        String initialCommitUser = UUID.randomUUID().toString();
+
+        // This input is sorted and compacted. So there is no shuffle here, we just assign bucket
+        // for each record, and sink them to table.
+
+        // bucket-assigner
+        BucketAssignerOperator<InternalRow> assignerOperator =
+                new BucketAssignerOperator<>(initialCommitUser, table, extractorFunction(), true);
+        TupleTypeInfo<Tuple2<InternalRow, Integer>> rowWithBucketType =
+                new TupleTypeInfo<>(input.getType(), BasicTypeInfo.INT_TYPE_INFO);
+        DataStream<Tuple2<InternalRow, Integer>> bucketAssigned =
+                input.transform("dynamic-bucket-assigner", rowWithBucketType, assignerOperator)
+                        .setParallelism(input.getParallelism());
+        return sinkFrom(bucketAssigned, initialCommitUser);

Review Comment:
   fixed this bug and added two tests for this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100#discussion_r1350136792


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSinkBuilder.java:
##########
@@ -108,9 +108,13 @@ public DataStreamSink<?> build() {
     private DataStreamSink<?> buildDynamicBucketSink(
             DataStream<InternalRow> input, boolean globalIndex) {
         checkArgument(logSinkFunction == null, "Dynamic bucket mode can not work with log system.");
-        return globalIndex
-                ? new GlobalDynamicBucketSink(table, overwritePartition).build(input, parallelism)
-                : new RowDynamicBucketSink(table, overwritePartition).build(input, parallelism);
+        return overwritePartition != null

Review Comment:
   Here should only work for `compact & !globalIndex`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100#discussion_r1349930990


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/DynamicBucketCompactSink.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.flink.sink;
+
+import org.apache.paimon.data.InternalRow;
+import org.apache.paimon.table.FileStoreTable;
+
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSink;
+
+import javax.annotation.Nullable;
+
+import java.util.Map;
+import java.util.UUID;
+
+/** This class is only used for generate compact sink topology for dynamic bucket table. */
+public class DynamicBucketCompactSink extends RowDynamicBucketSink {
+
+    public DynamicBucketCompactSink(
+            FileStoreTable table, @Nullable Map<String, String> overwritePartition) {
+        super(table, overwritePartition);
+    }
+
+    @Override
+    public DataStreamSink<?> build(DataStream<InternalRow> input, @Nullable Integer parallelism) {
+        String initialCommitUser = UUID.randomUUID().toString();
+
+        // This input is sorted and compacted. So there is no shuffle here, we just assign bucket
+        // for each record, and sink them to table.
+
+        // bucket-assigner
+        BucketAssignerOperator<InternalRow> assignerOperator =
+                new BucketAssignerOperator<>(initialCommitUser, table, extractorFunction(), true);
+        TupleTypeInfo<Tuple2<InternalRow, Integer>> rowWithBucketType =
+                new TupleTypeInfo<>(input.getType(), BasicTypeInfo.INT_TYPE_INFO);
+        DataStream<Tuple2<InternalRow, Integer>> bucketAssigned =
+                input.transform("dynamic-bucket-assigner", rowWithBucketType, assignerOperator)
+                        .setParallelism(input.getParallelism());
+        return sinkFrom(bucketAssigned, initialCommitUser);

Review Comment:
   Add test for overwritten, it seems that there is bug in AbstractFileStoreWrite for index maintaining.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100#discussion_r1349928090


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSinkBuilder.java:
##########
@@ -44,6 +44,7 @@ public class FlinkSinkBuilder {
     @Nullable private Map<String, String> overwritePartition;
     @Nullable private LogSinkFunction logSinkFunction;
     @Nullable private Integer parallelism;
+    private boolean compactSink = false;

Review Comment:
   You should just use overwrite to judge.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100#discussion_r1351200307


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSinkBuilder.java:
##########
@@ -108,9 +108,13 @@ public DataStreamSink<?> build() {
     private DataStreamSink<?> buildDynamicBucketSink(
             DataStream<InternalRow> input, boolean globalIndex) {
         checkArgument(logSinkFunction == null, "Dynamic bucket mode can not work with log system.");
-        return globalIndex
-                ? new GlobalDynamicBucketSink(table, overwritePartition).build(input, parallelism)
-                : new RowDynamicBucketSink(table, overwritePartition).build(input, parallelism);
+        return overwritePartition != null

Review Comment:
   done this judge
   



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/BucketAssignerOperator.java:
##########
@@ -32,7 +34,7 @@
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 
 /** Assign bucket for the input record, output record with bucket. */
-public class HashBucketAssignerOperator<T> extends AbstractStreamOperator<Tuple2<T, Integer>>
+public class BucketAssignerOperator<T> extends AbstractStreamOperator<Tuple2<T, Integer>>

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Introduce zorder/order sort compact for dynamic bucket table [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #2100:
URL: https://github.com/apache/incubator-paimon/pull/2100


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org