You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "yuzelin (via GitHub)" <gi...@apache.org> on 2024/03/22 06:52:10 UTC

[PR] [flink] Fix that compact actions haven't handled scan parallelism [incubator-paimon]

yuzelin opened a new pull request, #3077:
URL: https://github.com/apache/incubator-paimon/pull/3077

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Sort compact action have handled scan.parallelism but normal compact action haven't.
   
   <!-- What is the purpose of the change -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Fix that compact actions haven't handled scan parallelism [incubator-paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #3077:
URL: https://github.com/apache/incubator-paimon/pull/3077#discussion_r1535291669


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/source/BucketUnawareCompactSource.java:
##########
@@ -117,15 +117,21 @@ public static DataStreamSource<AppendOnlyCompactionTask> buildSource(
             StreamExecutionEnvironment env,
             BucketUnawareCompactSource source,
             boolean streaming,
+            @Nullable Integer parallelism,
             String tableIdentifier) {
         final StreamSource<AppendOnlyCompactionTask, BucketUnawareCompactSource> sourceOperator =
                 new StreamSource<>(source);
-        return new DataStreamSource<>(
-                env,
-                new CompactionTaskTypeInfo(),
-                sourceOperator,
-                false,
-                COMPACTION_COORDINATOR_NAME + " : " + tableIdentifier,
-                streaming ? Boundedness.CONTINUOUS_UNBOUNDED : Boundedness.BOUNDED);
+        DataStreamSource<AppendOnlyCompactionTask> dataStream =
+                new DataStreamSource<>(
+                        env,
+                        new CompactionTaskTypeInfo(),
+                        sourceOperator,
+                        false,
+                        COMPACTION_COORDINATOR_NAME + " : " + tableIdentifier,
+                        streaming ? Boundedness.CONTINUOUS_UNBOUNDED : Boundedness.BOUNDED);
+        if (parallelism != null) {
+            dataStream.setParallelism(parallelism);

Review Comment:
   Source for unaware bucket is not parallelism.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [flink] Fix that compact actions haven't handled scan parallelism [incubator-paimon]

Posted by "yuzelin (via GitHub)" <gi...@apache.org>.
yuzelin merged PR #3077:
URL: https://github.com/apache/incubator-paimon/pull/3077


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org