You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2023/06/20 04:27:11 UTC
[GitHub] [incubator-paimon] leaves12138 opened a new pull request, #1400: [flink] set spillable default on for batch mode
leaves12138 opened a new pull request, #1400:
URL: https://github.com/apache/incubator-paimon/pull/1400
KeyValueFileStoreWrite could enable spillable in batch mode by default.
We add optimizeForBatch interface to invoke this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1400: [flink] set spillable default on for batch mode
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1400:
URL: https://github.com/apache/incubator-paimon/pull/1400#discussion_r1240996251
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreWrite.java:
##########
@@ -98,6 +98,15 @@ public interface FileStoreWrite<T> {
List<CommitMessage> prepareCommit(boolean waitCompaction, long commitIdentifier)
throws Exception;
+ /**
+ * We detect whether it is in batch mode, if so, we do some optimization.
+ *
+ * @param isBatch whether in batch mode
+ */
+ default void isStreamingMode(boolean isBatch) {
Review Comment:
And don't use default method in per record code path. (It has performance issue)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] leaves12138 commented on a diff in pull request #1400: [flink] set spillable default on for batch mode
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #1400:
URL: https://github.com/apache/incubator-paimon/pull/1400#discussion_r1234737128
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/StoreSinkWriteImpl.java:
##########
@@ -83,7 +86,8 @@ private TableWriteImpl<?> newTableWrite(FileStoreTable table) {
: new HeapMemorySegmentPool(
table.coreOptions().writeBufferSize(),
table.coreOptions().pageSize()))
- .withIgnorePreviousFiles(ignorePreviousFiles);
+ .withIgnorePreviousFiles(ignorePreviousFiles)
+ .optimizeForBatch(!isStreamingMode);
Review Comment:
do optimize for batch here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1400: [flink] set spillable default on for batch mode
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1400:
URL: https://github.com/apache/incubator-paimon/pull/1400#discussion_r1234857143
##########
paimon-core/src/main/java/org/apache/paimon/operation/KeyValueFileStoreWrite.java:
##########
@@ -174,6 +175,11 @@ protected MergeTreeWriter createWriter(
restoreIncrement);
}
+ @Override
+ public void optimizeForBatch(boolean isBatch) {
Review Comment:
inStreamingMode(Boolean isStreamMode)
##########
paimon-core/src/main/java/org/apache/paimon/operation/KeyValueFileStoreWrite.java:
##########
@@ -161,7 +162,7 @@ protected MergeTreeWriter createWriter(
CompactManager compactManager =
createCompactManager(partition, bucket, compactStrategy, compactExecutor, levels);
return new MergeTreeWriter(
- bufferSpillable(),
+ bufferSpillable() || forceSpillable,
Review Comment:
Add isStreaming to `options.writeBufferSpillable` method.
```
public boolean writeBufferSpillable(boolean usingObjectStore, boolean isStreaming) {
return options.getOptional(WRITE_BUFFER_SPILLABLE).orElse(usingObjectStore || isStreaming);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi merged pull request #1400: [flink] set spillable default on for batch mode
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #1400:
URL: https://github.com/apache/incubator-paimon/pull/1400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1400: [flink] set spillable default on for batch mode
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1400:
URL: https://github.com/apache/incubator-paimon/pull/1400#discussion_r1240992935
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreWrite.java:
##########
@@ -98,6 +98,15 @@ public interface FileStoreWrite<T> {
List<CommitMessage> prepareCommit(boolean waitCompaction, long commitIdentifier)
throws Exception;
+ /**
+ * We detect whether it is in batch mode, if so, we do some optimization.
+ *
+ * @param isBatch whether in batch mode
+ */
+ default void isStreamingMode(boolean isBatch) {
Review Comment:
batch or streaming?
##########
paimon-core/src/main/java/org/apache/paimon/operation/KeyValueFileStoreWrite.java:
##########
@@ -161,7 +163,7 @@ protected MergeTreeWriter createWriter(
CompactManager compactManager =
createCompactManager(partition, bucket, compactStrategy, compactExecutor, levels);
return new MergeTreeWriter(
- bufferSpillable(),
+ bufferSpillable() || isStreamingMode,
Review Comment:
revert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org