You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2024/03/04 05:38:03 UTC

[PR] [core][bug] Write Spillable with memory preemption has bug : preempt memory will cause writer flush memory to disk which should not happen [incubator-paimon]

leaves12138 opened a new pull request, #2940:
URL: https://github.com/apache/incubator-paimon/pull/2940

   
   Fix bug when open spillable, still many small files generated.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core][bug] Write Spillable with memory preemption has bug : preempt memory will cause writer flush memory to disk which should not happen [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #2940:
URL: https://github.com/apache/incubator-paimon/pull/2940


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core][bug] Write Spillable with memory preemption has bug : preempt memory will cause writer flush memory to disk which should not happen [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #2940:
URL: https://github.com/apache/incubator-paimon/pull/2940#issuecomment-1976592105

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core][bug] Write Spillable with memory preemption has bug : preempt memory will cause writer flush memory to disk which should not happen [incubator-paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2940:
URL: https://github.com/apache/incubator-paimon/pull/2940#discussion_r1510614986


##########
paimon-core/src/main/java/org/apache/paimon/append/AppendOnlyWriter.java:
##########
@@ -279,7 +279,11 @@ public long memoryOccupancy() {
 
     @Override
     public void flushMemory() throws Exception {
-        flush(false, false);
+        if (sinkWriter.bufferSpillableWriter()) {

Review Comment:
   just same to `MergeTreeWriter`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core][bug] Write Spillable with memory preemption has bug : preempt memory will cause writer flush memory to disk which should not happen [incubator-paimon]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #2940:
URL: https://github.com/apache/incubator-paimon/pull/2940#discussion_r1510627513


##########
paimon-core/src/main/java/org/apache/paimon/append/AppendOnlyWriter.java:
##########
@@ -279,7 +279,11 @@ public long memoryOccupancy() {
 
     @Override
     public void flushMemory() throws Exception {
-        flush(false, false);
+        if (sinkWriter.bufferSpillableWriter()) {

Review Comment:
   `MergeTreeWriter` seems don't have this bug.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org