You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/08/02 13:03:45 UTC

[GitHub] [iceberg] hameizi opened a new pull request #2916: Records num threshold

hameizi opened a new pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916


   Add configuration write.records-num-threshold to control how many rows data will open new pos-delete file. Then we can control pos-delete file numbers.
   When iceberg write pos-delete file there is one default value 'DEFAULT_RECORDS_NUM_THRESHOLD = 100_000L'. When pos-delete records number attain 100_000 will open one new file, so when there is too much pos-delete records will create too much pos-delete. So in this PR add configuration write.records-num-threshold that users can control pos-delete file number.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] hameizi commented on pull request #2916: Add one configuration to config the records num threshold when write pos-delete files

Posted by GitBox <gi...@apache.org>.
hameizi commented on pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916#issuecomment-891572481


   > I have another question. why `DEFAULT_RECORDS_NUM_THRESHOLD` is set to `100_000L`. Is there any experience about how to set this value?
   
   This value depend on the scene that whether your flink job will update one primary key frequently in one checkpoint and whether you have enough memory in flink job because these pos-delete is cached in memory before they are flush in pos-delete file. If in your sence you found there is too much pos-delete in one checkpoint inteval you can try config this configuration and add you flink job memory. At most sence 100_000 is enough. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] hameizi commented on a change in pull request #2916: Add one configuration to config the records num threshold when write pos-delete files

Posted by GitBox <gi...@apache.org>.
hameizi commented on a change in pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916#discussion_r687430019



##########
File path: core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java
##########
@@ -105,7 +112,11 @@ protected BaseEqualityDeltaWriter(StructLike partition, Schema schema, Schema de
 
       this.dataWriter = new RollingFileWriter(partition);
       this.eqDeleteWriter = new RollingEqDeleteWriter(partition);
-      this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory, fileFactory, format, partition);
+      this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory,
+          fileFactory,
+          format,
+          partition,
+          getRecordsNumThreshold(properties));

Review comment:
       > > Sometimes certain maps made by Guava etc (ImmutableMap comes to mind) are not serializable, particularly when using Kryo (which is not the default but the de facto default with Spark).
   > 
   > I'm sorry i did't test this, i will try test these case. And pass around table properties conveniency for develop later. If there is some bug to pass around table properties i can fix it to just the value of this propertie.
   
   @kbendick I found all my change of this PR just RowDataTaskWriterFactory is serializable, but it will not be serializd by Kryo in flink, so i think there is no worry in Kyro serialize in this PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] hameizi commented on pull request #2916: Add one configuration to config the records num threshold when write pos-delete files

Posted by GitBox <gi...@apache.org>.
hameizi commented on pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916#issuecomment-895805995


   > > When iceberg write pos-delete file there is one default value 'DEFAULT_RECORDS_NUM_THRESHOLD = 100_000L'. When pos-delete records number attain 100_000 will open one new file, so when there is too much pos-delete records will create too much pos-delete. So in this PR add configuration write.records-num-threshold that users can control pos-delete file number.
   > 
   > When you say that there is one default value, does this value previously exist (and is simply not exposed)? I'm not seeing any reference to anything resembling this config in the diff.
   
   @kbendick Yes, there is one default value previously exist and it is 100_000L.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] hameizi commented on pull request #2916: Add one configuration to config the records num threshold when write pos-delete files

Posted by GitBox <gi...@apache.org>.
hameizi commented on pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916#issuecomment-919636735


   @kbendick Could you help review continue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] kbendick commented on pull request #2916: Add one configuration to config the records num threshold when write pos-delete files

Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916#issuecomment-895801700


   > When iceberg write pos-delete file there is one default value 'DEFAULT_RECORDS_NUM_THRESHOLD = 100_000L'. When pos-delete records number attain 100_000 will open one new file, so when there is too much pos-delete records will create too much pos-delete. So in this PR add configuration write.records-num-threshold that users can control pos-delete file number.
   
   When you say that there is one default value, does this value previously exist (and is simply not exposed)? I'm not seeing any reference to anything resembling this config in the diff. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] hameizi commented on a change in pull request #2916: Add one configuration to config the records num threshold when write pos-delete files

Posted by GitBox <gi...@apache.org>.
hameizi commented on a change in pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916#discussion_r685778592



##########
File path: core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java
##########
@@ -105,7 +112,11 @@ protected BaseEqualityDeltaWriter(StructLike partition, Schema schema, Schema de
 
       this.dataWriter = new RollingFileWriter(partition);
       this.eqDeleteWriter = new RollingEqDeleteWriter(partition);
-      this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory, fileFactory, format, partition);
+      this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory,
+          fileFactory,
+          format,
+          partition,
+          getRecordsNumThreshold(properties));

Review comment:
       > Sometimes certain maps made by Guava etc (ImmutableMap comes to mind) are not serializable, particularly when using Kryo (which is not the default but the de facto default with Spark).
   
   I'm sorry i did't test this, i will try test these case. And pass around table properties  conveniency for develop later. If there is some bug to pass around table properties i can fix it to just the value of this propertie.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] hameizi commented on a change in pull request #2916: Add one configuration to config the records num threshold when write pos-delete files

Posted by GitBox <gi...@apache.org>.
hameizi commented on a change in pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916#discussion_r685774699



##########
File path: core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java
##########
@@ -105,7 +112,11 @@ protected BaseEqualityDeltaWriter(StructLike partition, Schema schema, Schema de
 
       this.dataWriter = new RollingFileWriter(partition);
       this.eqDeleteWriter = new RollingEqDeleteWriter(partition);
-      this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory, fileFactory, format, partition);
+      this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory,
+          fileFactory,
+          format,
+          partition,
+          getRecordsNumThreshold(properties));

Review comment:
       > Nit: This function name is admittedly confusing for me.
   
   This function is just one private function, in it's class maybe this name is clear.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] coolderli commented on pull request #2916: Add one configuration to config the records num threshold when write pos-delete files

Posted by GitBox <gi...@apache.org>.
coolderli commented on pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916#issuecomment-891562500


   I have another question. why `DEFAULT_RECORDS_NUM_THRESHOLD` is set to `100_000L`. Is there any experience about how to set this value?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] kbendick commented on a change in pull request #2916: Add one configuration to config the records num threshold when write pos-delete files

Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916#discussion_r685765465



##########
File path: core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java
##########
@@ -105,7 +112,11 @@ protected BaseEqualityDeltaWriter(StructLike partition, Schema schema, Schema de
 
       this.dataWriter = new RollingFileWriter(partition);
       this.eqDeleteWriter = new RollingEqDeleteWriter(partition);
-      this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory, fileFactory, format, partition);
+      this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory,
+          fileFactory,
+          format,
+          partition,
+          getRecordsNumThreshold(properties));

Review comment:
       Nit: This function name is admittedly confusing for me.
   
   Also, we're passing around a potentially large table properties map across a large number of frameworks where previously we only passed around Iceberg specific classes. I'm not so worried about the size (though it does seem potentially wasteful to be passing the whole table properties map around when only one field is presently needed). I'm more worried about serializability concerns. Sometimes certain maps made by Guava etc (ImmutableMap comes to mind) are not serializable, particularly when using Kryo (which is not the default but the de facto default with Spark).
   
   Do we have any tests to ensure these new changes can be serialized? Or at the least, if you're going to pass around the whole table properties map like this, can you check that serialization doesn't break when using Kryo - I think there are some unit tests for checking kryo and java serde with Spark. Essentially, just like with Flink, the driver needs to be able to serialize all of this code and send it to the executors (similar to how the job manager sends generated code to the task managers responsible for individual subtasks).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org