You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/08/04 09:29:02 UTC

[GitHub] [iceberg] ConeyLiu opened a new pull request, #5435: Parquet: Set parquet bloom filter config with compatible column name

ConeyLiu opened a new pull request, #5435:
URL: https://github.com/apache/iceberg/pull/5435

   This PR improves the config setting for the parquet bloom filter. 
   1. Log warn for those don't exist column or the column type is unsupported (boolean or complex data type).
   2. Convert the column name to a compatible one. This is because we make the column name compatible if it contains special character when we write the parquet file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r939697256


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,46 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");

Review Comment:
   Constants should not be created as local variables.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r938432420


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,42 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);
+                if (field == null) {
+                  LOG.warn(
+                      "Column: {} is not found in the schema: {}, ignore it.", columnPath, schema);
+                  return;
+                }
+                Type fieldType = field.type();
+                // doesn't support boolean, list, map, struct type
+                if (!fieldType.isPrimitiveType()
+                    || fieldType.typeId().equals(Type.TypeID.BOOLEAN)) {
+                  LOG.warn(
+                      "The type: {} of column: {} is not supported for parquet bloom filter, ignore it",
+                      fieldType,
+                      columnPath);
+                  return;

Review Comment:
   Thanks @kbendick for reviewing.
   
   > Is this something we can prevent the user from doing? That might not be worth it though.
   
   I think we may add some checks for adding/updating table properties.
   
   > Id also format it as Invalid bloom filter column configuration: field %s is of type '%s' which cannot benefit from a bloom filter. This column configuration will be ignored.. I am a fan of a full constant string at the front for searching logs.
   
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r939698212


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,46 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);
+                if (field == null) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration, field: {} is not found in the schema: {}, "
+                          + "this column configuration will be ignored.",
+                      columnPath,
+                      schema);
+                  return;
+                }
+                Type fieldType = field.type();
+                // doesn't support boolean, list, map, struct type
+                if (!fieldType.isPrimitiveType()
+                    || fieldType.typeId().equals(Type.TypeID.BOOLEAN)) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration: field {} is of type '{}' which cannot "
+                          + "benefit from a bloom filter. This column configuration will be ignored.",
+                      columnPath,
+                      fieldType);

Review Comment:
   I don't think this is worth checking.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r939808320


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,46 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);

Review Comment:
   I think it should do in setting/updating properties. Actually, we already do some simple checkers in our company. I could submit a PR for this and waiting for @rdblue opinion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zhongyujiang commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
zhongyujiang commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r938720351


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,46 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);
+                if (field == null) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration, field: {} is not found in the schema: {}, "
+                          + "this column configuration will be ignored.",
+                      columnPath,
+                      schema);
+                  return;
+                }
+                Type fieldType = field.type();
+                // doesn't support boolean, list, map, struct type
+                if (!fieldType.isPrimitiveType()
+                    || fieldType.typeId().equals(Type.TypeID.BOOLEAN)) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration: field {} is of type '{}' which cannot "
+                          + "benefit from a bloom filter. This column configuration will be ignored.",
+                      columnPath,
+                      fieldType);
+                  return;
+                }
+
+                String compatibleName =
+                    dotJoiner.join(
+                        Streams.stream(dotSplitter.split(columnPath))

Review Comment:
   Using dot as splitter seems incorrect to me, cause Iceberg allows dots in field names.



##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,46 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);

Review Comment:
   Maybe we should add this check at the begining, filtering all invalid keys and making field names compatible before pushing these props down to writer. What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] zinking commented on pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by "zinking (via GitHub)" <gi...@apache.org>.
zinking commented on PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#issuecomment-1484993767

   @huaxingao can you take another look and push forward ? thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] huaxingao commented on pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#issuecomment-1485231481

   cc @rdblue Could you please take another look when you have a moment?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r939698338


##########
parquet/src/test/java/org/apache/iceberg/parquet/TestBloomRowGroupFilter.java:
##########
@@ -951,6 +968,72 @@ public void testStructFieldNotEq() {
     }
   }
 
+  @Test
+  public void testStructFieldWithSpecialCharacterLt() {

Review Comment:
   @huaxingao, can you review these tests?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] kbendick commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r938117606


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,42 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);
+                if (field == null) {
+                  LOG.warn(
+                      "Column: {} is not found in the schema: {}, ignore it.", columnPath, schema);
+                  return;
+                }
+                Type fieldType = field.type();
+                // doesn't support boolean, list, map, struct type
+                if (!fieldType.isPrimitiveType()
+                    || fieldType.typeId().equals(Type.TypeID.BOOLEAN)) {
+                  LOG.warn(
+                      "The type: {} of column: {} is not supported for parquet bloom filter, ignore it",
+                      fieldType,
+                      columnPath);
+                  return;

Review Comment:
   Nit / non-blocking question: Is this something we can prevent the user from doing? That might not be worth it though.
   
   Id also format it as `Invalid bloom filter column configuration: field `%s` is of type '%s' which parquet does not have bloom filter support for. This column configuration will be ignored.`. I am a fan of a full constant string at the front for searching logs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] kbendick commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r938117606


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,42 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);
+                if (field == null) {
+                  LOG.warn(
+                      "Column: {} is not found in the schema: {}, ignore it.", columnPath, schema);
+                  return;
+                }
+                Type fieldType = field.type();
+                // doesn't support boolean, list, map, struct type
+                if (!fieldType.isPrimitiveType()
+                    || fieldType.typeId().equals(Type.TypeID.BOOLEAN)) {
+                  LOG.warn(
+                      "The type: {} of column: {} is not supported for parquet bloom filter, ignore it",
+                      fieldType,
+                      columnPath);
+                  return;

Review Comment:
   Nit / non-blocking question: Is this something we can prevent the user from doing? That might not be worth it though.
   
   Id also format it as `Invalid bloom filter column configuration: field `%s` is of type '%s' which cannot benefit from a bloom filter. This column configuration will be ignored.`. I am a fan of a full constant string at the front for searching logs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] huaxingao commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r939708324


##########
parquet/src/test/java/org/apache/iceberg/parquet/TestBloomRowGroupFilter.java:
##########
@@ -951,6 +968,72 @@ public void testStructFieldNotEq() {
     }
   }
 
+  @Test
+  public void testStructFieldWithSpecialCharacterLt() {

Review Comment:
   @rdblue These tests look good. Thanks for pinging me. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r939697691


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,46 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);
+                if (field == null) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration, field: {} is not found in the schema: {}, "
+                          + "this column configuration will be ignored.",
+                      columnPath,
+                      schema);
+                  return;
+                }
+                Type fieldType = field.type();
+                // doesn't support boolean, list, map, struct type
+                if (!fieldType.isPrimitiveType()
+                    || fieldType.typeId().equals(Type.TypeID.BOOLEAN)) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration: field {} is of type '{}' which cannot "
+                          + "benefit from a bloom filter. This column configuration will be ignored.",
+                      columnPath,
+                      fieldType);
+                  return;
+                }
+
+                String compatibleName =
+                    dotJoiner.join(
+                        Streams.stream(dotSplitter.split(columnPath))
+                            .map(AvroSchemaUtil::makeCompatibleName)
+                            .collect(Collectors.toList()));

Review Comment:
   This should be separated into a util method in `ParquetSchemaUtil` that can be tested.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet: Set parquet bloom filter config with compatible column name [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5435: Parquet: Set parquet bloom filter config with compatible column name
URL: https://github.com/apache/iceberg/pull/5435


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] kbendick commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
kbendick commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r938117606


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,42 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);
+                if (field == null) {
+                  LOG.warn(
+                      "Column: {} is not found in the schema: {}, ignore it.", columnPath, schema);
+                  return;
+                }
+                Type fieldType = field.type();
+                // doesn't support boolean, list, map, struct type
+                if (!fieldType.isPrimitiveType()
+                    || fieldType.typeId().equals(Type.TypeID.BOOLEAN)) {
+                  LOG.warn(
+                      "The type: {} of column: {} is not supported for parquet bloom filter, ignore it",
+                      fieldType,
+                      columnPath);
+                  return;

Review Comment:
   Nit / non-blocking question: Is this something we can prevent the user from doing? That might not be worth it though.
   
   Id also format it as `Invalid bloom filter column configuration: field `%s` is of type '%s` which parquet does not have bloom filter support for. This column configuration will be ignored.`. I am a fan of a full constant string at the front for searching logs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r939698115


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,46 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);
+                if (field == null) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration, field: {} is not found in the schema: {}, "
+                          + "this column configuration will be ignored.",

Review Comment:
   This should be `LOG.warn("Skipping bloom filter config for missing field: %s", columnPath);`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r939804101


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,46 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);
+                if (field == null) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration, field: {} is not found in the schema: {}, "
+                          + "this column configuration will be ignored.",
+                      columnPath,
+                      schema);
+                  return;
+                }
+                Type fieldType = field.type();
+                // doesn't support boolean, list, map, struct type
+                if (!fieldType.isPrimitiveType()
+                    || fieldType.typeId().equals(Type.TypeID.BOOLEAN)) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration: field {} is of type '{}' which cannot "
+                          + "benefit from a bloom filter. This column configuration will be ignored.",
+                      columnPath,
+                      fieldType);
+                  return;
+                }
+
+                String compatibleName =
+                    dotJoiner.join(
+                        Streams.stream(dotSplitter.split(columnPath))

Review Comment:
   Good catch, updated to use the parquet column path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#issuecomment-1207637722

   Thanks @rdblue @kbendick @zhongyujiang @huaxingao for your time to review. Comments have been addressed. Pls take another look when you are free.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet: Set parquet bloom filter config with compatible column name [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#issuecomment-2294473766

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] huaxingao commented on pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#issuecomment-1485229348

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] huaxingao commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
huaxingao commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r939707183


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -291,11 +341,7 @@ public <D> FileAppender<D> build() throws IOException {
                 .withMaxRowCountForPageSizeCheck(rowGroupCheckMaxRecordCount)
                 .withMaxBloomFilterBytes(bloomFilterMaxBytes);
 
-        for (Map.Entry<String, String> entry : columnBloomFilterEnabled.entrySet()) {

Review Comment:
   nit: you forgot to delete the line
   ```
   Map<String, String> columnBloomFilterEnabled = context.columnBloomFilterEnabled();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#discussion_r939805792


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -231,6 +241,46 @@ private WriteBuilder createContextFunc(
       return this;
     }
 
+    private void setBloomFilterConfig(
+        Context context, BiConsumer<String, Boolean> withBloomFilterEnabled) {
+      Splitter dotSplitter = Splitter.on('.');
+      Joiner dotJoiner = Joiner.on(".");
+      context
+          .columnBloomFilterEnabled()
+          .forEach(
+              (columnPath, value) -> {
+                // check the column exists and support bloom filter
+                Types.NestedField field = schema.caseInsensitiveFindField(columnPath);
+                if (field == null) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration, field: {} is not found in the schema: {}, "
+                          + "this column configuration will be ignored.",
+                      columnPath,
+                      schema);
+                  return;
+                }
+                Type fieldType = field.type();
+                // doesn't support boolean, list, map, struct type
+                if (!fieldType.isPrimitiveType()
+                    || fieldType.typeId().equals(Type.TypeID.BOOLEAN)) {
+                  LOG.warn(
+                      "Invalid bloom filter column configuration: field {} is of type '{}' which cannot "
+                          + "benefit from a bloom filter. This column configuration will be ignored.",
+                      columnPath,
+                      fieldType);

Review Comment:
   Hi @rdblue, thanks for your time. The most end user doesn't understand the parquet bloom filter implementation details. The log warning here could give some users more information about their settings. Just as @zhongyujiang mentioned, maybe we should add some checking when setting/updating table properties. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet: Set parquet bloom filter config with compatible column name [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#issuecomment-2309064279

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on pull request #5435: Parquet: Set parquet bloom filter config with compatible column name

Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on PR #5435:
URL: https://github.com/apache/iceberg/pull/5435#issuecomment-1205003013

   Hi, @huaxingao @rdblue @kbendick . Could you help to review this when you are free? Thanks a lot.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org