You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/07/11 02:21:42 UTC
[GitHub] [iceberg] samredai opened a new pull request, #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
samredai opened a new pull request, #5239:
URL: https://github.com/apache/iceberg/pull/5239
This adds `BindVisitor` which is an implementation of `BooleanExpressionVisitor` for binding all references in an expression. As an example, here's binding an expression along the lines of `'a' in ('foo', 'bar', 'baz')`.
```py
from pyiceberg.expressions.base import BindVisitor, In, Reference, visit
from pyiceberg.expressions.literals import literal
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, StringType
schema = Schema(
NestedField(field_id=1, name="a", field_type=StringType(), required=True),
NestedField(field_id=2, name="b", field_type=StringType(), required=True),
NestedField(field_id=3, name="c", field_type=StringType(), required=False),
schema_id=1,
identifier_field_ids=[1],
)
bind_visitor = BindVisitor(schema=schema)
unbound_expression = In(Reference("a"), (literal("foo"), literal("bar"), literal("baz")))
visit(unbound_expression, visitor=bind_visitor)
```
result:
```py
BoundIn(
term=BoundReference(
field=NestedField(
field_id=1, name='a', field_type=StringType(), required=True
),
accessor=Accessor(position=0,inner=None)
),
literals=(
StringLiteral(foo),
StringLiteral(bar),
StringLiteral(baz)
)
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a diff in pull request #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
Posted by GitBox <gi...@apache.org>.
samredai commented on code in PR #5239:
URL: https://github.com/apache/iceberg/pull/5239#discussion_r917497169
##########
python/tests/test_types.py:
##########
@@ -211,7 +211,7 @@ def test_non_parameterized_type_equality(input_index, input_type, check_index, c
# Examples based on https://iceberg.apache.org/spec/#appendix-c-json-serialization
-class TestType(IcebergBaseModel):
Review Comment:
pytest gets fussy when it sees a class that starts with "Test"--it thinks it's an actual test. So I renamed these to `IcebergTestType`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a diff in pull request #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
Posted by GitBox <gi...@apache.org>.
samredai commented on code in PR #5239:
URL: https://github.com/apache/iceberg/pull/5239#discussion_r918063884
##########
python/tests/test_types.py:
##########
@@ -211,7 +211,7 @@ def test_non_parameterized_type_equality(input_index, input_type, check_index, c
# Examples based on https://iceberg.apache.org/spec/#appendix-c-json-serialization
-class TestType(IcebergBaseModel):
Review Comment:
No worries at all 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
Posted by GitBox <gi...@apache.org>.
Fokko commented on code in PR #5239:
URL: https://github.com/apache/iceberg/pull/5239#discussion_r918058842
##########
python/pyiceberg/expressions/base.py:
##########
@@ -433,9 +428,44 @@ def _(obj: And, visitor: BooleanExpressionVisitor[T]) -> T:
return visitor.visit_and(left_result=left_result, right_result=right_result)
+@visit.register(In)
+def _(obj: In, visitor: BooleanExpressionVisitor[T]) -> T:
+ """Visit an In boolean expression with a concrete BooleanExpressionVisitor"""
+ return visitor.visit_unbound_predicate(predicate=obj)
+
+
@visit.register(Or)
def _(obj: Or, visitor: BooleanExpressionVisitor[T]) -> T:
"""Visit an Or boolean expression with a concrete BooleanExpressionVisitor"""
left_result: T = visit(obj.left, visitor=visitor)
right_result: T = visit(obj.right, visitor=visitor)
return visitor.visit_or(left_result=left_result, right_result=right_result)
+
+
+class BindVisitor(BooleanExpressionVisitor[BooleanExpression]):
+ """Rewrites a boolean expression by replacing unbound references with references to fields in a struct schema"""
+
Review Comment:
Always good to add the variables in the class itself for clarity:
```python
_schema: Schema
_case_sensitive: bool
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on pull request #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
Posted by GitBox <gi...@apache.org>.
samredai commented on PR #5239:
URL: https://github.com/apache/iceberg/pull/5239#issuecomment-1179893366
cc: @CircArgs @Fokko @dramaticlly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a diff in pull request #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
Posted by GitBox <gi...@apache.org>.
samredai commented on code in PR #5239:
URL: https://github.com/apache/iceberg/pull/5239#discussion_r918071124
##########
python/pyiceberg/expressions/base.py:
##########
@@ -433,9 +428,44 @@ def _(obj: And, visitor: BooleanExpressionVisitor[T]) -> T:
return visitor.visit_and(left_result=left_result, right_result=right_result)
+@visit.register(In)
+def _(obj: In, visitor: BooleanExpressionVisitor[T]) -> T:
+ """Visit an In boolean expression with a concrete BooleanExpressionVisitor"""
+ return visitor.visit_unbound_predicate(predicate=obj)
+
+
@visit.register(Or)
def _(obj: Or, visitor: BooleanExpressionVisitor[T]) -> T:
"""Visit an Or boolean expression with a concrete BooleanExpressionVisitor"""
left_result: T = visit(obj.left, visitor=visitor)
right_result: T = visit(obj.right, visitor=visitor)
return visitor.visit_or(left_result=left_result, right_result=right_result)
+
+
+class BindVisitor(BooleanExpressionVisitor[BooleanExpression]):
+ """Rewrites a boolean expression by replacing unbound references with references to fields in a struct schema"""
+
Review Comment:
Added, thanks for catching this! 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #5239:
URL: https://github.com/apache/iceberg/pull/5239#issuecomment-1181073129
Nice work! Thanks, @samredai!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
Posted by GitBox <gi...@apache.org>.
Fokko commented on code in PR #5239:
URL: https://github.com/apache/iceberg/pull/5239#discussion_r918061677
##########
python/tests/expressions/test_expressions_base.py:
##########
@@ -364,3 +365,272 @@ def test_boolean_expression_visit_raise_not_implemented_error():
base.visit("foo", visitor=visitor)
assert str(exc_info.value) == "Cannot visit unsupported expression: foo"
+
+
+def test_always_true_expression_binding(table_schema_simple):
+ """Test that visiting an always-true expression returns always-true"""
+ unbound_expression = base.AlwaysTrue()
+ bound_expression = base.visit(unbound_expression, visitor=base.BindVisitor(schema=table_schema_simple))
+ assert bound_expression == base.AlwaysTrue()
+
+
+def test_always_false_expression_binding(table_schema_simple):
+ """Test that visiting an always-false expression returns always-false"""
+ unbound_expression = base.AlwaysFalse()
+ bound_expression = base.visit(unbound_expression, visitor=base.BindVisitor(schema=table_schema_simple))
+ assert bound_expression == base.AlwaysFalse()
+
+
+def test_always_false_and_always_true_expression_binding(table_schema_simple):
+ """Test that visiting both an always-true AND always-false expression returns always-false"""
+ unbound_expression = base.And(base.AlwaysTrue(), base.AlwaysFalse())
+ bound_expression = base.visit(unbound_expression, visitor=base.BindVisitor(schema=table_schema_simple))
+ assert bound_expression == base.AlwaysFalse()
+
+
+def test_always_false_or_always_true_expression_binding(table_schema_simple):
+ """Test that visiting always-true OR always-false expression returns always-true"""
+ unbound_expression = base.Or(base.AlwaysTrue(), base.AlwaysFalse())
+ bound_expression = base.visit(unbound_expression, visitor=base.BindVisitor(schema=table_schema_simple))
+ assert bound_expression == base.AlwaysTrue()
+
+
+@pytest.mark.parametrize(
+ "unbound_and_expression,expected_bound_expression",
+ [
+ (
+ base.And(
Review Comment:
Just curious, why `base.` everywhere? Personally, this introduces visual noise to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] samredai commented on a diff in pull request #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
Posted by GitBox <gi...@apache.org>.
samredai commented on code in PR #5239:
URL: https://github.com/apache/iceberg/pull/5239#discussion_r918075151
##########
python/tests/expressions/test_expressions_base.py:
##########
@@ -364,3 +365,272 @@ def test_boolean_expression_visit_raise_not_implemented_error():
base.visit("foo", visitor=visitor)
assert str(exc_info.value) == "Cannot visit unsupported expression: foo"
+
+
+def test_always_true_expression_binding(table_schema_simple):
+ """Test that visiting an always-true expression returns always-true"""
+ unbound_expression = base.AlwaysTrue()
+ bound_expression = base.visit(unbound_expression, visitor=base.BindVisitor(schema=table_schema_simple))
+ assert bound_expression == base.AlwaysTrue()
+
+
+def test_always_false_expression_binding(table_schema_simple):
+ """Test that visiting an always-false expression returns always-false"""
+ unbound_expression = base.AlwaysFalse()
+ bound_expression = base.visit(unbound_expression, visitor=base.BindVisitor(schema=table_schema_simple))
+ assert bound_expression == base.AlwaysFalse()
+
+
+def test_always_false_and_always_true_expression_binding(table_schema_simple):
+ """Test that visiting both an always-true AND always-false expression returns always-false"""
+ unbound_expression = base.And(base.AlwaysTrue(), base.AlwaysFalse())
+ bound_expression = base.visit(unbound_expression, visitor=base.BindVisitor(schema=table_schema_simple))
+ assert bound_expression == base.AlwaysFalse()
+
+
+def test_always_false_or_always_true_expression_binding(table_schema_simple):
+ """Test that visiting always-true OR always-false expression returns always-true"""
+ unbound_expression = base.Or(base.AlwaysTrue(), base.AlwaysFalse())
+ bound_expression = base.visit(unbound_expression, visitor=base.BindVisitor(schema=table_schema_simple))
+ assert bound_expression == base.AlwaysTrue()
+
+
+@pytest.mark.parametrize(
+ "unbound_and_expression,expected_bound_expression",
+ [
+ (
+ base.And(
Review Comment:
It's definitely noisier to look at, especially with these longer expressions. It's a habit I've picked up to fully import the module that's being tested and it helps quickly identify what's being tested. Anything that's `base.*` is being tested in this file that's testing `base.py`.
This may not be the best example since we have multiple base.py files in different directories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
Posted by GitBox <gi...@apache.org>.
rdblue merged PR #5239:
URL: https://github.com/apache/iceberg/pull/5239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #5239: [Python] BindVisitor for binding a BooleanExpression to a Schema
Posted by GitBox <gi...@apache.org>.
Fokko commented on code in PR #5239:
URL: https://github.com/apache/iceberg/pull/5239#discussion_r918062230
##########
python/tests/test_types.py:
##########
@@ -211,7 +211,7 @@ def test_non_parameterized_type_equality(input_index, input_type, check_index, c
# Examples based on https://iceberg.apache.org/spec/#appendix-c-json-serialization
-class TestType(IcebergBaseModel):
Review Comment:
Sorry about that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org