You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/11/23 18:02:59 UTC
[GitHub] [iceberg] rdblue opened a new pull request, #6259: Python: Add boolean expression parser
rdblue opened a new pull request, #6259:
URL: https://github.com/apache/iceberg/pull/6259
This adds a simple expression parser based on pyparsing.
This needs thorough tests and to add pyparsing to pyproject.toml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #6259: Python: Add boolean expression parser
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #6259:
URL: https://github.com/apache/iceberg/pull/6259#issuecomment-1329760486
Merged. Thanks for reviewing, @Fokko!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #6259: Python: Add boolean expression parser
Posted by GitBox <gi...@apache.org>.
rdblue merged PR #6259:
URL: https://github.com/apache/iceberg/pull/6259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #6259: Python: Add boolean expression parser
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #6259:
URL: https://github.com/apache/iceberg/pull/6259#discussion_r1032834290
##########
python/tests/expressions/test_parser.py:
##########
@@ -0,0 +1,151 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+from pyparsing import ParseException
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ parser,
+)
+
+
+def test_true():
+ assert AlwaysTrue() == parser.parse("true")
+
+
+def test_false():
+ assert AlwaysFalse() == parser.parse("false")
+
+
+def test_is_null():
+ assert IsNull("x") == parser.parse("x is null")
+ assert IsNull("x") == parser.parse("x IS NULL")
+
+
+def test_not_null():
+ assert NotNull("x") == parser.parse("x is not null")
+ assert NotNull("x") == parser.parse("x IS NOT NULL")
+
+
+def test_is_nan():
+ assert IsNaN("x") == parser.parse("x is nan")
+ assert IsNaN("x") == parser.parse("x IS NAN")
+
+
+def test_not_nan():
+ assert NotNaN("x") == parser.parse("x is not nan")
+ assert NotNaN("x") == parser.parse("x IS NOT NaN")
+
+
+def test_less_than():
+ assert LessThan("x", 5) == parser.parse("x < 5")
+ assert LessThan("x", "a") == parser.parse("'a' > x")
+
+
+def test_less_than_or_equal():
+ assert LessThanOrEqual("x", 5) == parser.parse("x <= 5")
+ assert LessThanOrEqual("x", "a") == parser.parse("'a' >= x")
+
+
+def test_greater_than():
+ assert GreaterThan("x", 5) == parser.parse("x > 5")
+ assert GreaterThan("x", "a") == parser.parse("'a' < x")
+
+
+def test_greater_than_or_equal():
+ assert GreaterThanOrEqual("x", 5) == parser.parse("x <= 5")
+ assert GreaterThanOrEqual("x", "a") == parser.parse("'a' >= x")
+
+
+def test_equal_to():
+ assert EqualTo("x", 5) == parser.parse("x = 5")
+ assert EqualTo("x", "a") == parser.parse("'a' = x")
+ assert EqualTo("x", "a") == parser.parse("x == 'a'")
+ assert EqualTo("x", 5) == parser.parse("5 == x")
+
+
+def test_not_equal_to():
+ assert NotEqualTo("x", 5) == parser.parse("x != 5")
+ assert NotEqualTo("x", "a") == parser.parse("'a' != x")
+ assert NotEqualTo("x", "a") == parser.parse("x <> 'a'")
+ assert NotEqualTo("x", 5) == parser.parse("5 <> x")
+
+
+def test_in():
+ assert In("x", {5, 6, 7}) == parser.parse("x in (5, 6, 7)")
+ assert In("x", {"a", "b", "c"}) == parser.parse("x IN ('a', 'b', 'c')")
+
+
+def test_in_different_types():
+ with pytest.raises(ParseException):
+ parser.parse("x in (5, 'a')")
+
+
+def test_not_in():
+ assert NotIn("x", {5, 6, 7}) == parser.parse("x not in (5, 6, 7)")
+ assert NotIn("x", {"a", "b", "c"}) == parser.parse("x NOT IN ('a', 'b', 'c')")
+
+
+def test_not_in_different_types():
+ with pytest.raises(ParseException):
+ parser.parse("x not in (5, 'a')")
+
+
+def test_simple_and():
+ assert And(GreaterThanOrEqual("x", 5), LessThan("x", 10)) == parser.parse("5 <= x and x < 10")
+
+
+def test_and_with_not():
+ assert And(Not(GreaterThanOrEqual("x", 5)), LessThan("x", 10)) == parser.parse("not 5 <= x and x < 10")
+ assert And(GreaterThanOrEqual("x", 5), Not(LessThan("x", 10))) == parser.parse("5 <= x and not x < 10")
+
+
+def test_or_with_not():
+ assert Or(Not(LessThan("x", 5)), GreaterThan("x", 10)) == parser.parse("not x < 5 or 10 < x")
+ assert Or(LessThan("x", 5), Not(GreaterThan("x", 10))) == parser.parse("x < 5 or not 10 < x")
+
+
+def test_simple_or():
+ assert Or(LessThan("x", 5), GreaterThan("x", 10)) == parser.parse("x < 5 or 10 < x")
+
+
+def test_and_or_without_parens():
+ assert Or(And(NotNull("x"), LessThan("x", 5)), GreaterThan("x", 10)) == parser.parse("x is not null and x < 5 or 10 < x")
+ assert Or(IsNull("x"), And(GreaterThanOrEqual("x", 5), LessThan("x", 10))) == parser.parse("x is null or 5 <= x and x < 10")
+
+
+def test_and_or_with_parens():
+ assert And(NotNull("x"), Or(LessThan("x", 5), GreaterThan("x", 10))) == parser.parse("x is not null and (x < 5 or 10 < x)")
+ assert Or(IsNull("x"), And(GreaterThanOrEqual("x", 5), Not(LessThan("x", 10)))) == parser.parse(
+ "(x is null) or (5 <= x) and not(x < 10)"
+ )
Review Comment:
I verified that the result of these complex cases matches Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #6259: Python: Add boolean expression parser
Posted by GitBox <gi...@apache.org>.
Fokko commented on code in PR #6259:
URL: https://github.com/apache/iceberg/pull/6259#discussion_r1033946989
##########
python/pyiceberg/expressions/literals.py:
##########
@@ -86,6 +86,8 @@ def __hash__(self) -> int:
return hash(self.value)
def __eq__(self, other: Any) -> bool:
+ if not isinstance(other, Literal):
Review Comment:
If we get something else, we could wrap it in a literal and compare it. This would allow `literal(1) == 1`.
##########
python/pyiceberg/expressions/literals.py:
##########
@@ -401,6 +403,22 @@ def _(self, type_var: DecimalType) -> Literal[Decimal]:
return self
raise ValueError(f"Could not convert {self.value} into a {type_var}")
+ @to.register(IntegerType)
+ def _(self, _: IntegerType) -> Literal[int]:
+ return LongLiteral(int(self.value.to_integral_value()))
+
+ @to.register(LongType)
+ def _(self, _: LongType) -> Literal[int]:
+ return LongLiteral(int(self.value.to_integral_value()))
Review Comment:
We want to check if they are below or above max.
##########
python/pyiceberg/expressions/literals.py:
##########
@@ -401,6 +403,22 @@ def _(self, type_var: DecimalType) -> Literal[Decimal]:
return self
raise ValueError(f"Could not convert {self.value} into a {type_var}")
+ @to.register(IntegerType)
+ def _(self, _: IntegerType) -> Literal[int]:
+ return LongLiteral(int(self.value.to_integral_value()))
Review Comment:
We want to check if they are below or above max.
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
Review Comment:
```suggestion
identifier = Word(alphas, f"{alphanums}_$").set_results_name("identifier")
```
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
+column = delimited_list(identifier, delim=".", combine=True).set_results_name("column")
+
+
+@column.set_parse_action
+def _(result: ParseResults):
Review Comment:
```suggestion
def _(result: ParseResults) -> Reference[Any]:
```
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
+column = delimited_list(identifier, delim=".", combine=True).set_results_name("column")
+
+
+@column.set_parse_action
+def _(result: ParseResults):
+ return Reference(result.column[0])
+
+
+boolean = one_of(["true", "false"], caseless=True).set_results_name("boolean")
+string = sgl_quoted_string.set_results_name("raw_quoted_string")
+decimal = common.real().set_results_name("decimal")
+integer = common.signed_integer().set_results_name("integer")
+literal = Group(string | decimal | integer).set_results_name("literal")
+literal_set = Group(delimited_list(string) | delimited_list(decimal) | delimited_list(integer)).set_results_name("literal_set")
+
+
+@boolean.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if "true" == result.boolean.lower():
+ return AlwaysTrue()
+ else:
+ return AlwaysFalse()
+
+
+@string.set_parse_action
+def _(result: ParseResults) -> Literal[str]:
+ return StringLiteral(result.raw_quoted_string[1:-1].replace("''", "'"))
+
+
+@decimal.set_parse_action
+def _(result: ParseResults) -> Literal[Decimal]:
+ return DecimalLiteral(Decimal(result.decimal))
+
+
+@integer.set_parse_action
+def _(result: ParseResults) -> Literal[int]:
+ return LongLiteral(int(result.integer))
+
+
+@literal.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0][0]
+
+
+@literal_set.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0]
+
+
+comparison_op = one_of(["<", "<=", ">", ">=", "=", "==", "!=", "<>"], caseless=True).set_results_name("op")
+left_ref = column + comparison_op + literal
+right_ref = literal + comparison_op + column
+comparison = left_ref | right_ref
+
+
+@left_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return LessThan(result.column, result.literal)
+ elif result.op == "<=":
+ return LessThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == ">=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
Review Comment:
```suggestion
elif result.op in ("=", "=="):
```
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
+column = delimited_list(identifier, delim=".", combine=True).set_results_name("column")
+
+
+@column.set_parse_action
+def _(result: ParseResults):
+ return Reference(result.column[0])
+
+
+boolean = one_of(["true", "false"], caseless=True).set_results_name("boolean")
+string = sgl_quoted_string.set_results_name("raw_quoted_string")
+decimal = common.real().set_results_name("decimal")
+integer = common.signed_integer().set_results_name("integer")
+literal = Group(string | decimal | integer).set_results_name("literal")
+literal_set = Group(delimited_list(string) | delimited_list(decimal) | delimited_list(integer)).set_results_name("literal_set")
+
+
+@boolean.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if "true" == result.boolean.lower():
Review Comment:
We could do this in a single line if we're feeling verbose today:
```suggestion
return AlwaysTrue() if "true" == result.boolean.lower() else AlwaysFalse()
```
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
+column = delimited_list(identifier, delim=".", combine=True).set_results_name("column")
+
+
+@column.set_parse_action
+def _(result: ParseResults):
+ return Reference(result.column[0])
+
+
+boolean = one_of(["true", "false"], caseless=True).set_results_name("boolean")
+string = sgl_quoted_string.set_results_name("raw_quoted_string")
+decimal = common.real().set_results_name("decimal")
+integer = common.signed_integer().set_results_name("integer")
+literal = Group(string | decimal | integer).set_results_name("literal")
+literal_set = Group(delimited_list(string) | delimited_list(decimal) | delimited_list(integer)).set_results_name("literal_set")
+
+
+@boolean.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if "true" == result.boolean.lower():
+ return AlwaysTrue()
+ else:
+ return AlwaysFalse()
+
+
+@string.set_parse_action
+def _(result: ParseResults) -> Literal[str]:
+ return StringLiteral(result.raw_quoted_string[1:-1].replace("''", "'"))
+
+
+@decimal.set_parse_action
+def _(result: ParseResults) -> Literal[Decimal]:
+ return DecimalLiteral(Decimal(result.decimal))
+
+
+@integer.set_parse_action
+def _(result: ParseResults) -> Literal[int]:
+ return LongLiteral(int(result.integer))
+
+
+@literal.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0][0]
+
+
+@literal_set.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0]
+
+
+comparison_op = one_of(["<", "<=", ">", ">=", "=", "==", "!=", "<>"], caseless=True).set_results_name("op")
+left_ref = column + comparison_op + literal
+right_ref = literal + comparison_op + column
+comparison = left_ref | right_ref
+
+
+@left_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return LessThan(result.column, result.literal)
+ elif result.op == "<=":
+ return LessThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == ">=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
+ return EqualTo(result.column, result.literal)
+ if result.op in ("!=", "<>"):
Review Comment:
```suggestion
elif result.op in ("!=", "<>"):
```
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
+column = delimited_list(identifier, delim=".", combine=True).set_results_name("column")
+
+
+@column.set_parse_action
+def _(result: ParseResults):
+ return Reference(result.column[0])
+
+
+boolean = one_of(["true", "false"], caseless=True).set_results_name("boolean")
+string = sgl_quoted_string.set_results_name("raw_quoted_string")
+decimal = common.real().set_results_name("decimal")
+integer = common.signed_integer().set_results_name("integer")
+literal = Group(string | decimal | integer).set_results_name("literal")
+literal_set = Group(delimited_list(string) | delimited_list(decimal) | delimited_list(integer)).set_results_name("literal_set")
+
+
+@boolean.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if "true" == result.boolean.lower():
+ return AlwaysTrue()
+ else:
+ return AlwaysFalse()
+
+
+@string.set_parse_action
+def _(result: ParseResults) -> Literal[str]:
+ return StringLiteral(result.raw_quoted_string[1:-1].replace("''", "'"))
+
+
+@decimal.set_parse_action
+def _(result: ParseResults) -> Literal[Decimal]:
+ return DecimalLiteral(Decimal(result.decimal))
+
+
+@integer.set_parse_action
+def _(result: ParseResults) -> Literal[int]:
+ return LongLiteral(int(result.integer))
+
+
+@literal.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0][0]
+
+
+@literal_set.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0]
+
+
+comparison_op = one_of(["<", "<=", ">", ">=", "=", "==", "!=", "<>"], caseless=True).set_results_name("op")
+left_ref = column + comparison_op + literal
+right_ref = literal + comparison_op + column
+comparison = left_ref | right_ref
+
+
+@left_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return LessThan(result.column, result.literal)
+ elif result.op == "<=":
+ return LessThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == ">=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
+ return EqualTo(result.column, result.literal)
+ if result.op in ("!=", "<>"):
+ return NotEqualTo(result.column, result.literal)
+ raise ValueError(f"Unsupported operation type: {result.op}")
+
+
+@right_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == "<=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return LessThan(result.column, result.literal)
+ elif result.op == ">=":
+ return LessThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
+ return EqualTo(result.column, result.literal)
+ if result.op in ("!=", "<>"):
Review Comment:
```suggestion
elif result.op in ("!=", "<>"):
```
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
+column = delimited_list(identifier, delim=".", combine=True).set_results_name("column")
+
+
+@column.set_parse_action
+def _(result: ParseResults):
+ return Reference(result.column[0])
+
+
+boolean = one_of(["true", "false"], caseless=True).set_results_name("boolean")
+string = sgl_quoted_string.set_results_name("raw_quoted_string")
+decimal = common.real().set_results_name("decimal")
+integer = common.signed_integer().set_results_name("integer")
+literal = Group(string | decimal | integer).set_results_name("literal")
+literal_set = Group(delimited_list(string) | delimited_list(decimal) | delimited_list(integer)).set_results_name("literal_set")
+
+
+@boolean.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if "true" == result.boolean.lower():
+ return AlwaysTrue()
+ else:
+ return AlwaysFalse()
+
+
+@string.set_parse_action
+def _(result: ParseResults) -> Literal[str]:
+ return StringLiteral(result.raw_quoted_string[1:-1].replace("''", "'"))
+
+
+@decimal.set_parse_action
+def _(result: ParseResults) -> Literal[Decimal]:
+ return DecimalLiteral(Decimal(result.decimal))
+
+
+@integer.set_parse_action
+def _(result: ParseResults) -> Literal[int]:
+ return LongLiteral(int(result.integer))
+
+
+@literal.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0][0]
+
+
+@literal_set.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0]
+
+
+comparison_op = one_of(["<", "<=", ">", ">=", "=", "==", "!=", "<>"], caseless=True).set_results_name("op")
+left_ref = column + comparison_op + literal
+right_ref = literal + comparison_op + column
+comparison = left_ref | right_ref
+
+
+@left_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return LessThan(result.column, result.literal)
+ elif result.op == "<=":
+ return LessThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == ">=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
+ return EqualTo(result.column, result.literal)
+ if result.op in ("!=", "<>"):
+ return NotEqualTo(result.column, result.literal)
+ raise ValueError(f"Unsupported operation type: {result.op}")
+
+
+@right_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == "<=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return LessThan(result.column, result.literal)
+ elif result.op == ">=":
+ return LessThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
+ return EqualTo(result.column, result.literal)
+ if result.op in ("!=", "<>"):
+ return NotEqualTo(result.column, result.literal)
+ raise ValueError(f"Unsupported operation type: {result.op}")
+
+
+is_null = column + IS + NULL
+not_null = column + IS + NOT + NULL
+null_check = is_null | not_null
+
+
+@is_null.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return IsNull(result.column)
+
+
+@not_null.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return NotNull(result.column)
+
+
+is_nan = column + IS + NAN
+not_nan = column + IS + NOT + NAN
+nan_check = is_nan | not_nan
+
+
+@is_nan.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return IsNaN(result.column)
+
+
+@not_nan.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return NotNaN(result.column)
+
+
+is_in = column + IN + "(" + literal_set + ")"
Review Comment:
```suggestion
is_in = f"{column}{IN}({literal_set})"
```
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
+column = delimited_list(identifier, delim=".", combine=True).set_results_name("column")
+
+
+@column.set_parse_action
+def _(result: ParseResults):
+ return Reference(result.column[0])
+
+
+boolean = one_of(["true", "false"], caseless=True).set_results_name("boolean")
+string = sgl_quoted_string.set_results_name("raw_quoted_string")
+decimal = common.real().set_results_name("decimal")
+integer = common.signed_integer().set_results_name("integer")
+literal = Group(string | decimal | integer).set_results_name("literal")
+literal_set = Group(delimited_list(string) | delimited_list(decimal) | delimited_list(integer)).set_results_name("literal_set")
+
+
+@boolean.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if "true" == result.boolean.lower():
+ return AlwaysTrue()
+ else:
+ return AlwaysFalse()
+
+
+@string.set_parse_action
+def _(result: ParseResults) -> Literal[str]:
+ return StringLiteral(result.raw_quoted_string[1:-1].replace("''", "'"))
+
+
+@decimal.set_parse_action
+def _(result: ParseResults) -> Literal[Decimal]:
+ return DecimalLiteral(Decimal(result.decimal))
+
+
+@integer.set_parse_action
+def _(result: ParseResults) -> Literal[int]:
+ return LongLiteral(int(result.integer))
+
+
+@literal.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0][0]
+
+
+@literal_set.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0]
+
+
+comparison_op = one_of(["<", "<=", ">", ">=", "=", "==", "!=", "<>"], caseless=True).set_results_name("op")
+left_ref = column + comparison_op + literal
+right_ref = literal + comparison_op + column
+comparison = left_ref | right_ref
+
+
+@left_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return LessThan(result.column, result.literal)
+ elif result.op == "<=":
+ return LessThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == ">=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
+ return EqualTo(result.column, result.literal)
+ if result.op in ("!=", "<>"):
+ return NotEqualTo(result.column, result.literal)
+ raise ValueError(f"Unsupported operation type: {result.op}")
+
+
+@right_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == "<=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return LessThan(result.column, result.literal)
+ elif result.op == ">=":
+ return LessThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
+ return EqualTo(result.column, result.literal)
+ if result.op in ("!=", "<>"):
+ return NotEqualTo(result.column, result.literal)
+ raise ValueError(f"Unsupported operation type: {result.op}")
+
+
+is_null = column + IS + NULL
+not_null = column + IS + NOT + NULL
+null_check = is_null | not_null
+
+
+@is_null.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return IsNull(result.column)
+
+
+@not_null.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return NotNull(result.column)
+
+
+is_nan = column + IS + NAN
+not_nan = column + IS + NOT + NAN
+nan_check = is_nan | not_nan
+
+
+@is_nan.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return IsNaN(result.column)
+
+
+@not_nan.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return NotNaN(result.column)
+
+
+is_in = column + IN + "(" + literal_set + ")"
+not_in = column + NOT + IN + "(" + literal_set + ")"
Review Comment:
```suggestion
not_in = f"{column}{NOT}{IN}({literal_set})"
```
##########
python/pyiceberg/expressions/literals.py:
##########
@@ -401,6 +403,22 @@ def _(self, type_var: DecimalType) -> Literal[Decimal]:
return self
raise ValueError(f"Could not convert {self.value} into a {type_var}")
+ @to.register(IntegerType)
+ def _(self, _: IntegerType) -> Literal[int]:
+ return LongLiteral(int(self.value.to_integral_value()))
+
+ @to.register(LongType)
+ def _(self, _: LongType) -> Literal[int]:
+ return LongLiteral(int(self.value.to_integral_value()))
+
+ @to.register(FloatType)
+ def _(self, _: FloatType):
Review Comment:
We want to check if they are below or above max.
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
+column = delimited_list(identifier, delim=".", combine=True).set_results_name("column")
+
+
+@column.set_parse_action
+def _(result: ParseResults):
+ return Reference(result.column[0])
+
+
+boolean = one_of(["true", "false"], caseless=True).set_results_name("boolean")
+string = sgl_quoted_string.set_results_name("raw_quoted_string")
+decimal = common.real().set_results_name("decimal")
+integer = common.signed_integer().set_results_name("integer")
+literal = Group(string | decimal | integer).set_results_name("literal")
+literal_set = Group(delimited_list(string) | delimited_list(decimal) | delimited_list(integer)).set_results_name("literal_set")
+
+
+@boolean.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if "true" == result.boolean.lower():
+ return AlwaysTrue()
+ else:
+ return AlwaysFalse()
+
+
+@string.set_parse_action
+def _(result: ParseResults) -> Literal[str]:
+ return StringLiteral(result.raw_quoted_string[1:-1].replace("''", "'"))
+
+
+@decimal.set_parse_action
+def _(result: ParseResults) -> Literal[Decimal]:
+ return DecimalLiteral(Decimal(result.decimal))
+
+
+@integer.set_parse_action
+def _(result: ParseResults) -> Literal[int]:
+ return LongLiteral(int(result.integer))
+
+
+@literal.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0][0]
+
+
+@literal_set.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0]
+
+
+comparison_op = one_of(["<", "<=", ">", ">=", "=", "==", "!=", "<>"], caseless=True).set_results_name("op")
+left_ref = column + comparison_op + literal
+right_ref = literal + comparison_op + column
+comparison = left_ref | right_ref
+
+
+@left_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return LessThan(result.column, result.literal)
+ elif result.op == "<=":
+ return LessThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == ">=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
Review Comment:
Do we want to move these operators into module-level variables? I also noticed that we have the same groups below.
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
+column = delimited_list(identifier, delim=".", combine=True).set_results_name("column")
+
+
+@column.set_parse_action
+def _(result: ParseResults):
+ return Reference(result.column[0])
+
+
+boolean = one_of(["true", "false"], caseless=True).set_results_name("boolean")
+string = sgl_quoted_string.set_results_name("raw_quoted_string")
+decimal = common.real().set_results_name("decimal")
+integer = common.signed_integer().set_results_name("integer")
+literal = Group(string | decimal | integer).set_results_name("literal")
+literal_set = Group(delimited_list(string) | delimited_list(decimal) | delimited_list(integer)).set_results_name("literal_set")
+
+
+@boolean.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if "true" == result.boolean.lower():
+ return AlwaysTrue()
+ else:
+ return AlwaysFalse()
+
+
+@string.set_parse_action
+def _(result: ParseResults) -> Literal[str]:
+ return StringLiteral(result.raw_quoted_string[1:-1].replace("''", "'"))
+
+
+@decimal.set_parse_action
+def _(result: ParseResults) -> Literal[Decimal]:
+ return DecimalLiteral(Decimal(result.decimal))
+
+
+@integer.set_parse_action
+def _(result: ParseResults) -> Literal[int]:
+ return LongLiteral(int(result.integer))
+
+
+@literal.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0][0]
+
+
+@literal_set.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0]
+
+
+comparison_op = one_of(["<", "<=", ">", ">=", "=", "==", "!=", "<>"], caseless=True).set_results_name("op")
+left_ref = column + comparison_op + literal
+right_ref = literal + comparison_op + column
+comparison = left_ref | right_ref
+
+
+@left_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return LessThan(result.column, result.literal)
+ elif result.op == "<=":
+ return LessThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == ">=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
+ return EqualTo(result.column, result.literal)
+ if result.op in ("!=", "<>"):
+ return NotEqualTo(result.column, result.literal)
+ raise ValueError(f"Unsupported operation type: {result.op}")
+
+
+@right_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == "<=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return LessThan(result.column, result.literal)
+ elif result.op == ">=":
+ return LessThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
Review Comment:
```suggestion
elif result.op in ("=", "=="):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #6259: Python: Add boolean expression parser
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #6259:
URL: https://github.com/apache/iceberg/pull/6259#discussion_r1034011441
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
+column = delimited_list(identifier, delim=".", combine=True).set_results_name("column")
+
+
+@column.set_parse_action
+def _(result: ParseResults):
+ return Reference(result.column[0])
+
+
+boolean = one_of(["true", "false"], caseless=True).set_results_name("boolean")
+string = sgl_quoted_string.set_results_name("raw_quoted_string")
+decimal = common.real().set_results_name("decimal")
+integer = common.signed_integer().set_results_name("integer")
+literal = Group(string | decimal | integer).set_results_name("literal")
+literal_set = Group(delimited_list(string) | delimited_list(decimal) | delimited_list(integer)).set_results_name("literal_set")
+
+
+@boolean.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if "true" == result.boolean.lower():
+ return AlwaysTrue()
+ else:
+ return AlwaysFalse()
+
+
+@string.set_parse_action
+def _(result: ParseResults) -> Literal[str]:
+ return StringLiteral(result.raw_quoted_string[1:-1].replace("''", "'"))
+
+
+@decimal.set_parse_action
+def _(result: ParseResults) -> Literal[Decimal]:
+ return DecimalLiteral(Decimal(result.decimal))
+
+
+@integer.set_parse_action
+def _(result: ParseResults) -> Literal[int]:
+ return LongLiteral(int(result.integer))
+
+
+@literal.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0][0]
+
+
+@literal_set.set_parse_action
+def _(result: ParseResults) -> Literal[L]:
+ return result[0]
+
+
+comparison_op = one_of(["<", "<=", ">", ">=", "=", "==", "!=", "<>"], caseless=True).set_results_name("op")
+left_ref = column + comparison_op + literal
+right_ref = literal + comparison_op + column
+comparison = left_ref | right_ref
+
+
+@left_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return LessThan(result.column, result.literal)
+ elif result.op == "<=":
+ return LessThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == ">=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
+ return EqualTo(result.column, result.literal)
+ if result.op in ("!=", "<>"):
+ return NotEqualTo(result.column, result.literal)
+ raise ValueError(f"Unsupported operation type: {result.op}")
+
+
+@right_ref.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ if result.op == "<":
+ return GreaterThan(result.column, result.literal)
+ elif result.op == "<=":
+ return GreaterThanOrEqual(result.column, result.literal)
+ elif result.op == ">":
+ return LessThan(result.column, result.literal)
+ elif result.op == ">=":
+ return LessThanOrEqual(result.column, result.literal)
+ if result.op in ("=", "=="):
+ return EqualTo(result.column, result.literal)
+ if result.op in ("!=", "<>"):
+ return NotEqualTo(result.column, result.literal)
+ raise ValueError(f"Unsupported operation type: {result.op}")
+
+
+is_null = column + IS + NULL
+not_null = column + IS + NOT + NULL
+null_check = is_null | not_null
+
+
+@is_null.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return IsNull(result.column)
+
+
+@not_null.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return NotNull(result.column)
+
+
+is_nan = column + IS + NAN
+not_nan = column + IS + NOT + NAN
+nan_check = is_nan | not_nan
+
+
+@is_nan.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return IsNaN(result.column)
+
+
+@not_nan.set_parse_action
+def _(result: ParseResults) -> BooleanExpression:
+ return NotNaN(result.column)
+
+
+is_in = column + IN + "(" + literal_set + ")"
Review Comment:
These aren't strings. They're parser rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #6259: Python: Add boolean expression parser
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #6259:
URL: https://github.com/apache/iceberg/pull/6259#discussion_r1034010412
##########
python/pyiceberg/expressions/parser.py:
##########
@@ -0,0 +1,237 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from decimal import Decimal
+
+from pyparsing import (
+ CaselessKeyword,
+ Group,
+ ParserElement,
+ ParseResults,
+ Suppress,
+ Word,
+ alphanums,
+ alphas,
+ delimited_list,
+ infix_notation,
+ one_of,
+ opAssoc,
+ sgl_quoted_string,
+)
+from pyparsing.common import pyparsing_common as common
+
+from pyiceberg.expressions import (
+ AlwaysFalse,
+ AlwaysTrue,
+ And,
+ BooleanExpression,
+ EqualTo,
+ GreaterThan,
+ GreaterThanOrEqual,
+ In,
+ IsNaN,
+ IsNull,
+ LessThan,
+ LessThanOrEqual,
+ Not,
+ NotEqualTo,
+ NotIn,
+ NotNaN,
+ NotNull,
+ Or,
+ Reference,
+)
+from pyiceberg.expressions.literals import (
+ DecimalLiteral,
+ Literal,
+ LongLiteral,
+ StringLiteral,
+)
+from pyiceberg.typedef import L
+
+ParserElement.enablePackrat()
+
+AND = CaselessKeyword("and")
+OR = CaselessKeyword("or")
+NOT = CaselessKeyword("not")
+IS = CaselessKeyword("is")
+IN = CaselessKeyword("in")
+NULL = CaselessKeyword("null")
+NAN = CaselessKeyword("nan")
+
+identifier = Word(alphas, alphanums + "_$").set_results_name("identifier")
Review Comment:
This is the recommendation from pyparsing and I don't think that it necessarily produces a string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #6259: Python: Add boolean expression parser
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #6259:
URL: https://github.com/apache/iceberg/pull/6259#discussion_r1034008760
##########
python/pyiceberg/expressions/literals.py:
##########
@@ -86,6 +86,8 @@ def __hash__(self) -> int:
return hash(self.value)
def __eq__(self, other: Any) -> bool:
+ if not isinstance(other, Literal):
Review Comment:
I don't think that we want to do that. Then equality would not be symmetric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org