You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pritampan (via GitHub)" <gi...@apache.org> on 2023/02/15 16:28:10 UTC
[GitHub] [iceberg] pritampan opened a new pull request, #6843: [Python Legacy] Convert string to boolean if the binding variable is Boolean
pritampan opened a new pull request, #6843:
URL: https://github.com/apache/iceberg/pull/6843
In case of `Boolean` binding column following expression `Expressions.convert_string_to_expr("dt='2023-02-13' and is_auth_action=true")` throwing following error -
``` file_scan_tasks = list(table.new_scan().filter(Expressions.convert_string_to_expr(partition)).plan_files())
File "/usr/local/lib/python3.7/dist-packages/iceberg/core/data_table_scan.py", line 72, in plan_files
return super(DataTableScan, self).plan_files()
File "/usr/local/lib/python3.7/dist-packages/iceberg/core/base_table_scan.py", line 164, in plan_files
return self.plan_files(ops, snapshot, row_filter)
File "/usr/local/lib/python3.7/dist-packages/iceberg/core/data_table_scan.py", line 74, in plan_files
matching_manifests = [manifest for manifest in snapshot.manifests
File "/usr/local/lib/python3.7/dist-packages/iceberg/core/data_table_scan.py", line 75, in <listcomp>
if self.cache_loader(manifest.spec_id).eval(manifest)]
File "/usr/local/lib/python3.7/dist-packages/iceberg/core/data_table_scan.py", line 105, in cache_loader
return InclusiveManifestEvaluator(spec, self.row_filter)
File "/usr/local/lib/python3.7/dist-packages/iceberg/api/expressions/inclusive_manifest_evaluator.py", line 35, in __init__
.project(row_filter)),
File "/usr/local/lib/python3.7/dist-packages/iceberg/api/expressions/projections.py", line 50, in project
return ExpressionVisitors.visit(ExpressionVisitors.visit(expr, RewriteNot.get()), self)
File "/usr/local/lib/python3.7/dist-packages/iceberg/api/expressions/expressions.py", line 170, in visit
ExpressionVisitors.visit(expr.right, visitor))
File "/usr/local/lib/python3.7/dist-packages/iceberg/api/expressions/expressions.py", line 160, in visit
return visitor.predicate(expr)
File "/usr/local/lib/python3.7/dist-packages/iceberg/api/expressions/projections.py", line 84, in predicate
return super(InclusiveProjection, self).predicate(pred)
File "/usr/local/lib/python3.7/dist-packages/iceberg/api/expressions/projections.py", line 68, in predicate
bound = pred.bind(self.spec.schema.as_struct(), case_sensitive=self.case_sensitive)
File "/usr/local/lib/python3.7/dist-packages/iceberg/api/expressions/predicate.py", line 228, in bind
return self.bind_literal_operation(bound)
File "/usr/local/lib/python3.7/dist-packages/iceberg/api/expressions/predicate.py", line 278, in bind_literal_operation
(bound_term.type, self.lit, self.lit.__class__.__name__))
File "/usr/local/lib/python3.7/dist-packages/iceberg/exceptions/exceptions.py", line 44, in check
raise ValidationException(message, args)
iceberg.exceptions.exceptions.ValidationException: Invalid Value for conversion to type boolean: "true" (StringLiteral)```
Created the PR to mitigate the issue, this bug is also present in `pyiceberg` api, I will create separate PR to resolve the same. After the change tests are passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pritampan commented on a diff in pull request #6843: [Python Legacy] Convert string to boolean if the binding variable is Boolean
Posted by "pritampan (via GitHub)" <gi...@apache.org>.
pritampan commented on code in PR #6843:
URL: https://github.com/apache/iceberg/pull/6843#discussion_r1107425732
##########
python_legacy/iceberg/api/expressions/literals.py:
##########
@@ -414,6 +414,8 @@ def to(self, type_var): # noqa: C901
return DecimalLiteral(Decimal(str(self.value))
.quantize(Decimal("." + "".join(["0" for i in range(1, type_var.scale)]) + "1"),
rounding=ROUND_HALF_UP))
+ elif type_var.type_id == TypeID.BOOLEAN and self.value.upper() in ['TRUE', 'FALSE']:
Review Comment:
thanks @puchengy for checking ,it will by default raise the following, let me know if it works.
`iceberg.exceptions.exceptions.ValidationException: Invalid Value for conversion to type boolean: "false1" (StringLiteral)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] puchengy commented on a diff in pull request #6843: [Python Legacy] Convert string to boolean if the binding variable is Boolean
Posted by "puchengy (via GitHub)" <gi...@apache.org>.
puchengy commented on code in PR #6843:
URL: https://github.com/apache/iceberg/pull/6843#discussion_r1107438791
##########
python_legacy/iceberg/api/expressions/literals.py:
##########
@@ -414,6 +414,8 @@ def to(self, type_var): # noqa: C901
return DecimalLiteral(Decimal(str(self.value))
.quantize(Decimal("." + "".join(["0" for i in range(1, type_var.scale)]) + "1"),
rounding=ROUND_HALF_UP))
+ elif type_var.type_id == TypeID.BOOLEAN and self.value.upper() in ['TRUE', 'FALSE']:
Review Comment:
got it, approve. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko merged pull request #6843: [Python Legacy] Convert string to boolean if the binding variable is Boolean
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #6843:
URL: https://github.com/apache/iceberg/pull/6843
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] puchengy commented on a diff in pull request #6843: [Python Legacy] Convert string to boolean if the binding variable is Boolean
Posted by "puchengy (via GitHub)" <gi...@apache.org>.
puchengy commented on code in PR #6843:
URL: https://github.com/apache/iceberg/pull/6843#discussion_r1107407490
##########
python_legacy/iceberg/api/expressions/literals.py:
##########
@@ -414,6 +414,8 @@ def to(self, type_var): # noqa: C901
return DecimalLiteral(Decimal(str(self.value))
.quantize(Decimal("." + "".join(["0" for i in range(1, type_var.scale)]) + "1"),
rounding=ROUND_HALF_UP))
+ elif type_var.type_id == TypeID.BOOLEAN and self.value.upper() in ['TRUE', 'FALSE']:
Review Comment:
What about throwing exception when value is either true or false?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on pull request #6843: [Python Legacy] Convert string to boolean if the binding variable is Boolean
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #6843:
URL: https://github.com/apache/iceberg/pull/6843#issuecomment-1432656125
@pritampan keep in mind that the Python Legacy won't be released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org