You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "paul8263 (via GitHub)" <gi...@apache.org> on 2024/01/17 07:13:37 UTC
[PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
paul8263 opened a new pull request, #10517:
URL: https://github.com/apache/hudi/pull/10517
…ng date comparison operator.
### Change Logs
When using between, less than (less than or equal) or greater than (greater than or equal) operators with field typed of date, the date type will unexpected convert to Long, which is incompatible with its primitive type INT32.
### Impact
No impact.
### Risk level (write none, low medium or high below)
Low risk level.
### Documentation Update
No need to update the documentation.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #10517:
URL: https://github.com/apache/hudi/pull/10517#discussion_r1456767443
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionPredicates.java:
##########
@@ -602,10 +602,10 @@ private static FilterPredicate toParquetPredicate(FunctionDefinition functionDef
case TINYINT:
case SMALLINT:
case INTEGER:
+ case DATE:
case TIME_WITHOUT_TIME_ZONE:
return predicateSupportsLtGt(functionDefinition, intColumn(columnName), (Integer) literal);
case BIGINT:
- case DATE:
case TIMESTAMP_WITHOUT_TIME_ZONE:
Review Comment:
Is there any exception thrown here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #10517:
URL: https://github.com/apache/hudi/pull/10517#issuecomment-1901638589
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=21999&view=logs&j=600e7de6-e133-5e69-e615-50ee129b3c08&t=bbbd7bcc-ae73-56b8-887a-cd2d6deaafc7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10517:
URL: https://github.com/apache/hudi/pull/10517#issuecomment-1903405196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "513d914fa72c497458c834d0b33962996b3d3e03",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21999",
"triggerID" : "513d914fa72c497458c834d0b33962996b3d3e03",
"triggerType" : "PUSH"
}, {
"hash" : "a16c9cd026ff8c822405189dbf07b3d77b042512",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a16c9cd026ff8c822405189dbf07b3d77b042512",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 513d914fa72c497458c834d0b33962996b3d3e03 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21999)
* a16c9cd026ff8c822405189dbf07b3d77b042512 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #10517:
URL: https://github.com/apache/hudi/pull/10517#discussion_r1458194926
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionPredicates.java:
##########
@@ -602,10 +602,10 @@ private static FilterPredicate toParquetPredicate(FunctionDefinition functionDef
case TINYINT:
case SMALLINT:
case INTEGER:
+ case DATE:
case TIME_WITHOUT_TIME_ZONE:
return predicateSupportsLtGt(functionDefinition, intColumn(columnName), (Integer) literal);
case BIGINT:
- case DATE:
case TIMESTAMP_WITHOUT_TIME_ZONE:
Review Comment:
Then why we need this fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10517:
URL: https://github.com/apache/hudi/pull/10517#issuecomment-1903414508
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "513d914fa72c497458c834d0b33962996b3d3e03",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21999",
"triggerID" : "513d914fa72c497458c834d0b33962996b3d3e03",
"triggerType" : "PUSH"
}, {
"hash" : "a16c9cd026ff8c822405189dbf07b3d77b042512",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22089",
"triggerID" : "a16c9cd026ff8c822405189dbf07b3d77b042512",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 513d914fa72c497458c834d0b33962996b3d3e03 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21999)
* a16c9cd026ff8c822405189dbf07b3d77b042512 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22089)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10517:
URL: https://github.com/apache/hudi/pull/10517#issuecomment-1895276998
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "513d914fa72c497458c834d0b33962996b3d3e03",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "513d914fa72c497458c834d0b33962996b3d3e03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 513d914fa72c497458c834d0b33962996b3d3e03 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "paul8263 (via GitHub)" <gi...@apache.org>.
paul8263 commented on code in PR #10517:
URL: https://github.com/apache/hudi/pull/10517#discussion_r1458202085
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionPredicates.java:
##########
@@ -602,10 +602,10 @@ private static FilterPredicate toParquetPredicate(FunctionDefinition functionDef
case TINYINT:
case SMALLINT:
case INTEGER:
+ case DATE:
case TIME_WITHOUT_TIME_ZONE:
return predicateSupportsLtGt(functionDefinition, intColumn(columnName), (Integer) literal);
case BIGINT:
- case DATE:
case TIMESTAMP_WITHOUT_TIME_ZONE:
Review Comment:
Hi @danny0405 ,
Please see: [HUDI-7303](https://issues.apache.org/jira/browse/HUDI-7303).
In parquet, date type is stored as INT32 (epoch day). But if we add some conditions with date typed field in SQL where clause, its type will unexpectedly convert to Long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #10517:
URL: https://github.com/apache/hudi/pull/10517#issuecomment-1899550901
There are test failures:
```xml
[ERROR] Errors:
[ERROR] TestHoodieTableSource.testBucketPruningSpecialKeyDataType:267 » ClassCast java...
[ERROR] TestHoodieTableSource.testBucketPruningSpecialKeyDataType:267 » ClassCast java...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "paul8263 (via GitHub)" <gi...@apache.org>.
paul8263 commented on PR #10517:
URL: https://github.com/apache/hudi/pull/10517#issuecomment-1899570791
> There are test failures:
>
> ```
> [ERROR] Errors:
> [ERROR] TestHoodieTableSource.testBucketPruningSpecialKeyDataType:267 » ClassCast java...
> [ERROR] TestHoodieTableSource.testBucketPruningSpecialKeyDataType:267 » ClassCast java...
> ```
Err... I could not find exceptions in CI check reports. Could you please help me about the detailed error stacktrace? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10517:
URL: https://github.com/apache/hudi/pull/10517#issuecomment-1895461041
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "513d914fa72c497458c834d0b33962996b3d3e03",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21999",
"triggerID" : "513d914fa72c497458c834d0b33962996b3d3e03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 513d914fa72c497458c834d0b33962996b3d3e03 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21999)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10517:
URL: https://github.com/apache/hudi/pull/10517#issuecomment-1895288131
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "513d914fa72c497458c834d0b33962996b3d3e03",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21999",
"triggerID" : "513d914fa72c497458c834d0b33962996b3d3e03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 513d914fa72c497458c834d0b33962996b3d3e03 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21999)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "paul8263 (via GitHub)" <gi...@apache.org>.
paul8263 commented on code in PR #10517:
URL: https://github.com/apache/hudi/pull/10517#discussion_r1457191018
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionPredicates.java:
##########
@@ -602,10 +602,10 @@ private static FilterPredicate toParquetPredicate(FunctionDefinition functionDef
case TINYINT:
case SMALLINT:
case INTEGER:
+ case DATE:
case TIME_WITHOUT_TIME_ZONE:
return predicateSupportsLtGt(functionDefinition, intColumn(columnName), (Integer) literal);
case BIGINT:
- case DATE:
case TIMESTAMP_WITHOUT_TIME_ZONE:
Review Comment:
Hi @danny0405 ,
As far as I could see there should be no exceptions thrown here. I have gone through all TPCDS queries. They could work well with all patches applied that I sumbitted recently.
If there might be some potential risks for this change, please let me know. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10517:
URL: https://github.com/apache/hudi/pull/10517#issuecomment-1903669814
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "513d914fa72c497458c834d0b33962996b3d3e03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21999",
"triggerID" : "513d914fa72c497458c834d0b33962996b3d3e03",
"triggerType" : "PUSH"
}, {
"hash" : "a16c9cd026ff8c822405189dbf07b3d77b042512",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22089",
"triggerID" : "a16c9cd026ff8c822405189dbf07b3d77b042512",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a16c9cd026ff8c822405189dbf07b3d77b042512 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22089)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7303] Fix date field type unexpectedly convert to Long when usi… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #10517:
URL: https://github.com/apache/hudi/pull/10517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org