You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/25 07:47:02 UTC
[GitHub] [hudi] waywtdcc opened a new issue #4683: [SUPPORT] Hive ro table read error
waywtdcc opened a new issue #4683:
URL: https://github.com/apache/hudi/issues/4683
**To Reproduce**
`
CREATE TABLE `hudi.user_cdc17`(
)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
`
`
CREATE EXTERNAL TABLE `hudi.user_cdc17_ro`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` bigint COMMENT '',
`name` string COMMENT '',
`birthday` bigint COMMENT '',
`ts` bigint COMMENT '')
PARTITIONED BY (
`date_str` string COMMENT '')
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
'hoodie.query.as.ro.table'='true')
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.HoodieParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
'hdfs://test/user/hive/warehouse/hudi.db/user_cdc17'
TBLPROPERTIES (
'last_commit_time_sync'='20220125081930201',
'spark.sql.sources.provider'='hudi',
'spark.sql.sources.schema.numPartCols'='1',
'spark.sql.sources.schema.numParts'='1'
`
`select count(*)
from test.users_cdc_hive t1
left join hudi.user_cdc17_ro t2 on t1.id = t2.id
and t1.name = t2.name and cast(t1.ts as bigint) = t2.ts ;`
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.10
* Spark version :
* Hive version : 2.3.6
* Hadoop version : 2.7.7
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4683:
URL: https://github.com/apache/hudi/issues/4683#issuecomment-1039623905
@waywtdcc : do you have any updates based on Raymond's suggestion above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4683:
URL: https://github.com/apache/hudi/issues/4683#issuecomment-1025261010
@waywtdcc the stacktrace shows "unsupported type: optional int96 ts" which does not originate from hudi. looks like a problem with the hive ql specifying bigint? have you tried different data type like TIMESTAMP for `ts` ? also does this `ts` come from `users_cdc_hive` or `user_cdc17_ro` ? you probably need to double check the schemas for both tables and make sure int96 is not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] waywtdcc closed issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
waywtdcc closed issue #4683:
URL: https://github.com/apache/hudi/issues/4683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4683:
URL: https://github.com/apache/hudi/issues/4683#issuecomment-1061360382
will wait for a week and close out the github issue if we don't hear back from you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4683:
URL: https://github.com/apache/hudi/issues/4683#issuecomment-1039623905
@waywtdcc : do you have any updates based on Raymond's suggestion above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] waywtdcc removed a comment on issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
waywtdcc removed a comment on issue #4683:
URL: https://github.com/apache/hudi/issues/4683#issuecomment-1061486598
> @waywtdcc 堆栈跟踪显示“不支持的类型：可选的 int96 ts”，它并非源自 hudi。看起来像 hive ql 指定 bigint 的问题？你尝试过不同的数据类型，比如 TIMESTAMP`ts`吗？这也是`ts`来自`users_cdc_hive`还是`user_cdc17_ro`？您可能需要仔细检查两个表的模式并确保未使用 int96。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] waywtdcc commented on issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
waywtdcc commented on issue #4683:
URL: https://github.com/apache/hudi/issues/4683#issuecomment-1061486598
> @waywtdcc 堆栈跟踪显示“不支持的类型：可选的 int96 ts”，它并非源自 hudi。看起来像 hive ql 指定 bigint 的问题？你尝试过不同的数据类型，比如 TIMESTAMP`ts`吗？这也是`ts`来自`users_cdc_hive`还是`user_cdc17_ro`？您可能需要仔细检查两个表的模式并确保未使用 int96。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] waywtdcc commented on issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
waywtdcc commented on issue #4683:
URL: https://github.com/apache/hudi/issues/4683#issuecomment-1061486790
> will wait for a week and close out the github issue if we don't hear back from you.
Sorry, I didn't notice. This is the problem of hive query. The timestamp type field stored in the Hudi flick table is automatically converted to bigint in the hive ro table, but the query of version 2.3.6 hive is not compatible with the bigint transferred from timestamp; The query result with spark is normal and no error is reported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4683:
URL: https://github.com/apache/hudi/issues/4683#issuecomment-1057514841
@waywtdcc : gentle ping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] waywtdcc edited a comment on issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
waywtdcc edited a comment on issue #4683:
URL: https://github.com/apache/hudi/issues/4683#issuecomment-1061486790
> will wait for a week and close out the github issue if we don't hear back from you.
Sorry, I didn't notice. This is the problem of hive query. The timestamp type field stored in the Hudi flink table is automatically converted to bigint in the hive ro table, but the query of version 2.3.6 hive is not compatible with the bigint transferred from timestamp; The query result with spark is normal and no error is reported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] waywtdcc closed issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
waywtdcc closed issue #4683:
URL: https://github.com/apache/hudi/issues/4683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4683: [SUPPORT] Hive ro table read error
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4683:
URL: https://github.com/apache/hudi/issues/4683#issuecomment-1047398703
@waywtdcc : if you got unblocked or resolved the issue, feel free to close the github issue. Or do furnish more details/updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org