You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/05/05 14:52:05 UTC
[GitHub] [hudi] jkdll opened a new issue #2916: [SUPPORT] Hudi + AWS Redshift Spectrum Table
jkdll opened a new issue #2916:
URL: https://github.com/apache/hudi/issues/2916
Hi,
We are running Hudi on AWS EMR 5.33, with Hive configured to use an external metastore hosted on a MySQL Database (via Amazon RDS). We are attempting to read data from AWS Redshift Spectrum using an external table.
First we ingest the data in Hudi, then add partition on Hive, then create the external schema. Upon querying, we get the following error (full error below):
`SQL Error [500310] [XX000]: [Amazon](500310) Invalid operation: Assert`
The create external schema command for Spectrum Redshift is:
```
CREATE EXTERNAL SCHEMA spectrum_hudi
FROM hive metastore
database 'hudi_db'
uri '**.***.*.***'
iam_role 'arn:aws:iam::************:role/**********************************'
```
After creation of the external schema, the Hudi table is visible, but when we query we get the following error:
```
SQL Error [500310] [XX000]: [Amazon](500310) Invalid operation: Assert
Details:
-----------------------------------------------
error: Assert
code: 1000
context: hudi::ParsedFilename::IsValidCommitTimestamp( std::string(ctx.hudi_commit_timestamp)) -
query: 93954011
location: scan_range_manager.cpp:826
process: query2_980_93954011 [pid=70458]
-----------------------------------------------;
```
Could you kindly assist?
The query via Hive CLI works well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2916: [SUPPORT] Hudi + AWS Redshift Spectrum Table
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2916:
URL: https://github.com/apache/hudi/issues/2916#issuecomment-842012462
@jkdll From what I can guess, this look like an issue with the way the Hudi table was registered with AWS Redshift. Can you shed some more light on
1. how you registered the hudi table in Redshift ?
2. Run describe table on the hudi table in Redshift to see the hudi table schema and properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2916: [SUPPORT] Hudi + AWS Redshift Spectrum Table
Posted by GitBox <gi...@apache.org>.
n3nash closed issue #2916:
URL: https://github.com/apache/hudi/issues/2916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2916: [SUPPORT] Hudi + AWS Redshift Spectrum Table
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2916:
URL: https://github.com/apache/hudi/issues/2916#issuecomment-859328811
@jkdll Closing this ticket due to inactivity. Please feel free to re-open with answers if you still need assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2916: [SUPPORT] Hudi + AWS Redshift Spectrum Table
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2916:
URL: https://github.com/apache/hudi/issues/2916#issuecomment-842012462
@jkdll From what I can guess, this look like an issue with the way the Hudi table was registered with AWS Redshift. Can you shed some more light on
1. how you registered the hudi table in Redshift ?
2. Run describe table on the hudi table in Redshift to see the hudi table schema and properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org