You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/09 16:47:09 UTC
[GitHub] [hudi] rubenssoto opened a new issue #3954: [Feature Request] Integration with AWS Glue
rubenssoto opened a new issue #3954:
URL: https://github.com/apache/hudi/issues/3954
Hello,
We are migrating our spark stack from EMR to EKS, we use Hudi in all our pipelines. Spark Doesn't have an integration with AWS Glue, so to make Hudi works with Glue I will have to have a EMR with Hive in the middle.
It would be good if Hudi had an integration direct to Aws Glue.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jasondavindev commented on issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
jasondavindev commented on issue #3954:
URL: https://github.com/apache/hudi/issues/3954#issuecomment-1011418585
At @emcasa we're using EMR on EKS. We overwritten EMR Hudi jars from 0.8.0 to 0.10.0, then we can leverage hive sync tool and sync tables with Glue Data Catalog.
For this, you must pass the follow spark config:
```
spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
```
For a new session, use
```python
.enableHiveSupport()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3954:
URL: https://github.com/apache/hudi/issues/3954#issuecomment-964737205
@rubenssoto Hudi hive sync works out-of-box with Glue metastore see section "Registering a Hudi dataset with your metastore" in https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-how-it-works.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rubenssoto commented on issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
rubenssoto commented on issue #3954:
URL: https://github.com/apache/hudi/issues/3954#issuecomment-966769977
@kenny291 Good tip
I will try this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #3954:
URL: https://github.com/apache/hudi/issues/3954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jasondavindev removed a comment on issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
jasondavindev removed a comment on issue #3954:
URL: https://github.com/apache/hudi/issues/3954#issuecomment-1011418585
At @emcasa we're using EMR on EKS. We overwritten EMR Hudi jars from 0.8.0 to 0.10.0, then we can leverage hive sync tool and sync tables with Glue Data Catalog.
For this, you must pass the follow spark config:
```
spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
```
For a new session, use
```python
.enableHiveSupport()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kenny291 commented on issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
kenny291 commented on issue #3954:
URL: https://github.com/apache/hudi/issues/3954#issuecomment-966767110
You can use this https://github.com/awslabs/aws-glue-catalog-sync-agent-for-hive to sync table metadata to Glue catalog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #3954:
URL: https://github.com/apache/hudi/issues/3954#issuecomment-965561631
@rmahindra123 did have a plain glue client. that uses direct Glue APIs for syncing. Is that what you are looking at?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
kazdy commented on issue #3954:
URL: https://github.com/apache/hudi/issues/3954#issuecomment-1018633780
Hi @jasondavindev do you have any issues with running 0.10.0 with EMR on EKS or it's straight forward and works with no issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3954:
URL: https://github.com/apache/hudi/issues/3954#issuecomment-968104802
Thanks all for chiming in. filed https://issues.apache.org/jira/browse/HUDI-2757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jasondavindev commented on issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
jasondavindev commented on issue #3954:
URL: https://github.com/apache/hudi/issues/3954#issuecomment-1011419612
@rubenssoto @xushiyan
At @emcasa we're using EMR on EKS. We overwritten EMR Hudi jars from 0.8.0 to 0.10.0, then we can leverage hive sync tool and sync tables with Glue Data Catalog.
For this, you must pass the follow spark config:
```
spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
```
For a new session, use
```python
.enableHiveSupport()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rubenssoto commented on issue #3954: [Feature Request] Integration with AWS Glue
Posted by GitBox <gi...@apache.org>.
rubenssoto commented on issue #3954:
URL: https://github.com/apache/hudi/issues/3954#issuecomment-966463841
@vinothchandar thats it...
For us is not urgent because probably we will deploy our own Hive Metastore, we are making tests, but I think it will a good feature for Hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org