RE: [EXTERNAL]Re: aws glue connector

["Katz, David L via user" <us...@flink.apache.org>]

Hi Danny-

Thanks for your response!

At first glance, I’m not sure, will drill in to look deeper.  For clarification, a few points I probably should have expressed initially:

  1.  I’m running flink as a Kinesis Data Application (AWS serverless application)
  2.  The reason I want to use  streaming (and also static) glue tables instead of the underlying kinesis data stream (or s3 buckets) as a source are that I will be leveraging multiple AWS accounts and this is our security strategy (as opposed to using cross account access).
  3.  So in other words, I might have a static glue table backed by an s3 bucket in one account and a streaming glue table backed by a kinesis data stream in another account and want to join the two as part of a streaming application by just going to the central lake database as a consumer.

Thanks again!
-Dave

David L Katz (he/him/his) why?<https://sites.jpmchase.com/sites/companyhome/about%20us/diversity/pages/language,-inclusion-%26-gender-pronous.aspx> | Executive Director | Macro Technology | 545 Washington Blvd, Floor 4 | Jersey City, NJ 07310

From: Danny Cranmer <da...@apache.org>
Sent: Tuesday, December 20, 2022 3:12 PM
To: Katz, David L (CIB Tech, USA) <da...@jpmchase.com>
Cc: user <us...@flink.apache.org>
Subject: [EXTERNAL]Re: aws glue connector

Hello David,

There is a FLIP [1] to add native Glue Catalog support and we already have Glue Schema Registry format plugins [2][3], however these are data stream API only. Are you intending on just using the Glue schema features or leveraging other features? Would either of the things I mentioned help with your usecase?

Thanks,
Danny

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-277%3A+Native+GlueCatalog+Support+in+Flink<https://secureweb.jpmchase.net/readonly/https:/cwiki.apache.org/confluence/display/FLINK/FLIP-277%3A+Native+GlueCatalog+Support+in+Flink>
[2] https://github.com/apache/flink/tree/master/flink-formats/flink-avro-glue-schema-registry<https://secureweb.jpmchase.net/readonly/https:/github.com/apache/flink/tree/master/flink-formats/flink-avro-glue-schema-registry>
[3] https://github.com/apache/flink/tree/master/flink-formats/flink-json-glue-schema-registry<https://secureweb.jpmchase.net/readonly/https:/github.com/apache/flink/tree/master/flink-formats/flink-json-glue-schema-registry>

On Mon, Dec 19, 2022 at 2:31 PM Katz, David L via user <us...@flink.apache.org>> wrote:
Hi-

Has anyone used a glue table connector (specifically trying to get a streaming glue table that sits on top of a kinesis data stream)?  I have been using the kinesis stream connector but want to integrate with lake formation so would like to take advantage of the glue table layer.

Thanks,
-Dave

David L Katz (he/him/his) why?<https://sites.jpmchase.com/sites/companyhome/about%20us/diversity/pages/language,-inclusion-%26-gender-pronous.aspx> | Executive Director | Macro Technology | 545 Washington Blvd, Floor 4 | Jersey City, NJ 07310


This message is confidential and subject to terms at: https://www.jpmorgan.com/emaildisclaimer including on confidential, privileged or legal entity information, malicious content and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.

This message is confidential and subject to terms at: https://www.jpmorgan.com/emaildisclaimer including on confidential, privileged or legal entity information, malicious content and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.