You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Samrat Deb <de...@gmail.com> on 2023/08/29 06:05:39 UTC
Re: [DISCUSS] FLIP-307: Flink connector Redshift
Hi Martijn,
I fully comprehend your apprehensions regarding AWS's reliance on JDBC and
the resultant impact on the release cycle. However, an effective approach
is to utilize the latest version of flink-connector-jdbc as a Maven
dependency. In this arrangement, flink-connector-aws can be associated with
an older version of flink-connector-jdbc. It's not obligatory for
flink-connector-aws to constantly align with the most current version. This
strategy disengages direct dependencies between the two in their respective
release cycles. By adopting this approach, flink-connector-aws can maintain
its specific dependencies on JDBC and the preferred version paralleling its
approach with the AWS SDK's handling of dependencies.
Bests,
Samrat
On Tue, Jul 4, 2023 at 7:09 PM Martijn Visser <ma...@apache.org>
wrote:
> Hi Samrat,
>
> The dependency between JDBC and AWS worries me a lot: we're already seeing
> that coupling causes a lot of issues down the line. Why can't we decouple
> these?
>
> Best regards,
>
> Martijn
>
> On Tue, Jul 4, 2023 at 3:35 PM Samrat Deb <de...@gmail.com> wrote:
>
> > Hi Leonard,
> >
> > Sorry for the late reply.
> >
> > > 1 Reusing the capabilities of JDBC and Filesystem in the Redshift
> > connector generally makes sense to me. However, since all of them are
> > managed in different repositories and depend on Flink dependency, could
> you
> > explain how you establish the versioning, release, and dependency
> > management process?
> >
> > We intend to maintain the current release cycle for the Flink Connector
> > AWS. However, we will be incorporating a dependency of Flink Connector
> JDBC
> > to most recent version, which will synchronize between the flink
> connector
> > AWS and flink connector JDBC connectors releases for new Flink version
> > support. Additionally, the Flink Connector Redshift will exclusively
> > utilize the public API, minimizing the occurrence of immediate breaking
> > changes.
> >
> > > 2 Some configuration option names can be improved to match the naming
> > style of existing configuration options, for example:
> > table -> table-name
> > query -> scan.query
> > aws-iam-role -> aws.iam-role
> > read.mode -> scan.read.mode: similar to scan.startup.mode , and maybe we
> > will have lookup.read.mode
> > write.mode -> sink.write.mode
> >
> > Updated the FLIP-307 [1]
> >
> > > 3 The source of Redshift connector supports JDBC queries, IIUC, we can
> > also support the LookupTableSource as well?
> >
> > we will support `LookupTableSource` in redshift connector source
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> >
> > Bests,
> > Samrat
> >
> > On Wed, Jun 21, 2023 at 4:47 PM Leonard Xu <xb...@gmail.com> wrote:
> >
> > > Thanks Samrat for driving this FLIP.
> > >
> > > Since the community has already built a set of basic components for the
> > > connector, I only have three comments.
> > >
> > > 1 Reusing the capabilities of JDBC and Filesystem in the Redshift
> > > connector generally makes sense to me. However, since all of them are
> > > managed in different repositories and depend on Flink dependency, could
> > you
> > > explain how you establish the versioning, release, and dependency
> > > management process?
> > >
> > > 2 Some configuration option names can be improved to match the naming
> > > style of existing configuration options, for example:
> > > table -> table-name
> > > query -> scan.query
> > > aws-iam-role -> aws.iam-role
> > > read.mode -> scan.read.mode: similar to scan.startup.mode , and maybe
> we
> > > will have lookup.read.mode
> > > write.mode -> sink.write.mode
> > >
> > > 3 The source of Redshift connector supports JDBC queries, IIUC, we can
> > > also support the LookupTableSource as well?
> > >
> > > Best,
> > > Leonard
> > >
> > > > On Jun 21, 2023, at 4:57 PM, Samrat Deb <de...@gmail.com>
> wrote:
> > > >
> > > > Hi Martijn,
> > > >
> > > > Thank you for sharing your thoughts on the matter.
> > > > I understand that you don't have a strong opinion on whether to
> support
> > > > exactly-once processing from the beginning or at a later stage.
> > > > For initial implementation I will go ahead with at-least-once
> > semantics.
> > > >
> > > >> The only consideration that I could think of is that
> > > > if you start with at-least-once, you could consider using the ASync
> > API,
> > > > but I don't think the ASync API yet supports exactly-once.
> > > >
> > > > Noted. It's a valid consideration to start compatibility with the
> Async
> > > > API.
> > > >
> > > > Bests,
> > > > Samrat
> > > >
> > > >
> > > > On Mon, Jun 19, 2023 at 5:28 PM Martijn Visser <
> > martijnvisser@apache.org
> > > >
> > > > wrote:
> > > >
> > > >> Hi Samrat,
> > > >>
> > > >> I have no strong opinion on whether to support exactly-once from the
> > > start
> > > >> or potentially later. The only consideration that I could think of
> is
> > > that
> > > >> if you start with at-least-once, you could consider using the ASync
> > API,
> > > >> but I don't think the ASync API yet supports exactly-once.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Martijn
> > > >>
> > > >> On Fri, Jun 9, 2023 at 7:22 PM Jing Ge <ji...@ververica.com.invalid>
> > > wrote:
> > > >>
> > > >>> Hi Samrat,
> > > >>>
> > > >>> The FLIP looks good, thanks!
> > > >>>
> > > >>> Best regards,
> > > >>> Jing
> > > >>>
> > > >>>
> > > >>> On Tue, Jun 6, 2023 at 8:16 PM Samrat Deb <de...@gmail.com>
> > > wrote:
> > > >>>
> > > >>>> Hi Jing,
> > > >>>>
> > > >>>>> I would suggest adding that information into the
> > > >>>> FLIP.
> > > >>>>
> > > >>>> Updated now, please review the new version of flip whenever time.
> > > >>>>
> > > >>>>> +1 Looking forward to your PR :-)
> > > >>>> I will request for your review once m ready with PR :-)
> > > >>>>
> > > >>>> Bests,
> > > >>>> Samrat
> > > >>>>
> > > >>>> On Tue, Jun 6, 2023 at 11:43 PM Samrat Deb <decordeapex@gmail.com
> >
> > > >>> wrote:
> > > >>>>
> > > >>>>> Hi Martijn,
> > > >>>>>
> > > >>>>>> If I understand this correctly, the Redshift sink
> > > >>>>> would not be able to support exactly-once, is that correct?
> > > >>>>>
> > > >>>>> As I delve deeper into the study of Redshift's capabilities, I
> have
> > > >>>>> discovered that it does support "merge into" operations [1] and
> > some
> > > >>>>> merge into examples [2].
> > > >>>>> This opens up the possibility of implementing exactly-once
> > semantics
> > > >>> with
> > > >>>>> the connector.
> > > >>>>> However, I believe it would be prudent to start with a more
> focused
> > > >>> scope
> > > >>>>> for the initial phase of implementation and defer the exact-once
> > > >>> support
> > > >>>>> for subsequent iterations.
> > > >>>>>
> > > >>>>> Before finalizing the approach, I would greatly appreciate your
> > > >>> thoughts
> > > >>>>> and suggestions on this matter.
> > > >>>>> Should we prioritize the initial implementation without
> > exactly-once
> > > >>>>> support, or would you advise incorporating it right from the
> start?
> > > >>>>> Your insights and experiences would be immensely valuable in
> making
> > > >>> this
> > > >>>>> decision.
> > > >>>>>
> > > >>>>>
> > > >>>>> [1]
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://docs.aws.amazon.com/redshift/latest/dg/t_updating-inserting-using-staging-tables-.html
> > > >>>>> [2]
> > > >> https://docs.aws.amazon.com/redshift/latest/dg/merge-examples.html
> > > >>>>>
> > > >>>>> Bests,
> > > >>>>> Samrat
> > > >>>>>
> > > >>>>> On Mon, Jun 5, 2023 at 7:09 PM Jing Ge
> <jing@ververica.com.invalid
> > >
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> Hi Samrat,
> > > >>>>>>
> > > >>>>>> Thanks for the feedback. I would suggest adding that information
> > > >> into
> > > >>>> the
> > > >>>>>> FLIP.
> > > >>>>>>
> > > >>>>>> +1 Looking forward to your PR :-)
> > > >>>>>>
> > > >>>>>> Best regards,
> > > >>>>>> Jing
> > > >>>>>>
> > > >>>>>> On Sat, Jun 3, 2023 at 9:19 PM Samrat Deb <
> decordeapex@gmail.com>
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi Jing Ge,
> > > >>>>>>>
> > > >>>>>>>>>> Do you already have any prototype? I'd like to join the
> > > >> reviews.
> > > >>>>>>> The prototype is in progress. I will raise the dedicated PR for
> > > >>> review
> > > >>>>>> soon
> > > >>>>>>> also notify in this thread as well .
> > > >>>>>>>
> > > >>>>>>>>>> Will the Redshift connector provide additional features
> > > >>>>>>> beyond the mediator/wrapper of the jdbc connector?
> > > >>>>>>>
> > > >>>>>>> Here are the additional features that the Flink connector for
> AWS
> > > >>>>>> Redshift
> > > >>>>>>> can provide on top of using JDBC:
> > > >>>>>>>
> > > >>>>>>> 1. Integration with AWS Redshift Workload Management (WLM): AWS
> > > >>>> Redshift
> > > >>>>>>> allows you to configure WLM[1] to manage query prioritization
> and
> > > >>>>>> resource
> > > >>>>>>> allocation. The Flink connector for Redshift will be agnostic
> to
> > > >> the
> > > >>>>>>> configured WLM and utilize it for scaling in and out for the
> > sink.
> > > >>>> This
> > > >>>>>>> means that the connector can leverage the WLM capabilities of
> > > >>> Redshift
> > > >>>>>> to
> > > >>>>>>> optimize the execution of queries and allocate resources
> > > >> efficiently
> > > >>>>>> based
> > > >>>>>>> on your defined workload priorities.
> > > >>>>>>>
> > > >>>>>>> 2. Abstraction of AWS Redshift Quotas and Limits: AWS Redshift
> > > >>> imposes
> > > >>>>>>> certain quotas and limits[2] on various aspects such as the
> > number
> > > >>> of
> > > >>>>>>> clusters, concurrent connections, queries per second, etc. The
> > > >> Flink
> > > >>>>>>> connector for Redshift will provide an abstraction layer for
> > > >> users,
> > > >>>>>>> allowing them to work with Redshift without having to worry
> about
> > > >>>> these
> > > >>>>>>> specific limits. The connector will handle the management of
> > > >>>> connections
> > > >>>>>>> and queries within the defined quotas and limits, abstracting
> > away
> > > >>> the
> > > >>>>>>> complexity and ensuring compliance with Redshift's
> restrictions.
> > > >>>>>>>
> > > >>>>>>> These features aim to simplify the integration of Flink with
> AWS
> > > >>>>>> Redshift,
> > > >>>>>>> providing optimized resource utilization and transparent
> handling
> > > >> of
> > > >>>>>>> Redshift-specific limitations.
> > > >>>>>>>
> > > >>>>>>> Bests,
> > > >>>>>>> Samrat
> > > >>>>>>>
> > > >>>>>>> [1]
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://docs.aws.amazon.com/redshift/latest/dg/cm-c-implementing-workload-management.html
> > > >>>>>>> [2]
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://docs.aws.amazon.com/redshift/latest/mgmt/amazon-redshift-limits.html
> > > >>>>>>>
> > > >>>>>>> On Sat, Jun 3, 2023 at 11:40 PM Samrat Deb <
> > decordeapex@gmail.com
> > > >>>
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi Ahmed,
> > > >>>>>>>>
> > > >>>>>>>>>>> please let me know If you need any collaboration regarding
> > > >>>>>>> integration
> > > >>>>>>>> with
> > > >>>>>>>> AWS connectors credential providers or regarding FLIP-171 I
> > > >> would
> > > >>> be
> > > >>>>>> more
> > > >>>>>>>> than happy to assist.
> > > >>>>>>>>
> > > >>>>>>>> Sure, I will reach out incase of any hands required.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Jun 2, 2023 at 6:12 PM Jing Ge
> > > >> <jing@ververica.com.invalid
> > > >>>>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi Samrat,
> > > >>>>>>>>>
> > > >>>>>>>>> Excited to see your proposal. Supporting data warehouses is
> one
> > > >>> of
> > > >>>>>> the
> > > >>>>>>>>> major tracks for Flink. Thanks for driving it! Happy to see
> > > >> that
> > > >>> we
> > > >>>>>>>>> reached
> > > >>>>>>>>> consensus to prioritize the Sink over Source in the previous
> > > >>>>>> discussion.
> > > >>>>>>>>> Do
> > > >>>>>>>>> you already have any prototype? I'd like to join the reviews.
> > > >>>>>>>>>
> > > >>>>>>>>> Just out of curiosity, speaking of JDBC mode, according to
> the
> > > >>>> FLIP,
> > > >>>>>> it
> > > >>>>>>>>> should be doable to directly use the jdbc connector with
> > > >>> Redshift,
> > > >>>>>> if I
> > > >>>>>>> am
> > > >>>>>>>>> not mistaken. Will the Redshift connector provide additional
> > > >>>> features
> > > >>>>>>>>> beyond the mediator/wrapper of the jdbc connector?
> > > >>>>>>>>>
> > > >>>>>>>>> Best regards,
> > > >>>>>>>>> Jing
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jun 1, 2023 at 8:22 PM Ahmed Hamdy <
> > > >> hamdy10024@gmail.com
> > > >>>>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Samrat
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for putting up this FLIP. I agree regarding the
> > > >>> importance
> > > >>>>>> of
> > > >>>>>>> the
> > > >>>>>>>>>> use case.
> > > >>>>>>>>>> please let me know If you need any collaboration regarding
> > > >>>>>> integration
> > > >>>>>>>>> with
> > > >>>>>>>>>> AWS connectors credential providers or regarding FLIP-171 I
> > > >>> would
> > > >>>>>> be
> > > >>>>>>>>> more
> > > >>>>>>>>>> than happy to assist.
> > > >>>>>>>>>> I also like Leonard's proposal for starting with
> > > >> DataStreamSink
> > > >>>> and
> > > >>>>>>>>>> TableSink, It would be great to have some milestones
> > > >> delivered
> > > >>> as
> > > >>>>>> soon
> > > >>>>>>>>> as
> > > >>>>>>>>>> ready.
> > > >>>>>>>>>> best regards
> > > >>>>>>>>>> Ahmed Hamdy
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Wed, 31 May 2023 at 11:15, Samrat Deb <
> > > >>> decordeapex@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi Liu Ron,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> 1. Regarding the `read.mode` and `write.mode`, you say
> > > >>> here
> > > >>>>>>>>> provides
> > > >>>>>>>>>> two
> > > >>>>>>>>>>> modes, respectively, jdbc and `unload or copy`, What is the
> > > >>>>>> default
> > > >>>>>>>>> value
> > > >>>>>>>>>>> for `read.mode` and `write.mode?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I have made an effort to make the configuration options
> > > >>>>>> `read.mode`
> > > >>>>>>>>> and
> > > >>>>>>>>>>> `write.mode` mandatory for the "flink-connector-redshift"
> > > >>>>>> according
> > > >>>>>>> to
> > > >>>>>>>>>>> FLIP[1]. The rationale behind this decision is to empower
> > > >>> users
> > > >>>>>> who
> > > >>>>>>>>> are
> > > >>>>>>>>>>> familiar with their Redshift setup and have specific
> > > >>>> expectations
> > > >>>>>>> for
> > > >>>>>>>>> the
> > > >>>>>>>>>>> sink. By making these configurations mandatory, users can
> > > >>> have
> > > >>>>>> more
> > > >>>>>>>>>> control
> > > >>>>>>>>>>> and flexibility in configuring the connector to meet their
> > > >>>>>>>>> requirements.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> However, I am open to receiving feedback on whether it
> > > >> would
> > > >>> be
> > > >>>>>>>>>> beneficial
> > > >>>>>>>>>>> to make the configuration options non-mandatory and set
> > > >>> default
> > > >>>>>>> values
> > > >>>>>>>>>> for
> > > >>>>>>>>>>> them. If you believe there are advantages to having default
> > > >>>>>> values
> > > >>>>>>> or
> > > >>>>>>>>> any
> > > >>>>>>>>>>> other suggestions, please share your thoughts. Your
> > > >> feedback
> > > >>> is
> > > >>>>>>> highly
> > > >>>>>>>>>>> appreciated.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> 2. For Source, does it both support batch read and
> > > >>> streaming
> > > >>>>>>> read?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Redshift currently does not provide native support for
> > > >>>> streaming
> > > >>>>>>>>> reads,
> > > >>>>>>>>>>> although it does support streaming writes[2]. As part of
> > > >> the
> > > >>>>>> plan, I
> > > >>>>>>>>>> intend
> > > >>>>>>>>>>> to conduct a proof of concept and benchmarking to explore
> > > >> the
> > > >>>>>>>>>> possibilities
> > > >>>>>>>>>>> of implementing streaming reads using the Flink JDBC
> > > >>> connector,
> > > >>>>>> as
> > > >>>>>>>>>> Redshift
> > > >>>>>>>>>>> is JDBC compatible.
> > > >>>>>>>>>>> However, it is important to note that, in the initial phase
> > > >>> of
> > > >>>>>>>>>>> implementation, the focus will primarily be on supporting
> > > >>> batch
> > > >>>>>>> reads
> > > >>>>>>>>>>> rather than streaming reads. This approach will allow us to
> > > >>>>>> deliver
> > > >>>>>>> a
> > > >>>>>>>>>>> robust and reliable solution for batch processing in phase
> > > >> 2
> > > >>> of
> > > >>>>>> the
> > > >>>>>>>>>>> implementation.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [1]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> > > >>>>>>>>>>> [2]
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion.html
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Bests,
> > > >>>>>>>>>>> Samrat
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, May 31, 2023 at 8:03 AM liu ron <
> > > >> ron9.liu@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi, Samrat
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for driving this FLIP. It looks like supporting
> > > >>>>>>>>>>>> flink-connector-redshift is very useful to Flink. I have
> > > >>> two
> > > >>>>>>>>> question:
> > > >>>>>>>>>>>> 1. Regarding the `read.mode` and `write.mode`, you say
> > > >>> here
> > > >>>>>>>>> provides
> > > >>>>>>>>>> two
> > > >>>>>>>>>>>> modes, respectively, jdbc and `unload or copy`, What is
> > > >> the
> > > >>>>>>> default
> > > >>>>>>>>>> value
> > > >>>>>>>>>>>> for `read.mode` and `write.mode?
> > > >>>>>>>>>>>> 2. For Source, does it both support batch read and
> > > >>> streaming
> > > >>>>>> read?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>> Ron
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Samrat Deb <de...@gmail.com> 于2023年5月30日周二
> > > >> 17:15写道：
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> [1]
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> [note] Missed the trailing link for previous mail
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Tue, May 30, 2023 at 2:43 PM Samrat Deb <
> > > >>>>>>> decordeapex@gmail.com
> > > >>>>>>>>>>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi Leonard,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> and I’m glad to help review the design as well as
> > > >> the
> > > >>>>>> code
> > > >>>>>>>>>> review.
> > > >>>>>>>>>>>>>> Thank you so much. It would be really great and
> > > >> helpful
> > > >>>> to
> > > >>>>>>> bring
> > > >>>>>>>>>>>>>> flink-connector-redshift for flink users :) .
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I have divided the implementation in 3 phases in the
> > > >>>>>> `Scope`
> > > >>>>>>>>>>>> Section[1].
> > > >>>>>>>>>>>>>> 1st phase is to
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> - Integrate with Flink Sink API (*FLIP-171*
> > > >>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> )
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> About the implementation phases, How about
> > > >>> prioritizing
> > > >>>>>>>>> support
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> Datastream Sink API and TableSink API in the first
> > > >>> phase?
> > > >>>>>>>>>>>>>> I can completely resonate with you to prioritize
> > > >>> support
> > > >>>>>> for
> > > >>>>>>>>>>> Datastream
> > > >>>>>>>>>>>>>> Sink API and TableSink API in the first phase.
> > > >>>>>>>>>>>>>> I will update the FLIP[1] as you have suggested.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> It seems that the primary use cases for the
> > > >> Redshift
> > > >>>>>>> connector
> > > >>>>>>>>>> are
> > > >>>>>>>>>>>>>> acting as a sink for processed data by Flink.
> > > >>>>>>>>>>>>>> Yes, majority ask and requirement for Redshift
> > > >>> connector
> > > >>>> is
> > > >>>>>>> sink
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>>> processed data by Flink.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Bests,
> > > >>>>>>>>>>>>>> Samrat
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Tue, May 30, 2023 at 12:35 PM Leonard Xu <
> > > >>>>>>> xbjtdcq@gmail.com>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks @Samrat for bringing this discussion.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> It makes sense to me to introduce AWS Redshift
> > > >>> connector
> > > >>>>>> for
> > > >>>>>>>>>> Apache
> > > >>>>>>>>>>>>>>> Flink, and I’m glad to help review the design as
> > > >> well
> > > >>> as
> > > >>>>>> the
> > > >>>>>>>>> code
> > > >>>>>>>>>>>>> review.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> About the implementation phases, How about
> > > >>> prioritizing
> > > >>>>>>> support
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> Datastream Sink API and TableSink API in the first
> > > >>>> phase?
> > > >>>>>> It
> > > >>>>>>>>> seems
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> primary use cases for the Redshift connector are
> > > >>> acting
> > > >>>>>> as a
> > > >>>>>>>>> sink
> > > >>>>>>>>>>> for
> > > >>>>>>>>>>>>>>> processed data by Flink.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>> Leonard
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On May 29, 2023, at 12:51 PM, Samrat Deb <
> > > >>>>>>>>> decordeapex@gmail.com
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hello all ,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Context:
> > > >>>>>>>>>>>>>>>> Amazon Redshift [1] is a fully managed,
> > > >>> petabyte-scale
> > > >>>>>> data
> > > >>>>>>>>>>>> warehouse
> > > >>>>>>>>>>>>>>>> service in the cloud. It allows analyzing data
> > > >>> without
> > > >>>>>> all
> > > >>>>>>> of
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> configurations of a provisioned data warehouse.
> > > >>>>>> Resources
> > > >>>>>>> are
> > > >>>>>>>>>>>>>>> automatically
> > > >>>>>>>>>>>>>>>> provisioned and data warehouse capacity is
> > > >>>> intelligently
> > > >>>>>>>>> scaled
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>>>>> deliver
> > > >>>>>>>>>>>>>>>> fast performance for even the most demanding and
> > > >>>>>>>>> unpredictable
> > > >>>>>>>>>>>>>>> workloads.
> > > >>>>>>>>>>>>>>>> Redshift is one of the widely used warehouse
> > > >>> solutions
> > > >>>>>> in
> > > >>>>>>> the
> > > >>>>>>>>>>>> current
> > > >>>>>>>>>>>>>>>> market.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Building flink connector redshift will allow flink
> > > >>>>>> users to
> > > >>>>>>>>> have
> > > >>>>>>>>>>>>> source
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>> sink directly to redshift. It will help flink to
> > > >>>> expand
> > > >>>>>> the
> > > >>>>>>>>>> scope
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> redshift as a new connector in the ecosystem.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I would like to start a discussion on the
> > > >> FLIP-307:
> > > >>>>>> Flink
> > > >>>>>>>>>>> connector
> > > >>>>>>>>>>>>>>>> redshift [2].
> > > >>>>>>>>>>>>>>>> Looking forward to comments, feedbacks and
> > > >>> suggestions
> > > >>>>>> from
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>>>> community
> > > >>>>>>>>>>>>>>>> on the proposal.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> [1]
> > > >>>>>>>>>>
> > > >> https://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html
> > > >>>>>>>>>>>>>>>> [2]
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Bests,
> > > >>>>>>>>>>>>>>>> Samrat
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] FLIP-307: Flink connector Redshift
Posted by Danny Cranmer <da...@apache.org>.
Thanks Samrat, this makes sense to me and addresses connector coupling
concerns.
On Sat, Sep 16, 2023 at 4:30 PM Samrat Deb <de...@gmail.com> wrote:
> Hi ,
>
> I've made updates to the FLIP[1] by incorporating relevant changes to avoid
> using the Flink connector JDBC. This decision was based on the following
> reasons:
>
> AWS Redshift utilizes its specialized JDBC driver[2]. Given that their JDBC
> driver may undergo evolutions over time, using the Flink connector JDBC
> might face compatibility issues.
> Considering the dependency issues mentioned in the thread.
> Additionally, a Proof of Concept (POC) has been implemented for a
> DynamicSink for Redshift[3]. This showcases that utilizing
> flink-connector-jdbc for Redshift doesn't provide significant benefits.
> The JDBC mode offers more flexibility by allowing direct use of the JDBC
> driver, enabling Flink connector Redshift to evolve independently.
>
> [1] -
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> [2] -
>
> https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
> [3] -
> https://github.com/Samrat002/flink-connector-aws/tree/redshift-connector
>
> Bests,
> Samrat
>
>
>
>
> On Sat, Sep 16, 2023 at 10:46 AM Samrat Deb <de...@gmail.com> wrote:
>
> > Hello Martijn,
> >
> > I apologize for the delay in responding.
> >
> > Regarding your question about integrating Redshift directly into the JDBC
> > connector, we are planning to offer two modes: JDBC and UNLOAD. Through
> our
> > internal benchmarking, we have observed good performance in the UNLOAD
> > flow. Additionally, there is a need for both flows based on different
> user
> > use cases.
> >
> > If we were to explicitly add the JDBC mode to the flink-connector-jdbc,
> we
> > would have two options:
> >
> > 1. Include flink-connector-redshift in flink-connector-jdbc: This would
> > involve incorporating the Redshift connector into the JDBC connector.
> Since
> > Redshift is an AWS proprietary product, some authentication utilities can
> > be utilized from flink-connector-aws-base. If additional utilities are
> > required from the Redshift connector, they could be added to
> > flink-connector-aws-base. In my opinion, this approach is favorable as it
> > keeps everything related to AWS in flink-connector-aws.
> >
> > 2. Implement JDBC mode for Redshift sink in flink-connector-jdbc and
> > UNLOAD in flink-connector-aws: This alternative is not advisable as it
> > could lead to maintenance challenges and complexities.
> >
> >
> > Furthermore, it's important to highlight that Redshift has its own
> > customized JDBC driver[1], specifically optimized for compatibility with
> > Redshift. While I cannot confirm this definitively, there is a
> possibility
> > that the Redshift JDBC driver [1] might have differences in compatibility
> > when compared to the JDBC driver used in flink-connector-jdbc. This
> > suggests that if flink-connector-redshift were to rely on the JDBC
> > connector, it could potentially lead to future compatibility issues.
> >
> > Given these considerations, it seems prudent to maintain the
> > Redshift-related functionality within flink-connector-aws and keep the
> > Redshift connector independent of the JDBC connector. This approach can
> > help ensure that the Redshift connector remains flexible and adaptable to
> > any potential changes in JDBC compatibility.
> >
> > I will update the FLIP[2] to remove dependencies on flink-connector-jdbc.
> >
> > [1]
> >
> https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> >
> > Bests,
> > Samrat
> >
> >
> > On Mon, Sep 11, 2023 at 1:21 PM Martijn Visser <martijnvisser@apache.org
> >
> > wrote:
> >
> >> Hi Samrat,
> >>
> >> I'm still having doubts about the dependency on the JDBC connector.
> When a
> >> user specifies 'read mode', it will use the JDBC connector under the
> hood.
> >> Why not integrate Redshift then directly in the JDBC connector itself?
> It
> >> removes the need for a dependency on the JDBC driver, especially keeping
> >> in
> >> mind that this driver uses the old SourceFunction/SinkFunction
> interfaces
> >> because it hasn't been migrated yet.
> >>
> >> Best regards,
> >>
> >> Martijn
> >>
> >> On Mon, Sep 11, 2023 at 8:54 AM Samrat Deb <de...@gmail.com>
> wrote:
> >>
> >> > Hi Leonard,
> >> >
> >> > > Do we have to rely on the latest version of JDBC Connector here?
> >> >
> >> > No, there's no need for us to depend on the latest version of the JDBC
> >> > Connector. Redshift has its dedicated JDBC driver [1], which includes
> >> > custom modifications tailored to Redshift's specific implementation
> >> needs.
> >> > This driver is the most suitable choice for our purposes.
> >> >
> >> >
> >> > > Could you collect the APIs that Redshift generally needs to use?
> >> >
> >> > I am actively working on it and making progress towards creating the
> >> POC.
> >> >
> >> > Bests,
> >> > Samrat
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
> >> >
> >> > On Mon, Sep 11, 2023 at 12:02 PM Samrat Deb <de...@gmail.com>
> >> wrote:
> >> >
> >> > > Hello Danny,
> >> > >
> >> > > I wanted to express my gratitude for your valuable feedback and
> >> > insightful
> >> > > suggestions.
> >> > >
> >> > > I will be revising the FLIP to incorporate all of your queries and
> >> review
> >> > > suggestions. Additionally, I plan to provide a Proof of Concept
> (POC)
> >> for
> >> > > the connector by the end of this week. This POC will address the
> >> points
> >> > > you've raised and ensure that the FLIP aligns with your
> >> recommendations.
> >> > >
> >> > > Thank you once again for your input.
> >> > >
> >> > > Bests,
> >> > > Samrat
> >> > >
> >> > > On Thu, Sep 7, 2023 at 10:21 PM Danny Cranmer <
> >> dannycranmer@apache.org>
> >> > > wrote:
> >> > >
> >> > >> Hello Leonard,
> >> > >>
> >> > >> > Do we have to rely on the latest version of JDBC Connector here?
> I
> >> > >> understand that as long as the version of flink minor is the same
> as
> >> the
> >> > >> JDBC Connector, Could you collect the APIs that Redshift generally
> >> needs
> >> > >> to
> >> > >> use?
> >> > >>
> >> > >> I agree we do not necessarily need to rely on the latest patch
> >> version,
> >> > >> only the same minor. The main issue for me is the dependency
> >> introduces
> >> > a
> >> > >> blocker following a new Flink version. For example, when Flink
> >> 1.18.0 is
> >> > >> released we cannot release the AWS connectors until the JDBC is
> >> > complete.
> >> > >> But I think this is a good tradeoff.
> >> > >>
> >> > >> > Splitting a separate redshift repository does not solve this
> >> coupling
> >> > >> problem
> >> > >>
> >> > >> Arguably it solves the AWS<>JDBC coupling problem, but creates a
> new,
> >> > more
> >> > >> complex one!
> >> > >>
> >> > >> Thanks,
> >> > >>
> >> > >> On Thu, Sep 7, 2023 at 5:26 PM Leonard Xu <xb...@gmail.com>
> wrote:
> >> > >>
> >> > >> > Thanks Samrat and Danny for driving this FLIP.
> >> > >> >
> >> > >> > >> an effective approach is to utilize the latest version of
> >> > >> > flink-connector-jdbc
> >> > >> > > as a Maven dependency
> >> > >> > >
> >> > >> > > When we have stable source/sink APIs and the connector versions
> >> are
> >> > >> > > decoupled from Flink this makes sense. But right now this would
> >> mean
> >> > >> that
> >> > >> > > the JDBC connector will block the AWS connector for each new
> >> Flink
> >> > >> > version
> >> > >> > > support release (1.18, 1.19, 1.20, 2.0 etc). That being said, I
> >> > cannot
> >> > >> > > think of a cleaner alternative, without pulling the core JDBC
> >> bits
> >> > out
> >> > >> > into
> >> > >> > > a dedicated project that is decoupled from and released
> >> > independently
> >> > >> of
> >> > >> > > Flink. Splitting flink-connector-redshift into a dedicated repo
> >> > would
> >> > >> > > decouple AWS/JDBC, but obviously introduce a new connector that
> >> is
> >> > >> > blocked
> >> > >> > > by both AWS and JDBC.
> >> > >> >
> >> > >> > Do we have to rely on the latest version of JDBC Connector here?
> I
> >> > >> > understand that as long as the version of flink minor is the same
> >> as
> >> > the
> >> > >> > JDBC Connector, Could you collect the APIs that Redshift
> generally
> >> > >> needs to
> >> > >> > use?
> >> > >> >
> >> > >> > Assuming that AWS Connector(Redshift) depends on JDBC Connector
> and
> >> > >> wants
> >> > >> > a higher version of JDBC Connector, I understand that the correct
> >> > >> approach
> >> > >> > is to promote the release of JDBC Connector and looks like we
> have
> >> no
> >> > >> more
> >> > >> > options.
> >> > >> >
> >> > >> > Splitting a separate redshift repository does not solve this
> >> coupling
> >> > >> > problem, from a user perspective, redshift should also be in the
> >> AWS
> >> > >> > Connector repo.
> >> > >> >
> >> > >> > Best,
> >> > >> > Leonard
> >> > >>
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] FLIP-307: Flink connector Redshift
Posted by Samrat Deb <de...@gmail.com>.
Hi ,
I've made updates to the FLIP[1] by incorporating relevant changes to avoid
using the Flink connector JDBC. This decision was based on the following
reasons:
AWS Redshift utilizes its specialized JDBC driver[2]. Given that their JDBC
driver may undergo evolutions over time, using the Flink connector JDBC
might face compatibility issues.
Considering the dependency issues mentioned in the thread.
Additionally, a Proof of Concept (POC) has been implemented for a
DynamicSink for Redshift[3]. This showcases that utilizing
flink-connector-jdbc for Redshift doesn't provide significant benefits.
The JDBC mode offers more flexibility by allowing direct use of the JDBC
driver, enabling Flink connector Redshift to evolve independently.
[1] -
https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
[2] -
https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
[3] -
https://github.com/Samrat002/flink-connector-aws/tree/redshift-connector
Bests,
Samrat
On Sat, Sep 16, 2023 at 10:46 AM Samrat Deb <de...@gmail.com> wrote:
> Hello Martijn,
>
> I apologize for the delay in responding.
>
> Regarding your question about integrating Redshift directly into the JDBC
> connector, we are planning to offer two modes: JDBC and UNLOAD. Through our
> internal benchmarking, we have observed good performance in the UNLOAD
> flow. Additionally, there is a need for both flows based on different user
> use cases.
>
> If we were to explicitly add the JDBC mode to the flink-connector-jdbc, we
> would have two options:
>
> 1. Include flink-connector-redshift in flink-connector-jdbc: This would
> involve incorporating the Redshift connector into the JDBC connector. Since
> Redshift is an AWS proprietary product, some authentication utilities can
> be utilized from flink-connector-aws-base. If additional utilities are
> required from the Redshift connector, they could be added to
> flink-connector-aws-base. In my opinion, this approach is favorable as it
> keeps everything related to AWS in flink-connector-aws.
>
> 2. Implement JDBC mode for Redshift sink in flink-connector-jdbc and
> UNLOAD in flink-connector-aws: This alternative is not advisable as it
> could lead to maintenance challenges and complexities.
>
>
> Furthermore, it's important to highlight that Redshift has its own
> customized JDBC driver[1], specifically optimized for compatibility with
> Redshift. While I cannot confirm this definitively, there is a possibility
> that the Redshift JDBC driver [1] might have differences in compatibility
> when compared to the JDBC driver used in flink-connector-jdbc. This
> suggests that if flink-connector-redshift were to rely on the JDBC
> connector, it could potentially lead to future compatibility issues.
>
> Given these considerations, it seems prudent to maintain the
> Redshift-related functionality within flink-connector-aws and keep the
> Redshift connector independent of the JDBC connector. This approach can
> help ensure that the Redshift connector remains flexible and adaptable to
> any potential changes in JDBC compatibility.
>
> I will update the FLIP[2] to remove dependencies on flink-connector-jdbc.
>
> [1]
> https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
>
> Bests,
> Samrat
>
>
> On Mon, Sep 11, 2023 at 1:21 PM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi Samrat,
>>
>> I'm still having doubts about the dependency on the JDBC connector. When a
>> user specifies 'read mode', it will use the JDBC connector under the hood.
>> Why not integrate Redshift then directly in the JDBC connector itself? It
>> removes the need for a dependency on the JDBC driver, especially keeping
>> in
>> mind that this driver uses the old SourceFunction/SinkFunction interfaces
>> because it hasn't been migrated yet.
>>
>> Best regards,
>>
>> Martijn
>>
>> On Mon, Sep 11, 2023 at 8:54 AM Samrat Deb <de...@gmail.com> wrote:
>>
>> > Hi Leonard,
>> >
>> > > Do we have to rely on the latest version of JDBC Connector here?
>> >
>> > No, there's no need for us to depend on the latest version of the JDBC
>> > Connector. Redshift has its dedicated JDBC driver [1], which includes
>> > custom modifications tailored to Redshift's specific implementation
>> needs.
>> > This driver is the most suitable choice for our purposes.
>> >
>> >
>> > > Could you collect the APIs that Redshift generally needs to use?
>> >
>> > I am actively working on it and making progress towards creating the
>> POC.
>> >
>> > Bests,
>> > Samrat
>> >
>> > [1]
>> >
>> >
>> https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
>> >
>> > On Mon, Sep 11, 2023 at 12:02 PM Samrat Deb <de...@gmail.com>
>> wrote:
>> >
>> > > Hello Danny,
>> > >
>> > > I wanted to express my gratitude for your valuable feedback and
>> > insightful
>> > > suggestions.
>> > >
>> > > I will be revising the FLIP to incorporate all of your queries and
>> review
>> > > suggestions. Additionally, I plan to provide a Proof of Concept (POC)
>> for
>> > > the connector by the end of this week. This POC will address the
>> points
>> > > you've raised and ensure that the FLIP aligns with your
>> recommendations.
>> > >
>> > > Thank you once again for your input.
>> > >
>> > > Bests,
>> > > Samrat
>> > >
>> > > On Thu, Sep 7, 2023 at 10:21 PM Danny Cranmer <
>> dannycranmer@apache.org>
>> > > wrote:
>> > >
>> > >> Hello Leonard,
>> > >>
>> > >> > Do we have to rely on the latest version of JDBC Connector here? I
>> > >> understand that as long as the version of flink minor is the same as
>> the
>> > >> JDBC Connector, Could you collect the APIs that Redshift generally
>> needs
>> > >> to
>> > >> use?
>> > >>
>> > >> I agree we do not necessarily need to rely on the latest patch
>> version,
>> > >> only the same minor. The main issue for me is the dependency
>> introduces
>> > a
>> > >> blocker following a new Flink version. For example, when Flink
>> 1.18.0 is
>> > >> released we cannot release the AWS connectors until the JDBC is
>> > complete.
>> > >> But I think this is a good tradeoff.
>> > >>
>> > >> > Splitting a separate redshift repository does not solve this
>> coupling
>> > >> problem
>> > >>
>> > >> Arguably it solves the AWS<>JDBC coupling problem, but creates a new,
>> > more
>> > >> complex one!
>> > >>
>> > >> Thanks,
>> > >>
>> > >> On Thu, Sep 7, 2023 at 5:26 PM Leonard Xu <xb...@gmail.com> wrote:
>> > >>
>> > >> > Thanks Samrat and Danny for driving this FLIP.
>> > >> >
>> > >> > >> an effective approach is to utilize the latest version of
>> > >> > flink-connector-jdbc
>> > >> > > as a Maven dependency
>> > >> > >
>> > >> > > When we have stable source/sink APIs and the connector versions
>> are
>> > >> > > decoupled from Flink this makes sense. But right now this would
>> mean
>> > >> that
>> > >> > > the JDBC connector will block the AWS connector for each new
>> Flink
>> > >> > version
>> > >> > > support release (1.18, 1.19, 1.20, 2.0 etc). That being said, I
>> > cannot
>> > >> > > think of a cleaner alternative, without pulling the core JDBC
>> bits
>> > out
>> > >> > into
>> > >> > > a dedicated project that is decoupled from and released
>> > independently
>> > >> of
>> > >> > > Flink. Splitting flink-connector-redshift into a dedicated repo
>> > would
>> > >> > > decouple AWS/JDBC, but obviously introduce a new connector that
>> is
>> > >> > blocked
>> > >> > > by both AWS and JDBC.
>> > >> >
>> > >> > Do we have to rely on the latest version of JDBC Connector here? I
>> > >> > understand that as long as the version of flink minor is the same
>> as
>> > the
>> > >> > JDBC Connector, Could you collect the APIs that Redshift generally
>> > >> needs to
>> > >> > use?
>> > >> >
>> > >> > Assuming that AWS Connector(Redshift) depends on JDBC Connector and
>> > >> wants
>> > >> > a higher version of JDBC Connector, I understand that the correct
>> > >> approach
>> > >> > is to promote the release of JDBC Connector and looks like we have
>> no
>> > >> more
>> > >> > options.
>> > >> >
>> > >> > Splitting a separate redshift repository does not solve this
>> coupling
>> > >> > problem, from a user perspective, redshift should also be in the
>> AWS
>> > >> > Connector repo.
>> > >> >
>> > >> > Best,
>> > >> > Leonard
>> > >>
>> > >
>> >
>>
>
Re: [DISCUSS] FLIP-307: Flink connector Redshift
Posted by Samrat Deb <de...@gmail.com>.
Hello Martijn,
I apologize for the delay in responding.
Regarding your question about integrating Redshift directly into the JDBC
connector, we are planning to offer two modes: JDBC and UNLOAD. Through our
internal benchmarking, we have observed good performance in the UNLOAD
flow. Additionally, there is a need for both flows based on different user
use cases.
If we were to explicitly add the JDBC mode to the flink-connector-jdbc, we
would have two options:
1. Include flink-connector-redshift in flink-connector-jdbc: This would
involve incorporating the Redshift connector into the JDBC connector. Since
Redshift is an AWS proprietary product, some authentication utilities can
be utilized from flink-connector-aws-base. If additional utilities are
required from the Redshift connector, they could be added to
flink-connector-aws-base. In my opinion, this approach is favorable as it
keeps everything related to AWS in flink-connector-aws.
2. Implement JDBC mode for Redshift sink in flink-connector-jdbc and UNLOAD
in flink-connector-aws: This alternative is not advisable as it could lead
to maintenance challenges and complexities.
Furthermore, it's important to highlight that Redshift has its own
customized JDBC driver[1], specifically optimized for compatibility with
Redshift. While I cannot confirm this definitively, there is a possibility
that the Redshift JDBC driver [1] might have differences in compatibility
when compared to the JDBC driver used in flink-connector-jdbc. This
suggests that if flink-connector-redshift were to rely on the JDBC
connector, it could potentially lead to future compatibility issues.
Given these considerations, it seems prudent to maintain the
Redshift-related functionality within flink-connector-aws and keep the
Redshift connector independent of the JDBC connector. This approach can
help ensure that the Redshift connector remains flexible and adaptable to
any potential changes in JDBC compatibility.
I will update the FLIP[2] to remove dependencies on flink-connector-jdbc.
[1]
https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
Bests,
Samrat
On Mon, Sep 11, 2023 at 1:21 PM Martijn Visser <ma...@apache.org>
wrote:
> Hi Samrat,
>
> I'm still having doubts about the dependency on the JDBC connector. When a
> user specifies 'read mode', it will use the JDBC connector under the hood.
> Why not integrate Redshift then directly in the JDBC connector itself? It
> removes the need for a dependency on the JDBC driver, especially keeping in
> mind that this driver uses the old SourceFunction/SinkFunction interfaces
> because it hasn't been migrated yet.
>
> Best regards,
>
> Martijn
>
> On Mon, Sep 11, 2023 at 8:54 AM Samrat Deb <de...@gmail.com> wrote:
>
> > Hi Leonard,
> >
> > > Do we have to rely on the latest version of JDBC Connector here?
> >
> > No, there's no need for us to depend on the latest version of the JDBC
> > Connector. Redshift has its dedicated JDBC driver [1], which includes
> > custom modifications tailored to Redshift's specific implementation
> needs.
> > This driver is the most suitable choice for our purposes.
> >
> >
> > > Could you collect the APIs that Redshift generally needs to use?
> >
> > I am actively working on it and making progress towards creating the POC.
> >
> > Bests,
> > Samrat
> >
> > [1]
> >
> >
> https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
> >
> > On Mon, Sep 11, 2023 at 12:02 PM Samrat Deb <de...@gmail.com>
> wrote:
> >
> > > Hello Danny,
> > >
> > > I wanted to express my gratitude for your valuable feedback and
> > insightful
> > > suggestions.
> > >
> > > I will be revising the FLIP to incorporate all of your queries and
> review
> > > suggestions. Additionally, I plan to provide a Proof of Concept (POC)
> for
> > > the connector by the end of this week. This POC will address the points
> > > you've raised and ensure that the FLIP aligns with your
> recommendations.
> > >
> > > Thank you once again for your input.
> > >
> > > Bests,
> > > Samrat
> > >
> > > On Thu, Sep 7, 2023 at 10:21 PM Danny Cranmer <dannycranmer@apache.org
> >
> > > wrote:
> > >
> > >> Hello Leonard,
> > >>
> > >> > Do we have to rely on the latest version of JDBC Connector here? I
> > >> understand that as long as the version of flink minor is the same as
> the
> > >> JDBC Connector, Could you collect the APIs that Redshift generally
> needs
> > >> to
> > >> use?
> > >>
> > >> I agree we do not necessarily need to rely on the latest patch
> version,
> > >> only the same minor. The main issue for me is the dependency
> introduces
> > a
> > >> blocker following a new Flink version. For example, when Flink 1.18.0
> is
> > >> released we cannot release the AWS connectors until the JDBC is
> > complete.
> > >> But I think this is a good tradeoff.
> > >>
> > >> > Splitting a separate redshift repository does not solve this
> coupling
> > >> problem
> > >>
> > >> Arguably it solves the AWS<>JDBC coupling problem, but creates a new,
> > more
> > >> complex one!
> > >>
> > >> Thanks,
> > >>
> > >> On Thu, Sep 7, 2023 at 5:26 PM Leonard Xu <xb...@gmail.com> wrote:
> > >>
> > >> > Thanks Samrat and Danny for driving this FLIP.
> > >> >
> > >> > >> an effective approach is to utilize the latest version of
> > >> > flink-connector-jdbc
> > >> > > as a Maven dependency
> > >> > >
> > >> > > When we have stable source/sink APIs and the connector versions
> are
> > >> > > decoupled from Flink this makes sense. But right now this would
> mean
> > >> that
> > >> > > the JDBC connector will block the AWS connector for each new Flink
> > >> > version
> > >> > > support release (1.18, 1.19, 1.20, 2.0 etc). That being said, I
> > cannot
> > >> > > think of a cleaner alternative, without pulling the core JDBC bits
> > out
> > >> > into
> > >> > > a dedicated project that is decoupled from and released
> > independently
> > >> of
> > >> > > Flink. Splitting flink-connector-redshift into a dedicated repo
> > would
> > >> > > decouple AWS/JDBC, but obviously introduce a new connector that is
> > >> > blocked
> > >> > > by both AWS and JDBC.
> > >> >
> > >> > Do we have to rely on the latest version of JDBC Connector here? I
> > >> > understand that as long as the version of flink minor is the same as
> > the
> > >> > JDBC Connector, Could you collect the APIs that Redshift generally
> > >> needs to
> > >> > use?
> > >> >
> > >> > Assuming that AWS Connector(Redshift) depends on JDBC Connector and
> > >> wants
> > >> > a higher version of JDBC Connector, I understand that the correct
> > >> approach
> > >> > is to promote the release of JDBC Connector and looks like we have
> no
> > >> more
> > >> > options.
> > >> >
> > >> > Splitting a separate redshift repository does not solve this
> coupling
> > >> > problem, from a user perspective, redshift should also be in the AWS
> > >> > Connector repo.
> > >> >
> > >> > Best,
> > >> > Leonard
> > >>
> > >
> >
>
Re: [DISCUSS] FLIP-307: Flink connector Redshift
Posted by Martijn Visser <ma...@apache.org>.
Hi Samrat,
I'm still having doubts about the dependency on the JDBC connector. When a
user specifies 'read mode', it will use the JDBC connector under the hood.
Why not integrate Redshift then directly in the JDBC connector itself? It
removes the need for a dependency on the JDBC driver, especially keeping in
mind that this driver uses the old SourceFunction/SinkFunction interfaces
because it hasn't been migrated yet.
Best regards,
Martijn
On Mon, Sep 11, 2023 at 8:54 AM Samrat Deb <de...@gmail.com> wrote:
> Hi Leonard,
>
> > Do we have to rely on the latest version of JDBC Connector here?
>
> No, there's no need for us to depend on the latest version of the JDBC
> Connector. Redshift has its dedicated JDBC driver [1], which includes
> custom modifications tailored to Redshift's specific implementation needs.
> This driver is the most suitable choice for our purposes.
>
>
> > Could you collect the APIs that Redshift generally needs to use?
>
> I am actively working on it and making progress towards creating the POC.
>
> Bests,
> Samrat
>
> [1]
>
> https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
>
> On Mon, Sep 11, 2023 at 12:02 PM Samrat Deb <de...@gmail.com> wrote:
>
> > Hello Danny,
> >
> > I wanted to express my gratitude for your valuable feedback and
> insightful
> > suggestions.
> >
> > I will be revising the FLIP to incorporate all of your queries and review
> > suggestions. Additionally, I plan to provide a Proof of Concept (POC) for
> > the connector by the end of this week. This POC will address the points
> > you've raised and ensure that the FLIP aligns with your recommendations.
> >
> > Thank you once again for your input.
> >
> > Bests,
> > Samrat
> >
> > On Thu, Sep 7, 2023 at 10:21 PM Danny Cranmer <da...@apache.org>
> > wrote:
> >
> >> Hello Leonard,
> >>
> >> > Do we have to rely on the latest version of JDBC Connector here? I
> >> understand that as long as the version of flink minor is the same as the
> >> JDBC Connector, Could you collect the APIs that Redshift generally needs
> >> to
> >> use?
> >>
> >> I agree we do not necessarily need to rely on the latest patch version,
> >> only the same minor. The main issue for me is the dependency introduces
> a
> >> blocker following a new Flink version. For example, when Flink 1.18.0 is
> >> released we cannot release the AWS connectors until the JDBC is
> complete.
> >> But I think this is a good tradeoff.
> >>
> >> > Splitting a separate redshift repository does not solve this coupling
> >> problem
> >>
> >> Arguably it solves the AWS<>JDBC coupling problem, but creates a new,
> more
> >> complex one!
> >>
> >> Thanks,
> >>
> >> On Thu, Sep 7, 2023 at 5:26 PM Leonard Xu <xb...@gmail.com> wrote:
> >>
> >> > Thanks Samrat and Danny for driving this FLIP.
> >> >
> >> > >> an effective approach is to utilize the latest version of
> >> > flink-connector-jdbc
> >> > > as a Maven dependency
> >> > >
> >> > > When we have stable source/sink APIs and the connector versions are
> >> > > decoupled from Flink this makes sense. But right now this would mean
> >> that
> >> > > the JDBC connector will block the AWS connector for each new Flink
> >> > version
> >> > > support release (1.18, 1.19, 1.20, 2.0 etc). That being said, I
> cannot
> >> > > think of a cleaner alternative, without pulling the core JDBC bits
> out
> >> > into
> >> > > a dedicated project that is decoupled from and released
> independently
> >> of
> >> > > Flink. Splitting flink-connector-redshift into a dedicated repo
> would
> >> > > decouple AWS/JDBC, but obviously introduce a new connector that is
> >> > blocked
> >> > > by both AWS and JDBC.
> >> >
> >> > Do we have to rely on the latest version of JDBC Connector here? I
> >> > understand that as long as the version of flink minor is the same as
> the
> >> > JDBC Connector, Could you collect the APIs that Redshift generally
> >> needs to
> >> > use?
> >> >
> >> > Assuming that AWS Connector(Redshift) depends on JDBC Connector and
> >> wants
> >> > a higher version of JDBC Connector, I understand that the correct
> >> approach
> >> > is to promote the release of JDBC Connector and looks like we have no
> >> more
> >> > options.
> >> >
> >> > Splitting a separate redshift repository does not solve this coupling
> >> > problem, from a user perspective, redshift should also be in the AWS
> >> > Connector repo.
> >> >
> >> > Best,
> >> > Leonard
> >>
> >
>
Re: [DISCUSS] FLIP-307: Flink connector Redshift
Posted by Samrat Deb <de...@gmail.com>.
Hi Leonard,
> Do we have to rely on the latest version of JDBC Connector here?
No, there's no need for us to depend on the latest version of the JDBC
Connector. Redshift has its dedicated JDBC driver [1], which includes
custom modifications tailored to Redshift's specific implementation needs.
This driver is the most suitable choice for our purposes.
> Could you collect the APIs that Redshift generally needs to use?
I am actively working on it and making progress towards creating the POC.
Bests,
Samrat
[1]
https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-download-driver.html
On Mon, Sep 11, 2023 at 12:02 PM Samrat Deb <de...@gmail.com> wrote:
> Hello Danny,
>
> I wanted to express my gratitude for your valuable feedback and insightful
> suggestions.
>
> I will be revising the FLIP to incorporate all of your queries and review
> suggestions. Additionally, I plan to provide a Proof of Concept (POC) for
> the connector by the end of this week. This POC will address the points
> you've raised and ensure that the FLIP aligns with your recommendations.
>
> Thank you once again for your input.
>
> Bests,
> Samrat
>
> On Thu, Sep 7, 2023 at 10:21 PM Danny Cranmer <da...@apache.org>
> wrote:
>
>> Hello Leonard,
>>
>> > Do we have to rely on the latest version of JDBC Connector here? I
>> understand that as long as the version of flink minor is the same as the
>> JDBC Connector, Could you collect the APIs that Redshift generally needs
>> to
>> use?
>>
>> I agree we do not necessarily need to rely on the latest patch version,
>> only the same minor. The main issue for me is the dependency introduces a
>> blocker following a new Flink version. For example, when Flink 1.18.0 is
>> released we cannot release the AWS connectors until the JDBC is complete.
>> But I think this is a good tradeoff.
>>
>> > Splitting a separate redshift repository does not solve this coupling
>> problem
>>
>> Arguably it solves the AWS<>JDBC coupling problem, but creates a new, more
>> complex one!
>>
>> Thanks,
>>
>> On Thu, Sep 7, 2023 at 5:26 PM Leonard Xu <xb...@gmail.com> wrote:
>>
>> > Thanks Samrat and Danny for driving this FLIP.
>> >
>> > >> an effective approach is to utilize the latest version of
>> > flink-connector-jdbc
>> > > as a Maven dependency
>> > >
>> > > When we have stable source/sink APIs and the connector versions are
>> > > decoupled from Flink this makes sense. But right now this would mean
>> that
>> > > the JDBC connector will block the AWS connector for each new Flink
>> > version
>> > > support release (1.18, 1.19, 1.20, 2.0 etc). That being said, I cannot
>> > > think of a cleaner alternative, without pulling the core JDBC bits out
>> > into
>> > > a dedicated project that is decoupled from and released independently
>> of
>> > > Flink. Splitting flink-connector-redshift into a dedicated repo would
>> > > decouple AWS/JDBC, but obviously introduce a new connector that is
>> > blocked
>> > > by both AWS and JDBC.
>> >
>> > Do we have to rely on the latest version of JDBC Connector here? I
>> > understand that as long as the version of flink minor is the same as the
>> > JDBC Connector, Could you collect the APIs that Redshift generally
>> needs to
>> > use?
>> >
>> > Assuming that AWS Connector(Redshift) depends on JDBC Connector and
>> wants
>> > a higher version of JDBC Connector, I understand that the correct
>> approach
>> > is to promote the release of JDBC Connector and looks like we have no
>> more
>> > options.
>> >
>> > Splitting a separate redshift repository does not solve this coupling
>> > problem, from a user perspective, redshift should also be in the AWS
>> > Connector repo.
>> >
>> > Best,
>> > Leonard
>>
>
Re: [DISCUSS] FLIP-307: Flink connector Redshift
Posted by Samrat Deb <de...@gmail.com>.
Hello Danny,
I wanted to express my gratitude for your valuable feedback and insightful
suggestions.
I will be revising the FLIP to incorporate all of your queries and review
suggestions. Additionally, I plan to provide a Proof of Concept (POC) for
the connector by the end of this week. This POC will address the points
you've raised and ensure that the FLIP aligns with your recommendations.
Thank you once again for your input.
Bests,
Samrat
On Thu, Sep 7, 2023 at 10:21 PM Danny Cranmer <da...@apache.org>
wrote:
> Hello Leonard,
>
> > Do we have to rely on the latest version of JDBC Connector here? I
> understand that as long as the version of flink minor is the same as the
> JDBC Connector, Could you collect the APIs that Redshift generally needs to
> use?
>
> I agree we do not necessarily need to rely on the latest patch version,
> only the same minor. The main issue for me is the dependency introduces a
> blocker following a new Flink version. For example, when Flink 1.18.0 is
> released we cannot release the AWS connectors until the JDBC is complete.
> But I think this is a good tradeoff.
>
> > Splitting a separate redshift repository does not solve this coupling
> problem
>
> Arguably it solves the AWS<>JDBC coupling problem, but creates a new, more
> complex one!
>
> Thanks,
>
> On Thu, Sep 7, 2023 at 5:26 PM Leonard Xu <xb...@gmail.com> wrote:
>
> > Thanks Samrat and Danny for driving this FLIP.
> >
> > >> an effective approach is to utilize the latest version of
> > flink-connector-jdbc
> > > as a Maven dependency
> > >
> > > When we have stable source/sink APIs and the connector versions are
> > > decoupled from Flink this makes sense. But right now this would mean
> that
> > > the JDBC connector will block the AWS connector for each new Flink
> > version
> > > support release (1.18, 1.19, 1.20, 2.0 etc). That being said, I cannot
> > > think of a cleaner alternative, without pulling the core JDBC bits out
> > into
> > > a dedicated project that is decoupled from and released independently
> of
> > > Flink. Splitting flink-connector-redshift into a dedicated repo would
> > > decouple AWS/JDBC, but obviously introduce a new connector that is
> > blocked
> > > by both AWS and JDBC.
> >
> > Do we have to rely on the latest version of JDBC Connector here? I
> > understand that as long as the version of flink minor is the same as the
> > JDBC Connector, Could you collect the APIs that Redshift generally needs
> to
> > use?
> >
> > Assuming that AWS Connector(Redshift) depends on JDBC Connector and wants
> > a higher version of JDBC Connector, I understand that the correct
> approach
> > is to promote the release of JDBC Connector and looks like we have no
> more
> > options.
> >
> > Splitting a separate redshift repository does not solve this coupling
> > problem, from a user perspective, redshift should also be in the AWS
> > Connector repo.
> >
> > Best,
> > Leonard
>
Re: [DISCUSS] FLIP-307: Flink connector Redshift
Posted by Danny Cranmer <da...@apache.org>.
Hello Leonard,
> Do we have to rely on the latest version of JDBC Connector here? I
understand that as long as the version of flink minor is the same as the
JDBC Connector, Could you collect the APIs that Redshift generally needs to
use?
I agree we do not necessarily need to rely on the latest patch version,
only the same minor. The main issue for me is the dependency introduces a
blocker following a new Flink version. For example, when Flink 1.18.0 is
released we cannot release the AWS connectors until the JDBC is complete.
But I think this is a good tradeoff.
> Splitting a separate redshift repository does not solve this coupling
problem
Arguably it solves the AWS<>JDBC coupling problem, but creates a new, more
complex one!
Thanks,
On Thu, Sep 7, 2023 at 5:26 PM Leonard Xu <xb...@gmail.com> wrote:
> Thanks Samrat and Danny for driving this FLIP.
>
> >> an effective approach is to utilize the latest version of
> flink-connector-jdbc
> > as a Maven dependency
> >
> > When we have stable source/sink APIs and the connector versions are
> > decoupled from Flink this makes sense. But right now this would mean that
> > the JDBC connector will block the AWS connector for each new Flink
> version
> > support release (1.18, 1.19, 1.20, 2.0 etc). That being said, I cannot
> > think of a cleaner alternative, without pulling the core JDBC bits out
> into
> > a dedicated project that is decoupled from and released independently of
> > Flink. Splitting flink-connector-redshift into a dedicated repo would
> > decouple AWS/JDBC, but obviously introduce a new connector that is
> blocked
> > by both AWS and JDBC.
>
> Do we have to rely on the latest version of JDBC Connector here? I
> understand that as long as the version of flink minor is the same as the
> JDBC Connector, Could you collect the APIs that Redshift generally needs to
> use?
>
> Assuming that AWS Connector(Redshift) depends on JDBC Connector and wants
> a higher version of JDBC Connector, I understand that the correct approach
> is to promote the release of JDBC Connector and looks like we have no more
> options.
>
> Splitting a separate redshift repository does not solve this coupling
> problem, from a user perspective, redshift should also be in the AWS
> Connector repo.
>
> Best,
> Leonard
Re: [DISCUSS] FLIP-307: Flink connector Redshift
Posted by Leonard Xu <xb...@gmail.com>.
Thanks Samrat and Danny for driving this FLIP.
>> an effective approach is to utilize the latest version of flink-connector-jdbc
> as a Maven dependency
>
> When we have stable source/sink APIs and the connector versions are
> decoupled from Flink this makes sense. But right now this would mean that
> the JDBC connector will block the AWS connector for each new Flink version
> support release (1.18, 1.19, 1.20, 2.0 etc). That being said, I cannot
> think of a cleaner alternative, without pulling the core JDBC bits out into
> a dedicated project that is decoupled from and released independently of
> Flink. Splitting flink-connector-redshift into a dedicated repo would
> decouple AWS/JDBC, but obviously introduce a new connector that is blocked
> by both AWS and JDBC.
Do we have to rely on the latest version of JDBC Connector here? I understand that as long as the version of flink minor is the same as the JDBC Connector, Could you collect the APIs that Redshift generally needs to use?
Assuming that AWS Connector(Redshift) depends on JDBC Connector and wants a higher version of JDBC Connector, I understand that the correct approach is to promote the release of JDBC Connector and looks like we have no more options.
Splitting a separate redshift repository does not solve this coupling problem, from a user perspective, redshift should also be in the AWS Connector repo.
Best,
Leonard
Re: [DISCUSS] FLIP-307: Flink connector Redshift
Posted by Danny Cranmer <da...@apache.org>.
Hey Samrat,
Thanks for driving this.
> new FlinkRedshiftSink(prop)
One design nit for "new FlinkRedshiftSink(prop)" I would prefer to use
first class fields such as database/table-name/url/etc, it makes the API
cleaner and easier to use.
> an effective approach is to utilize the latest version of flink-connector-jdbc
as a Maven dependency
When we have stable source/sink APIs and the connector versions are
decoupled from Flink this makes sense. But right now this would mean that
the JDBC connector will block the AWS connector for each new Flink version
support release (1.18, 1.19, 1.20, 2.0 etc). That being said, I cannot
think of a cleaner alternative, without pulling the core JDBC bits out into
a dedicated project that is decoupled from and released independently of
Flink. Splitting flink-connector-redshift into a dedicated repo would
decouple AWS/JDBC, but obviously introduce a new connector that is blocked
by both AWS and JDBC. Martijn do you have any suggestions?
> The Flink connector for Redshift will be agnostic to the configured WLM
and utilise it for scaling in and out for the source/sink
Can you elaborate how this will work? Are you going to scale the
source/sink operator parallelism, or threads within the operator. If you
intend on scaling the operator parallelism, how will this work?
> i) Sink configurations(mandatory) > aws-iam-role
For the sink, what is the "aws-iam-role" property used for if we also
provide username and password?
> Flink connects to S3 using both the Hadoop FileSystem interfaces and
directly using the Amazon Java SDK's S3 client.
IAM auth is managed differently today for S3 filesystem and the AWS
connectors. You mentioned you will utilise the AWS base libs to help with
creds, but can you elaborate on how the user will configure these two
authentications?
> AWS credentials will automatically be retrieved through the
DefaultAWSCredentialsProviderChain.
Can we use the AWS base credential provide property format here instead?
Generally I am struggling to understand exactly what will be reused from
the flink-connector-aws repo? Maybe the AWS client versions from the parent
pom?
Thanks,
Danny
On Tue, Aug 29, 2023 at 7:06 AM Samrat Deb <de...@gmail.com> wrote:
> Hi Martijn,
> I fully comprehend your apprehensions regarding AWS's reliance on JDBC and
> the resultant impact on the release cycle. However, an effective approach
> is to utilize the latest version of flink-connector-jdbc as a Maven
> dependency. In this arrangement, flink-connector-aws can be associated with
> an older version of flink-connector-jdbc. It's not obligatory for
> flink-connector-aws to constantly align with the most current version. This
> strategy disengages direct dependencies between the two in their respective
> release cycles. By adopting this approach, flink-connector-aws can maintain
> its specific dependencies on JDBC and the preferred version paralleling its
> approach with the AWS SDK's handling of dependencies.
>
> Bests,
> Samrat
>
> On Tue, Jul 4, 2023 at 7:09 PM Martijn Visser <ma...@apache.org>
> wrote:
>
> > Hi Samrat,
> >
> > The dependency between JDBC and AWS worries me a lot: we're already
> seeing
> > that coupling causes a lot of issues down the line. Why can't we decouple
> > these?
> >
> > Best regards,
> >
> > Martijn
> >
> > On Tue, Jul 4, 2023 at 3:35 PM Samrat Deb <de...@gmail.com> wrote:
> >
> > > Hi Leonard,
> > >
> > > Sorry for the late reply.
> > >
> > > > 1 Reusing the capabilities of JDBC and Filesystem in the Redshift
> > > connector generally makes sense to me. However, since all of them are
> > > managed in different repositories and depend on Flink dependency, could
> > you
> > > explain how you establish the versioning, release, and dependency
> > > management process?
> > >
> > > We intend to maintain the current release cycle for the Flink Connector
> > > AWS. However, we will be incorporating a dependency of Flink Connector
> > JDBC
> > > to most recent version, which will synchronize between the flink
> > connector
> > > AWS and flink connector JDBC connectors releases for new Flink version
> > > support. Additionally, the Flink Connector Redshift will exclusively
> > > utilize the public API, minimizing the occurrence of immediate breaking
> > > changes.
> > >
> > > > 2 Some configuration option names can be improved to match the naming
> > > style of existing configuration options, for example:
> > > table -> table-name
> > > query -> scan.query
> > > aws-iam-role -> aws.iam-role
> > > read.mode -> scan.read.mode: similar to scan.startup.mode , and maybe
> we
> > > will have lookup.read.mode
> > > write.mode -> sink.write.mode
> > >
> > > Updated the FLIP-307 [1]
> > >
> > > > 3 The source of Redshift connector supports JDBC queries, IIUC, we
> can
> > > also support the LookupTableSource as well?
> > >
> > > we will support `LookupTableSource` in redshift connector source
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> > >
> > > Bests,
> > > Samrat
> > >
> > > On Wed, Jun 21, 2023 at 4:47 PM Leonard Xu <xb...@gmail.com> wrote:
> > >
> > > > Thanks Samrat for driving this FLIP.
> > > >
> > > > Since the community has already built a set of basic components for
> the
> > > > connector, I only have three comments.
> > > >
> > > > 1 Reusing the capabilities of JDBC and Filesystem in the Redshift
> > > > connector generally makes sense to me. However, since all of them are
> > > > managed in different repositories and depend on Flink dependency,
> could
> > > you
> > > > explain how you establish the versioning, release, and dependency
> > > > management process?
> > > >
> > > > 2 Some configuration option names can be improved to match the naming
> > > > style of existing configuration options, for example:
> > > > table -> table-name
> > > > query -> scan.query
> > > > aws-iam-role -> aws.iam-role
> > > > read.mode -> scan.read.mode: similar to scan.startup.mode , and maybe
> > we
> > > > will have lookup.read.mode
> > > > write.mode -> sink.write.mode
> > > >
> > > > 3 The source of Redshift connector supports JDBC queries, IIUC, we
> can
> > > > also support the LookupTableSource as well?
> > > >
> > > > Best,
> > > > Leonard
> > > >
> > > > > On Jun 21, 2023, at 4:57 PM, Samrat Deb <de...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi Martijn,
> > > > >
> > > > > Thank you for sharing your thoughts on the matter.
> > > > > I understand that you don't have a strong opinion on whether to
> > support
> > > > > exactly-once processing from the beginning or at a later stage.
> > > > > For initial implementation I will go ahead with at-least-once
> > > semantics.
> > > > >
> > > > >> The only consideration that I could think of is that
> > > > > if you start with at-least-once, you could consider using the ASync
> > > API,
> > > > > but I don't think the ASync API yet supports exactly-once.
> > > > >
> > > > > Noted. It's a valid consideration to start compatibility with the
> > Async
> > > > > API.
> > > > >
> > > > > Bests,
> > > > > Samrat
> > > > >
> > > > >
> > > > > On Mon, Jun 19, 2023 at 5:28 PM Martijn Visser <
> > > martijnvisser@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Samrat,
> > > > >>
> > > > >> I have no strong opinion on whether to support exactly-once from
> the
> > > > start
> > > > >> or potentially later. The only consideration that I could think of
> > is
> > > > that
> > > > >> if you start with at-least-once, you could consider using the
> ASync
> > > API,
> > > > >> but I don't think the ASync API yet supports exactly-once.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Martijn
> > > > >>
> > > > >> On Fri, Jun 9, 2023 at 7:22 PM Jing Ge <jing@ververica.com.invalid
> >
> > > > wrote:
> > > > >>
> > > > >>> Hi Samrat,
> > > > >>>
> > > > >>> The FLIP looks good, thanks!
> > > > >>>
> > > > >>> Best regards,
> > > > >>> Jing
> > > > >>>
> > > > >>>
> > > > >>> On Tue, Jun 6, 2023 at 8:16 PM Samrat Deb <decordeapex@gmail.com
> >
> > > > wrote:
> > > > >>>
> > > > >>>> Hi Jing,
> > > > >>>>
> > > > >>>>> I would suggest adding that information into the
> > > > >>>> FLIP.
> > > > >>>>
> > > > >>>> Updated now, please review the new version of flip whenever
> time.
> > > > >>>>
> > > > >>>>> +1 Looking forward to your PR :-)
> > > > >>>> I will request for your review once m ready with PR :-)
> > > > >>>>
> > > > >>>> Bests,
> > > > >>>> Samrat
> > > > >>>>
> > > > >>>> On Tue, Jun 6, 2023 at 11:43 PM Samrat Deb <
> decordeapex@gmail.com
> > >
> > > > >>> wrote:
> > > > >>>>
> > > > >>>>> Hi Martijn,
> > > > >>>>>
> > > > >>>>>> If I understand this correctly, the Redshift sink
> > > > >>>>> would not be able to support exactly-once, is that correct?
> > > > >>>>>
> > > > >>>>> As I delve deeper into the study of Redshift's capabilities, I
> > have
> > > > >>>>> discovered that it does support "merge into" operations [1] and
> > > some
> > > > >>>>> merge into examples [2].
> > > > >>>>> This opens up the possibility of implementing exactly-once
> > > semantics
> > > > >>> with
> > > > >>>>> the connector.
> > > > >>>>> However, I believe it would be prudent to start with a more
> > focused
> > > > >>> scope
> > > > >>>>> for the initial phase of implementation and defer the
> exact-once
> > > > >>> support
> > > > >>>>> for subsequent iterations.
> > > > >>>>>
> > > > >>>>> Before finalizing the approach, I would greatly appreciate your
> > > > >>> thoughts
> > > > >>>>> and suggestions on this matter.
> > > > >>>>> Should we prioritize the initial implementation without
> > > exactly-once
> > > > >>>>> support, or would you advise incorporating it right from the
> > start?
> > > > >>>>> Your insights and experiences would be immensely valuable in
> > making
> > > > >>> this
> > > > >>>>> decision.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> [1]
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://docs.aws.amazon.com/redshift/latest/dg/t_updating-inserting-using-staging-tables-.html
> > > > >>>>> [2]
> > > > >>
> https://docs.aws.amazon.com/redshift/latest/dg/merge-examples.html
> > > > >>>>>
> > > > >>>>> Bests,
> > > > >>>>> Samrat
> > > > >>>>>
> > > > >>>>> On Mon, Jun 5, 2023 at 7:09 PM Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>>> Hi Samrat,
> > > > >>>>>>
> > > > >>>>>> Thanks for the feedback. I would suggest adding that
> information
> > > > >> into
> > > > >>>> the
> > > > >>>>>> FLIP.
> > > > >>>>>>
> > > > >>>>>> +1 Looking forward to your PR :-)
> > > > >>>>>>
> > > > >>>>>> Best regards,
> > > > >>>>>> Jing
> > > > >>>>>>
> > > > >>>>>> On Sat, Jun 3, 2023 at 9:19 PM Samrat Deb <
> > decordeapex@gmail.com>
> > > > >>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hi Jing Ge,
> > > > >>>>>>>
> > > > >>>>>>>>>> Do you already have any prototype? I'd like to join the
> > > > >> reviews.
> > > > >>>>>>> The prototype is in progress. I will raise the dedicated PR
> for
> > > > >>> review
> > > > >>>>>> soon
> > > > >>>>>>> also notify in this thread as well .
> > > > >>>>>>>
> > > > >>>>>>>>>> Will the Redshift connector provide additional features
> > > > >>>>>>> beyond the mediator/wrapper of the jdbc connector?
> > > > >>>>>>>
> > > > >>>>>>> Here are the additional features that the Flink connector for
> > AWS
> > > > >>>>>> Redshift
> > > > >>>>>>> can provide on top of using JDBC:
> > > > >>>>>>>
> > > > >>>>>>> 1. Integration with AWS Redshift Workload Management (WLM):
> AWS
> > > > >>>> Redshift
> > > > >>>>>>> allows you to configure WLM[1] to manage query prioritization
> > and
> > > > >>>>>> resource
> > > > >>>>>>> allocation. The Flink connector for Redshift will be agnostic
> > to
> > > > >> the
> > > > >>>>>>> configured WLM and utilize it for scaling in and out for the
> > > sink.
> > > > >>>> This
> > > > >>>>>>> means that the connector can leverage the WLM capabilities of
> > > > >>> Redshift
> > > > >>>>>> to
> > > > >>>>>>> optimize the execution of queries and allocate resources
> > > > >> efficiently
> > > > >>>>>> based
> > > > >>>>>>> on your defined workload priorities.
> > > > >>>>>>>
> > > > >>>>>>> 2. Abstraction of AWS Redshift Quotas and Limits: AWS
> Redshift
> > > > >>> imposes
> > > > >>>>>>> certain quotas and limits[2] on various aspects such as the
> > > number
> > > > >>> of
> > > > >>>>>>> clusters, concurrent connections, queries per second, etc.
> The
> > > > >> Flink
> > > > >>>>>>> connector for Redshift will provide an abstraction layer for
> > > > >> users,
> > > > >>>>>>> allowing them to work with Redshift without having to worry
> > about
> > > > >>>> these
> > > > >>>>>>> specific limits. The connector will handle the management of
> > > > >>>> connections
> > > > >>>>>>> and queries within the defined quotas and limits, abstracting
> > > away
> > > > >>> the
> > > > >>>>>>> complexity and ensuring compliance with Redshift's
> > restrictions.
> > > > >>>>>>>
> > > > >>>>>>> These features aim to simplify the integration of Flink with
> > AWS
> > > > >>>>>> Redshift,
> > > > >>>>>>> providing optimized resource utilization and transparent
> > handling
> > > > >> of
> > > > >>>>>>> Redshift-specific limitations.
> > > > >>>>>>>
> > > > >>>>>>> Bests,
> > > > >>>>>>> Samrat
> > > > >>>>>>>
> > > > >>>>>>> [1]
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://docs.aws.amazon.com/redshift/latest/dg/cm-c-implementing-workload-management.html
> > > > >>>>>>> [2]
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://docs.aws.amazon.com/redshift/latest/mgmt/amazon-redshift-limits.html
> > > > >>>>>>>
> > > > >>>>>>> On Sat, Jun 3, 2023 at 11:40 PM Samrat Deb <
> > > decordeapex@gmail.com
> > > > >>>
> > > > >>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Hi Ahmed,
> > > > >>>>>>>>
> > > > >>>>>>>>>>> please let me know If you need any collaboration
> regarding
> > > > >>>>>>> integration
> > > > >>>>>>>> with
> > > > >>>>>>>> AWS connectors credential providers or regarding FLIP-171 I
> > > > >> would
> > > > >>> be
> > > > >>>>>> more
> > > > >>>>>>>> than happy to assist.
> > > > >>>>>>>>
> > > > >>>>>>>> Sure, I will reach out incase of any hands required.
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> On Fri, Jun 2, 2023 at 6:12 PM Jing Ge
> > > > >> <jing@ververica.com.invalid
> > > > >>>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Hi Samrat,
> > > > >>>>>>>>>
> > > > >>>>>>>>> Excited to see your proposal. Supporting data warehouses is
> > one
> > > > >>> of
> > > > >>>>>> the
> > > > >>>>>>>>> major tracks for Flink. Thanks for driving it! Happy to see
> > > > >> that
> > > > >>> we
> > > > >>>>>>>>> reached
> > > > >>>>>>>>> consensus to prioritize the Sink over Source in the
> previous
> > > > >>>>>> discussion.
> > > > >>>>>>>>> Do
> > > > >>>>>>>>> you already have any prototype? I'd like to join the
> reviews.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Just out of curiosity, speaking of JDBC mode, according to
> > the
> > > > >>>> FLIP,
> > > > >>>>>> it
> > > > >>>>>>>>> should be doable to directly use the jdbc connector with
> > > > >>> Redshift,
> > > > >>>>>> if I
> > > > >>>>>>> am
> > > > >>>>>>>>> not mistaken. Will the Redshift connector provide
> additional
> > > > >>>> features
> > > > >>>>>>>>> beyond the mediator/wrapper of the jdbc connector?
> > > > >>>>>>>>>
> > > > >>>>>>>>> Best regards,
> > > > >>>>>>>>> Jing
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Jun 1, 2023 at 8:22 PM Ahmed Hamdy <
> > > > >> hamdy10024@gmail.com
> > > > >>>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Hi Samrat
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thanks for putting up this FLIP. I agree regarding the
> > > > >>> importance
> > > > >>>>>> of
> > > > >>>>>>> the
> > > > >>>>>>>>>> use case.
> > > > >>>>>>>>>> please let me know If you need any collaboration regarding
> > > > >>>>>> integration
> > > > >>>>>>>>> with
> > > > >>>>>>>>>> AWS connectors credential providers or regarding FLIP-171
> I
> > > > >>> would
> > > > >>>>>> be
> > > > >>>>>>>>> more
> > > > >>>>>>>>>> than happy to assist.
> > > > >>>>>>>>>> I also like Leonard's proposal for starting with
> > > > >> DataStreamSink
> > > > >>>> and
> > > > >>>>>>>>>> TableSink, It would be great to have some milestones
> > > > >> delivered
> > > > >>> as
> > > > >>>>>> soon
> > > > >>>>>>>>> as
> > > > >>>>>>>>>> ready.
> > > > >>>>>>>>>> best regards
> > > > >>>>>>>>>> Ahmed Hamdy
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Wed, 31 May 2023 at 11:15, Samrat Deb <
> > > > >>> decordeapex@gmail.com>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Hi Liu Ron,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> 1. Regarding the `read.mode` and `write.mode`, you say
> > > > >>> here
> > > > >>>>>>>>> provides
> > > > >>>>>>>>>> two
> > > > >>>>>>>>>>> modes, respectively, jdbc and `unload or copy`, What is
> the
> > > > >>>>>> default
> > > > >>>>>>>>> value
> > > > >>>>>>>>>>> for `read.mode` and `write.mode?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I have made an effort to make the configuration options
> > > > >>>>>> `read.mode`
> > > > >>>>>>>>> and
> > > > >>>>>>>>>>> `write.mode` mandatory for the "flink-connector-redshift"
> > > > >>>>>> according
> > > > >>>>>>> to
> > > > >>>>>>>>>>> FLIP[1]. The rationale behind this decision is to empower
> > > > >>> users
> > > > >>>>>> who
> > > > >>>>>>>>> are
> > > > >>>>>>>>>>> familiar with their Redshift setup and have specific
> > > > >>>> expectations
> > > > >>>>>>> for
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>> sink. By making these configurations mandatory, users can
> > > > >>> have
> > > > >>>>>> more
> > > > >>>>>>>>>> control
> > > > >>>>>>>>>>> and flexibility in configuring the connector to meet
> their
> > > > >>>>>>>>> requirements.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> However, I am open to receiving feedback on whether it
> > > > >> would
> > > > >>> be
> > > > >>>>>>>>>> beneficial
> > > > >>>>>>>>>>> to make the configuration options non-mandatory and set
> > > > >>> default
> > > > >>>>>>> values
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>> them. If you believe there are advantages to having
> default
> > > > >>>>>> values
> > > > >>>>>>> or
> > > > >>>>>>>>> any
> > > > >>>>>>>>>>> other suggestions, please share your thoughts. Your
> > > > >> feedback
> > > > >>> is
> > > > >>>>>>> highly
> > > > >>>>>>>>>>> appreciated.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> 2. For Source, does it both support batch read and
> > > > >>> streaming
> > > > >>>>>>> read?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Redshift currently does not provide native support for
> > > > >>>> streaming
> > > > >>>>>>>>> reads,
> > > > >>>>>>>>>>> although it does support streaming writes[2]. As part of
> > > > >> the
> > > > >>>>>> plan, I
> > > > >>>>>>>>>> intend
> > > > >>>>>>>>>>> to conduct a proof of concept and benchmarking to explore
> > > > >> the
> > > > >>>>>>>>>> possibilities
> > > > >>>>>>>>>>> of implementing streaming reads using the Flink JDBC
> > > > >>> connector,
> > > > >>>>>> as
> > > > >>>>>>>>>> Redshift
> > > > >>>>>>>>>>> is JDBC compatible.
> > > > >>>>>>>>>>> However, it is important to note that, in the initial
> phase
> > > > >>> of
> > > > >>>>>>>>>>> implementation, the focus will primarily be on supporting
> > > > >>> batch
> > > > >>>>>>> reads
> > > > >>>>>>>>>>> rather than streaming reads. This approach will allow us
> to
> > > > >>>>>> deliver
> > > > >>>>>>> a
> > > > >>>>>>>>>>> robust and reliable solution for batch processing in
> phase
> > > > >> 2
> > > > >>> of
> > > > >>>>>> the
> > > > >>>>>>>>>>> implementation.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> [1]
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> > > > >>>>>>>>>>> [2]
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion.html
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Bests,
> > > > >>>>>>>>>>> Samrat
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Wed, May 31, 2023 at 8:03 AM liu ron <
> > > > >> ron9.liu@gmail.com>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi, Samrat
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Thanks for driving this FLIP. It looks like supporting
> > > > >>>>>>>>>>>> flink-connector-redshift is very useful to Flink. I have
> > > > >>> two
> > > > >>>>>>>>> question:
> > > > >>>>>>>>>>>> 1. Regarding the `read.mode` and `write.mode`, you say
> > > > >>> here
> > > > >>>>>>>>> provides
> > > > >>>>>>>>>> two
> > > > >>>>>>>>>>>> modes, respectively, jdbc and `unload or copy`, What is
> > > > >> the
> > > > >>>>>>> default
> > > > >>>>>>>>>> value
> > > > >>>>>>>>>>>> for `read.mode` and `write.mode?
> > > > >>>>>>>>>>>> 2. For Source, does it both support batch read and
> > > > >>> streaming
> > > > >>>>>> read?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>> Ron
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Samrat Deb <de...@gmail.com> 于2023年5月30日周二
> > > > >> 17:15写道：
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> [1]
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> [note] Missed the trailing link for previous mail
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Tue, May 30, 2023 at 2:43 PM Samrat Deb <
> > > > >>>>>>> decordeapex@gmail.com
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Hi Leonard,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> and I’m glad to help review the design as well as
> > > > >> the
> > > > >>>>>> code
> > > > >>>>>>>>>> review.
> > > > >>>>>>>>>>>>>> Thank you so much. It would be really great and
> > > > >> helpful
> > > > >>>> to
> > > > >>>>>>> bring
> > > > >>>>>>>>>>>>>> flink-connector-redshift for flink users :) .
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> I have divided the implementation in 3 phases in the
> > > > >>>>>> `Scope`
> > > > >>>>>>>>>>>> Section[1].
> > > > >>>>>>>>>>>>>> 1st phase is to
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> - Integrate with Flink Sink API (*FLIP-171*
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> )
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> About the implementation phases, How about
> > > > >>> prioritizing
> > > > >>>>>>>>> support
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> Datastream Sink API and TableSink API in the first
> > > > >>> phase?
> > > > >>>>>>>>>>>>>> I can completely resonate with you to prioritize
> > > > >>> support
> > > > >>>>>> for
> > > > >>>>>>>>>>> Datastream
> > > > >>>>>>>>>>>>>> Sink API and TableSink API in the first phase.
> > > > >>>>>>>>>>>>>> I will update the FLIP[1] as you have suggested.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> It seems that the primary use cases for the
> > > > >> Redshift
> > > > >>>>>>> connector
> > > > >>>>>>>>>> are
> > > > >>>>>>>>>>>>>> acting as a sink for processed data by Flink.
> > > > >>>>>>>>>>>>>> Yes, majority ask and requirement for Redshift
> > > > >>> connector
> > > > >>>> is
> > > > >>>>>>> sink
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>>>>> processed data by Flink.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Bests,
> > > > >>>>>>>>>>>>>> Samrat
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Tue, May 30, 2023 at 12:35 PM Leonard Xu <
> > > > >>>>>>> xbjtdcq@gmail.com>
> > > > >>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks @Samrat for bringing this discussion.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> It makes sense to me to introduce AWS Redshift
> > > > >>> connector
> > > > >>>>>> for
> > > > >>>>>>>>>> Apache
> > > > >>>>>>>>>>>>>>> Flink, and I’m glad to help review the design as
> > > > >> well
> > > > >>> as
> > > > >>>>>> the
> > > > >>>>>>>>> code
> > > > >>>>>>>>>>>>> review.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> About the implementation phases, How about
> > > > >>> prioritizing
> > > > >>>>>>> support
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> Datastream Sink API and TableSink API in the first
> > > > >>>> phase?
> > > > >>>>>> It
> > > > >>>>>>>>> seems
> > > > >>>>>>>>>>>> that
> > > > >>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> primary use cases for the Redshift connector are
> > > > >>> acting
> > > > >>>>>> as a
> > > > >>>>>>>>> sink
> > > > >>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>> processed data by Flink.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>> Leonard
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On May 29, 2023, at 12:51 PM, Samrat Deb <
> > > > >>>>>>>>> decordeapex@gmail.com
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Hello all ,
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Context:
> > > > >>>>>>>>>>>>>>>> Amazon Redshift [1] is a fully managed,
> > > > >>> petabyte-scale
> > > > >>>>>> data
> > > > >>>>>>>>>>>> warehouse
> > > > >>>>>>>>>>>>>>>> service in the cloud. It allows analyzing data
> > > > >>> without
> > > > >>>>>> all
> > > > >>>>>>> of
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> configurations of a provisioned data warehouse.
> > > > >>>>>> Resources
> > > > >>>>>>> are
> > > > >>>>>>>>>>>>>>> automatically
> > > > >>>>>>>>>>>>>>>> provisioned and data warehouse capacity is
> > > > >>>> intelligently
> > > > >>>>>>>>> scaled
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>>>>>> deliver
> > > > >>>>>>>>>>>>>>>> fast performance for even the most demanding and
> > > > >>>>>>>>> unpredictable
> > > > >>>>>>>>>>>>>>> workloads.
> > > > >>>>>>>>>>>>>>>> Redshift is one of the widely used warehouse
> > > > >>> solutions
> > > > >>>>>> in
> > > > >>>>>>> the
> > > > >>>>>>>>>>>> current
> > > > >>>>>>>>>>>>>>>> market.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Building flink connector redshift will allow flink
> > > > >>>>>> users to
> > > > >>>>>>>>> have
> > > > >>>>>>>>>>>>> source
> > > > >>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>> sink directly to redshift. It will help flink to
> > > > >>>> expand
> > > > >>>>>> the
> > > > >>>>>>>>>> scope
> > > > >>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>> redshift as a new connector in the ecosystem.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> I would like to start a discussion on the
> > > > >> FLIP-307:
> > > > >>>>>> Flink
> > > > >>>>>>>>>>> connector
> > > > >>>>>>>>>>>>>>>> redshift [2].
> > > > >>>>>>>>>>>>>>>> Looking forward to comments, feedbacks and
> > > > >>> suggestions
> > > > >>>>>> from
> > > > >>>>>>>>> the
> > > > >>>>>>>>>>>>>>> community
> > > > >>>>>>>>>>>>>>>> on the proposal.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> [1]
> > > > >>>>>>>>>>
> > > > >> https://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html
> > > > >>>>>>>>>>>>>>>> [2]
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Bests,
> > > > >>>>>>>>>>>>>>>> Samrat
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>