Re: Maven plugin to detect issues early on

[Jing Ge via user <us...@flink.apache.org>]

cc user ML to get more attention, since the plugin will be used by Flink
application developers.

Best regards,
Jing

On Mon, May 22, 2023 at 3:32 PM Jing Ge <ji...@ververica.com> wrote:

> Hi Emre,
>
> Thanks for clarifying it. Afaiac, it is a quite interesting proposal,
> especially for Flink job developers who are heavily using the Datastream
> API. Publishing the plugin in your Github would be a feasible way for the
> first move. As I mentioned previously, in order to help the community
> understand the plugin, you might want to describe all those attractive
> features you mentioned in e.g. the readme.md with more technical details. I
> personally was wondering how those connector compatibility rules will be
> defined and maintained, given that almost all connectors have been
> externalized.
>
> Best regards,
> Jing
>
> On Mon, May 22, 2023 at 11:24 AM Kartoglu, Emre
> <ka...@amazon.co.uk.invalid> wrote:
>
>> Hi Jing,
>>
>> The proposed plugin would be used by Flink application developers, when
>> they are writing their Flink job. It would trigger during
>> compilation/packaging and would look for known incompatibilities, bad
>> practices, or bugs.
>> For instance one cause of frustration for our customers is connector
>> incompatibilities (specifically Kafka and Kinesis) with certain Flink
>> versions. This plugin would be a quick way to update a list of known
>> incompatibilities, bugs, bad practices, so customers get errors during
>> compilation/packaging and not after they've deployed their Flink job.
>>
>> From what you're saying, the FLIP route might not be the best way to go.
>> We might publish this plugin in our own GitHub namespace/group first, and
>> then get community acknowledgement/support for it. I believe working with
>> the Flink community on this is key as we'd need their support/opinion to do
>> this the right way and reach more Flink users.
>>
>> Thanks
>> Emre
>>
>> On 21/05/2023, 16:48, "Jing Ge" <jing@ververica.com.INVA <mailto:
>> jing@ververica.com.INVA>LID> wrote:
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>>
>>
>>
>> Hi Emre,
>>
>>
>> Thanks for your proposal. It looks very interesting! Please pay attention
>> that most connectors have been externalized. Will your proposed plug be
>> used for building Flink Connectors or Flink itself? Furthermore, it would
>> be great if you could elaborate features wrt best practices so that we
>> could understand how the plugin will help us.
>>
>>
>> Afaik, FLIP is recommended for improvement ideas that will change public
>> APIs. I am not sure if a new maven plugin belongs to it.
>>
>>
>> Best regards,
>> Jing
>>
>>
>> On Tue, May 16, 2023 at 11:29 AM Kartoglu, Emre <kartoi@amazon.co.uk.inva
>> <ma...@amazon.co.uk.inva>lid>
>> wrote:
>>
>>
>> > Hello all,
>> >
>> > Myself and 2 colleagues developed a Maven plugin (no support for Gradle
>> or
>> > other build tools yet) that we use internally to detect potential
>> issues in
>> > Flink apps at compilation/packaging stage:
>> >
>> >
>> > * Known connector version incompatibilities – so far covering Kafka
>> > and Kinesis
>> > * Best practices e.g. setting operator IDs
>> >
>> > We’d like to make this open-source. Ideally with the Flink community’s
>> > support/mention of it on the Flink website, so more people use it.
>> >
>> > Going forward, I believe we have at least the following options:
>> >
>> > * Get community support: Create a FLIP to discuss where the plugin
>> > should live, what kind of problems it should detect etc.
>> > * We still open-source it but without the community support (if the
>> > community has objections to officially supporting it for instance).
>> >
>> > Just wanted to gauge the feeling/thoughts towards this tool from the
>> > community before going ahead.
>> >
>> > Thanks,
>> > Emre
>> >
>> >
>>
>>
>>
>>