You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by 陈卓宇 <mo...@apache.org> on 2023/04/07 09:53:16 UTC
Discuss whether the `partition` parameter of the kafka sink linker can be removed.
A user recently gave feedback about a kafkaSink exception that could not be
written [https://github.com/apache/incubator-seatunnel/issues/4498], and I
set out to track down the problem.
By reading the source code, I found that the error was reported because the
`parameter` parameter filled in by the user was not in the range of the
real partition number of the specified topic, but the program did not
report an explicit exception.
Later, I communicated with the user what the intent of configuring this
parameter was, and he said: he copied the official website example. I then
felt that there is no scenario to send data to the specified partition, why
not remove this parameter Remove this function
Re: Discuss whether the `partition` parameter of the kafka sink linker can be removed.
Posted by hailin0 <wa...@apache.org>.
+1
陈卓宇 <mo...@apache.org> 于2023年4月7日周五 17:53写道：
> A user recently gave feedback about a kafkaSink exception that could not be
> written [https://github.com/apache/incubator-seatunnel/issues/4498], and I
> set out to track down the problem.
> By reading the source code, I found that the error was reported because the
> `parameter` parameter filled in by the user was not in the range of the
> real partition number of the specified topic, but the program did not
> report an explicit exception.
> Later, I communicated with the user what the intent of configuring this
> parameter was, and he said: he copied the official website example. I then
> felt that there is no scenario to send data to the specified partition, why
> not remove this parameter Remove this function
>
Re: Discuss whether the `partition` parameter of the kafka sink linker can be removed.
Posted by 陈卓宇 <mo...@apache.org>.
Supplementary：https://github.com/apache/incubator-seatunnel/pull/4513
陈卓宇 <mo...@apache.org> 于2023年4月7日周五 19:00写道：
> A scenario comes to mind: to ensure the orderliness of the messages, the
> messages are typed under a specified partition
>
> 陈卓宇 <mo...@apache.org> 于2023年4月7日周五 17:53写道：
>
>> A user recently gave feedback about a kafkaSink exception that could not
>> be written [https://github.com/apache/incubator-seatunnel/issues/4498],
>> and I set out to track down the problem.
>> By reading the source code, I found that the error was reported because
>> the `parameter` parameter filled in by the user was not in the range of the
>> real partition number of the specified topic, but the program did not
>> report an explicit exception.
>> Later, I communicated with the user what the intent of configuring this
>> parameter was, and he said: he copied the official website example. I then
>> felt that there is no scenario to send data to the specified partition, why
>> not remove this parameter Remove this function
>>
>
Re: Discuss whether the `partition` parameter of the kafka sink linker can be removed.
Posted by JUN GAO <ga...@apache.org>.
I'm not sure. After deleting the partition parameter, how do users specify
which specific partition to send data to？
JUN GAO <ga...@apache.org> 于2023年4月10日周一 15:30写道：
> +1
>
> hailin0 <wa...@apache.org> 于2023年4月8日周六 22:47写道：
>
>> CDC usually only needs to send the data of the same table or the same
>> primary key to the same partition, but does not care which partition.
>>
>> It can be done by partition_key_fields
>>
>> 陈卓宇 <mo...@apache.org> 于2023年4月7日周五 19:01写道：
>>
>> > A scenario comes to mind: to ensure the orderliness of the messages, the
>> > messages are typed under a specified partition
>> >
>> > 陈卓宇 <mo...@apache.org> 于2023年4月7日周五 17:53写道：
>> >
>> > > A user recently gave feedback about a kafkaSink exception that could
>> not
>> > > be written [https://github.com/apache/incubator-seatunnel/issues/4498
>> ],
>> > > and I set out to track down the problem.
>> > > By reading the source code, I found that the error was reported
>> because
>> > > the `parameter` parameter filled in by the user was not in the range
>> of
>> > the
>> > > real partition number of the specified topic, but the program did not
>> > > report an explicit exception.
>> > > Later, I communicated with the user what the intent of configuring
>> this
>> > > parameter was, and he said: he copied the official website example. I
>> > then
>> > > felt that there is no scenario to send data to the specified
>> partition,
>> > why
>> > > not remove this parameter Remove this function
>> > >
>> >
>>
>
>
> --
>
> Best Regards
>
> ------------
>
> EricJoy2048
> gaojun2048@gmail.com
>
>
--
Best Regards
------------
EricJoy2048
gaojun2048@gmail.com
Re: Discuss whether the `partition` parameter of the kafka sink linker can be removed.
Posted by JUN GAO <ga...@apache.org>.
+1
hailin0 <wa...@apache.org> 于2023年4月8日周六 22:47写道：
> CDC usually only needs to send the data of the same table or the same
> primary key to the same partition, but does not care which partition.
>
> It can be done by partition_key_fields
>
> 陈卓宇 <mo...@apache.org> 于2023年4月7日周五 19:01写道：
>
> > A scenario comes to mind: to ensure the orderliness of the messages, the
> > messages are typed under a specified partition
> >
> > 陈卓宇 <mo...@apache.org> 于2023年4月7日周五 17:53写道：
> >
> > > A user recently gave feedback about a kafkaSink exception that could
> not
> > > be written [https://github.com/apache/incubator-seatunnel/issues/4498
> ],
> > > and I set out to track down the problem.
> > > By reading the source code, I found that the error was reported because
> > > the `parameter` parameter filled in by the user was not in the range of
> > the
> > > real partition number of the specified topic, but the program did not
> > > report an explicit exception.
> > > Later, I communicated with the user what the intent of configuring this
> > > parameter was, and he said: he copied the official website example. I
> > then
> > > felt that there is no scenario to send data to the specified partition,
> > why
> > > not remove this parameter Remove this function
> > >
> >
>
--
Best Regards
------------
EricJoy2048
gaojun2048@gmail.com
Re: Discuss whether the `partition` parameter of the kafka sink linker can be removed.
Posted by hailin0 <wa...@apache.org>.
CDC usually only needs to send the data of the same table or the same
primary key to the same partition, but does not care which partition.
It can be done by partition_key_fields
陈卓宇 <mo...@apache.org> 于2023年4月7日周五 19:01写道：
> A scenario comes to mind: to ensure the orderliness of the messages, the
> messages are typed under a specified partition
>
> 陈卓宇 <mo...@apache.org> 于2023年4月7日周五 17:53写道：
>
> > A user recently gave feedback about a kafkaSink exception that could not
> > be written [https://github.com/apache/incubator-seatunnel/issues/4498],
> > and I set out to track down the problem.
> > By reading the source code, I found that the error was reported because
> > the `parameter` parameter filled in by the user was not in the range of
> the
> > real partition number of the specified topic, but the program did not
> > report an explicit exception.
> > Later, I communicated with the user what the intent of configuring this
> > parameter was, and he said: he copied the official website example. I
> then
> > felt that there is no scenario to send data to the specified partition,
> why
> > not remove this parameter Remove this function
> >
>
Re: Discuss whether the `partition` parameter of the kafka sink linker can be removed.
Posted by 陈卓宇 <mo...@apache.org>.
A scenario comes to mind: to ensure the orderliness of the messages, the
messages are typed under a specified partition
陈卓宇 <mo...@apache.org> 于2023年4月7日周五 17:53写道：
> A user recently gave feedback about a kafkaSink exception that could not
> be written [https://github.com/apache/incubator-seatunnel/issues/4498],
> and I set out to track down the problem.
> By reading the source code, I found that the error was reported because
> the `parameter` parameter filled in by the user was not in the range of the
> real partition number of the specified topic, but the program did not
> report an explicit exception.
> Later, I communicated with the user what the intent of configuring this
> parameter was, and he said: he copied the official website example. I then
> felt that there is no scenario to send data to the specified partition, why
> not remove this parameter Remove this function
>