You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Soheil Pourbafrani <so...@gmail.com> on 2017/12/02 21:28:47 UTC
using regular expression to specify Kafka topics
I use Flink Kafka connector 10 to subscribe topics and get data. Now I want
to specify topics not using String, but regular expression. I want to do
that just because it can recognize future topics added to the Kafka and get
their data.
Spark Kafka connector has a method named SubscribePattern and we can use it
to subscribe to topics using regular expressions.
Is there something equivalent or some other way to do that?
Re: using regular expression to specify Kafka topics
Posted by Aljoscha Krettek <al...@apache.org>.
+1 for including in the next RC.
> On 4. Dec 2017, at 13:10, Gyula Fóra <gy...@gmail.com> wrote:
>
> Hi,
>
> I think what we need to ask here is whether the custom workaround solutions people are using over the place are any more robust than our own implementation.
>
> I would be in favor of including it if the PR is merged in time for the next RC.
>
> Gyula
>
> Tzu-Li (Gordon) Tai <tzulitai@apache.org <ma...@apache.org>> ezt írta (időpont: 2017. dec. 4., H, 12:36):
> Hi,
>
> I’ve created a PR to publicly expose the feature: https://github.com/apache/flink/pull/5117 <https://github.com/apache/flink/pull/5117>.
> Whether or not we should include this in the next release candidate for 1.4 is still up for discussion.
>
> Best,
> Gordon
> On 4 December 2017 at 3:02:29 PM, Tzu-Li (Gordon) Tai (tzulitai@apache.org <ma...@apache.org>) wrote:
>
>> Hi Soheil,
>>
>> That feature is actually already internally available.
>> The only issue is that the functionality is not yet exposed via any public APIs on the Kafka consumer.
>> Please see this JIRA here: https://issues.apache.org/jira/browse/FLINK-8190 <https://issues.apache.org/jira/browse/FLINK-8190>.
>>
>> I’m not sure of exposing the pattern-based subscription API now for 1.4, though.
>> Although it is only a matter of exposing already implemented features, AFAIK pattern-based subscription is not thoroughly tested in the release candidates.
>> cc @Aljoscha, what do you think about this?
>>
>> Best,
>> Gordon
>>
>> On 3 December 2017 at 5:29:32 AM, Soheil Pourbafrani (soheil.ir08@gmail.com <ma...@gmail.com>) wrote:
>>
>>> I use Flink Kafka connector 10 to subscribe topics and get data. Now I want to specify topics not using String, but regular expression. I want to do that just because it can recognize future topics added to the Kafka and get their data.
>>>
>>> Spark Kafka connector has a method named SubscribePattern and we can use it to subscribe to topics using regular expressions.
>>>
>>> Is there something equivalent or some other way to do that?
Re: using regular expression to specify Kafka topics
Posted by Gyula Fóra <gy...@gmail.com>.
Hi,
I think what we need to ask here is whether the custom workaround solutions
people are using over the place are any more robust than our own
implementation.
I would be in favor of including it if the PR is merged in time for the
next RC.
Gyula
Tzu-Li (Gordon) Tai <tz...@apache.org> ezt írta (időpont: 2017. dec. 4.,
H, 12:36):
> Hi,
>
> I’ve created a PR to publicly expose the feature:
> https://github.com/apache/flink/pull/5117.
> Whether or not we should include this in the next release candidate for
> 1.4 is still up for discussion.
>
> Best,
> Gordon
>
> On 4 December 2017 at 3:02:29 PM, Tzu-Li (Gordon) Tai (tzulitai@apache.org)
> wrote:
>
> Hi Soheil,
>
> That feature is actually already internally available.
> The only issue is that the functionality is not yet exposed via any public
> APIs on the Kafka consumer.
> Please see this JIRA here:
> https://issues.apache.org/jira/browse/FLINK-8190.
>
> I’m not sure of exposing the pattern-based subscription API now for 1.4,
> though.
> Although it is only a matter of exposing already implemented features,
> AFAIK pattern-based subscription is not thoroughly tested in the release
> candidates.
> cc @Aljoscha, what do you think about this?
>
> Best,
> Gordon
>
> On 3 December 2017 at 5:29:32 AM, Soheil Pourbafrani (
> soheil.ir08@gmail.com) wrote:
>
> I use Flink Kafka connector 10 to subscribe topics and get data. Now I
> want to specify topics not using String, but regular expression. I want to
> do that just because it can recognize future topics added to the Kafka and
> get their data.
>
> Spark Kafka connector has a method named SubscribePattern and we can use
> it to subscribe to topics using regular expressions.
>
> Is there something equivalent or some other way to do that?
>
>
Re: using regular expression to specify Kafka topics
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi,
I’ve created a PR to publicly expose the feature: https://github.com/apache/flink/pull/5117.
Whether or not we should include this in the next release candidate for 1.4 is still up for discussion.
Best,
Gordon
On 4 December 2017 at 3:02:29 PM, Tzu-Li (Gordon) Tai (tzulitai@apache.org) wrote:
Hi Soheil,
That feature is actually already internally available.
The only issue is that the functionality is not yet exposed via any public APIs on the Kafka consumer.
Please see this JIRA here: https://issues.apache.org/jira/browse/FLINK-8190.
I’m not sure of exposing the pattern-based subscription API now for 1.4, though.
Although it is only a matter of exposing already implemented features, AFAIK pattern-based subscription is not thoroughly tested in the release candidates.
cc @Aljoscha, what do you think about this?
Best,
Gordon
On 3 December 2017 at 5:29:32 AM, Soheil Pourbafrani (soheil.ir08@gmail.com) wrote:
I use Flink Kafka connector 10 to subscribe topics and get data. Now I want to specify topics not using String, but regular expression. I want to do that just because it can recognize future topics added to the Kafka and get their data.
Spark Kafka connector has a method named SubscribePattern and we can use it to subscribe to topics using regular expressions.
Is there something equivalent or some other way to do that?
Re: using regular expression to specify Kafka topics
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Soheil,
That feature is actually already internally available.
The only issue is that the functionality is not yet exposed via any public APIs on the Kafka consumer.
Please see this JIRA here: https://issues.apache.org/jira/browse/FLINK-8190.
I’m not sure of exposing the pattern-based subscription API now for 1.4, though.
Although it is only a matter of exposing already implemented features, AFAIK pattern-based subscription is not thoroughly tested in the release candidates.
cc @Aljoscha, what do you think about this?
Best,
Gordon
On 3 December 2017 at 5:29:32 AM, Soheil Pourbafrani (soheil.ir08@gmail.com) wrote:
I use Flink Kafka connector 10 to subscribe topics and get data. Now I want to specify topics not using String, but regular expression. I want to do that just because it can recognize future topics added to the Kafka and get their data.
Spark Kafka connector has a method named SubscribePattern and we can use it to subscribe to topics using regular expressions.
Is there something equivalent or some other way to do that?