You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Srikanth <sr...@gmail.com> on 2016/11/16 19:20:57 UTC
Kafka Streams internal topic naming
Hello,
Does kafka stream provide an API to control how internal topics are named?
Right now it uses appId, operator name, etc.
In a shared kafka cluster its common to have naming convention that may
require some prefix/suffix.
Srikanth
Re: Kafka Streams internal topic naming
Posted by Srikanth <sr...@gmail.com>.
Thanks Matthias/Michael/Guozhang!
Using app id may help to some extent. Will have to think & test this
through.
Good to know there will be more direct support for this in the future. May
be it will play well with KIP-37.
Srikanth
On Fri, Nov 18, 2016 at 1:12 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Srikanth,
>
> Are you checking to see if you can manually set the internal topic names to
> follow your own naming convention in your shared cluster? For that the
> current answer is no, as Streams are trying to abstract users from worrying
> about them since they are treated as "internals" anyways. But I think it is
> a faire request moving forward for multi-tenant clusters, and also for
> allowing different Streams apps to have authorizations to create topics
> with a different naming patterns, etc.
>
> Guozhang
>
>
> On Fri, Nov 18, 2016 at 1:32 AM, Michael Noll <mi...@confluent.io>
> wrote:
>
> > Srikanth,
> >
> > as Matthias said, you can achieve some namespacing effects through the
> use
> > of (your own in-house) conventions of defining `application.id` across
> > teams. The id is used as the prefix for topics, see
> > http://docs.confluent.io/current/streams/developer-guide.html#required-
> > configuration-parameters
> > for further details.
> >
> > Best,
> > Michael
> >
> >
> >
> > On Thu, Nov 17, 2016 at 6:30 PM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> > > The only way to influence the naming is via application.id which you
> can
> > > set as you wish. Hope this is good enough to meet your naming
> > conventions.
> > >
> > > As Michael mentioned, there is no way to manually specify internal
> topic
> > > names right now.
> > >
> > > -Matthias
> > >
> > > On 11/17/16 8:45 AM, Srikanth wrote:
> > > > That is right Michael. Most teams that use kafka library can adhere
> to
> > > > certain naming convention.
> > > > Using streams API will break that.
> > > >
> > > > Srikanth
> > > >
> > > > On Wed, Nov 16, 2016 at 2:32 PM, Michael Noll <mi...@confluent.io>
> > > wrote:
> > > >
> > > >> Srikanth,
> > > >>
> > > >> no, there's isn't any API to control the naming of internal topics.
> > > >>
> > > >> Is the reason you're asking for such functionality only/mostly about
> > > >> multi-tenancy issues (as you mentioned in your first message)?
> > > >>
> > > >> -Michael
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Nov 16, 2016 at 8:20 PM, Srikanth <sr...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hello,
> > > >>>
> > > >>> Does kafka stream provide an API to control how internal topics are
> > > >> named?
> > > >>> Right now it uses appId, operator name, etc.
> > > >>> In a shared kafka cluster its common to have naming convention that
> > may
> > > >>> require some prefix/suffix.
> > > >>>
> > > >>> Srikanth
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
> >
> > --
> > *Michael G. Noll*
> > Product Manager | Confluent
> > +1 650 453 5860 | @miguno <https://twitter.com/miguno>
> > Follow us: Twitter <https://twitter.com/ConfluentInc> | Blog
> > <http://www.confluent.io/blog>
> >
>
>
>
> --
> -- Guozhang
>
Re: Kafka Streams internal topic naming
Posted by Guozhang Wang <wa...@gmail.com>.
Srikanth,
Are you checking to see if you can manually set the internal topic names to
follow your own naming convention in your shared cluster? For that the
current answer is no, as Streams are trying to abstract users from worrying
about them since they are treated as "internals" anyways. But I think it is
a faire request moving forward for multi-tenant clusters, and also for
allowing different Streams apps to have authorizations to create topics
with a different naming patterns, etc.
Guozhang
On Fri, Nov 18, 2016 at 1:32 AM, Michael Noll <mi...@confluent.io> wrote:
> Srikanth,
>
> as Matthias said, you can achieve some namespacing effects through the use
> of (your own in-house) conventions of defining `application.id` across
> teams. The id is used as the prefix for topics, see
> http://docs.confluent.io/current/streams/developer-guide.html#required-
> configuration-parameters
> for further details.
>
> Best,
> Michael
>
>
>
> On Thu, Nov 17, 2016 at 6:30 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > The only way to influence the naming is via application.id which you can
> > set as you wish. Hope this is good enough to meet your naming
> conventions.
> >
> > As Michael mentioned, there is no way to manually specify internal topic
> > names right now.
> >
> > -Matthias
> >
> > On 11/17/16 8:45 AM, Srikanth wrote:
> > > That is right Michael. Most teams that use kafka library can adhere to
> > > certain naming convention.
> > > Using streams API will break that.
> > >
> > > Srikanth
> > >
> > > On Wed, Nov 16, 2016 at 2:32 PM, Michael Noll <mi...@confluent.io>
> > wrote:
> > >
> > >> Srikanth,
> > >>
> > >> no, there's isn't any API to control the naming of internal topics.
> > >>
> > >> Is the reason you're asking for such functionality only/mostly about
> > >> multi-tenancy issues (as you mentioned in your first message)?
> > >>
> > >> -Michael
> > >>
> > >>
> > >>
> > >> On Wed, Nov 16, 2016 at 8:20 PM, Srikanth <sr...@gmail.com>
> > wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> Does kafka stream provide an API to control how internal topics are
> > >> named?
> > >>> Right now it uses appId, operator name, etc.
> > >>> In a shared kafka cluster its common to have naming convention that
> may
> > >>> require some prefix/suffix.
> > >>>
> > >>> Srikanth
> > >>>
> > >>
> > >
> >
> >
>
>
> --
> *Michael G. Noll*
> Product Manager | Confluent
> +1 650 453 5860 | @miguno <https://twitter.com/miguno>
> Follow us: Twitter <https://twitter.com/ConfluentInc> | Blog
> <http://www.confluent.io/blog>
>
--
-- Guozhang
Re: Kafka Streams internal topic naming
Posted by Michael Noll <mi...@confluent.io>.
Srikanth,
as Matthias said, you can achieve some namespacing effects through the use
of (your own in-house) conventions of defining `application.id` across
teams. The id is used as the prefix for topics, see
http://docs.confluent.io/current/streams/developer-guide.html#required-configuration-parameters
for further details.
Best,
Michael
On Thu, Nov 17, 2016 at 6:30 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> The only way to influence the naming is via application.id which you can
> set as you wish. Hope this is good enough to meet your naming conventions.
>
> As Michael mentioned, there is no way to manually specify internal topic
> names right now.
>
> -Matthias
>
> On 11/17/16 8:45 AM, Srikanth wrote:
> > That is right Michael. Most teams that use kafka library can adhere to
> > certain naming convention.
> > Using streams API will break that.
> >
> > Srikanth
> >
> > On Wed, Nov 16, 2016 at 2:32 PM, Michael Noll <mi...@confluent.io>
> wrote:
> >
> >> Srikanth,
> >>
> >> no, there's isn't any API to control the naming of internal topics.
> >>
> >> Is the reason you're asking for such functionality only/mostly about
> >> multi-tenancy issues (as you mentioned in your first message)?
> >>
> >> -Michael
> >>
> >>
> >>
> >> On Wed, Nov 16, 2016 at 8:20 PM, Srikanth <sr...@gmail.com>
> wrote:
> >>
> >>> Hello,
> >>>
> >>> Does kafka stream provide an API to control how internal topics are
> >> named?
> >>> Right now it uses appId, operator name, etc.
> >>> In a shared kafka cluster its common to have naming convention that may
> >>> require some prefix/suffix.
> >>>
> >>> Srikanth
> >>>
> >>
> >
>
>
--
*Michael G. Noll*
Product Manager | Confluent
+1 650 453 5860 | @miguno <https://twitter.com/miguno>
Follow us: Twitter <https://twitter.com/ConfluentInc> | Blog
<http://www.confluent.io/blog>
Re: Kafka Streams internal topic naming
Posted by "Matthias J. Sax" <ma...@confluent.io>.
The only way to influence the naming is via application.id which you can
set as you wish. Hope this is good enough to meet your naming conventions.
As Michael mentioned, there is no way to manually specify internal topic
names right now.
-Matthias
On 11/17/16 8:45 AM, Srikanth wrote:
> That is right Michael. Most teams that use kafka library can adhere to
> certain naming convention.
> Using streams API will break that.
>
> Srikanth
>
> On Wed, Nov 16, 2016 at 2:32 PM, Michael Noll <mi...@confluent.io> wrote:
>
>> Srikanth,
>>
>> no, there's isn't any API to control the naming of internal topics.
>>
>> Is the reason you're asking for such functionality only/mostly about
>> multi-tenancy issues (as you mentioned in your first message)?
>>
>> -Michael
>>
>>
>>
>> On Wed, Nov 16, 2016 at 8:20 PM, Srikanth <sr...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Does kafka stream provide an API to control how internal topics are
>> named?
>>> Right now it uses appId, operator name, etc.
>>> In a shared kafka cluster its common to have naming convention that may
>>> require some prefix/suffix.
>>>
>>> Srikanth
>>>
>>
>
Re: Kafka Streams internal topic naming
Posted by Srikanth <sr...@gmail.com>.
That is right Michael. Most teams that use kafka library can adhere to
certain naming convention.
Using streams API will break that.
Srikanth
On Wed, Nov 16, 2016 at 2:32 PM, Michael Noll <mi...@confluent.io> wrote:
> Srikanth,
>
> no, there's isn't any API to control the naming of internal topics.
>
> Is the reason you're asking for such functionality only/mostly about
> multi-tenancy issues (as you mentioned in your first message)?
>
> -Michael
>
>
>
> On Wed, Nov 16, 2016 at 8:20 PM, Srikanth <sr...@gmail.com> wrote:
>
> > Hello,
> >
> > Does kafka stream provide an API to control how internal topics are
> named?
> > Right now it uses appId, operator name, etc.
> > In a shared kafka cluster its common to have naming convention that may
> > require some prefix/suffix.
> >
> > Srikanth
> >
>
Re: Kafka Streams internal topic naming
Posted by Michael Noll <mi...@confluent.io>.
Srikanth,
no, there's isn't any API to control the naming of internal topics.
Is the reason you're asking for such functionality only/mostly about
multi-tenancy issues (as you mentioned in your first message)?
-Michael
On Wed, Nov 16, 2016 at 8:20 PM, Srikanth <sr...@gmail.com> wrote:
> Hello,
>
> Does kafka stream provide an API to control how internal topics are named?
> Right now it uses appId, operator name, etc.
> In a shared kafka cluster its common to have naming convention that may
> require some prefix/suffix.
>
> Srikanth
>