You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Meeiling.Bradley" <Me...@target.com> on 2018/09/05 03:07:37 UTC
Controlled topics creation in Kafka cluster, how does that affect a
Kafka Streams App that uses join?
Would I be able to run a Kafka streams application contains join KStream-KStream that consumes and produces to a Kafka cluster that has a formal process to provision topics creation? Since there are behind scene creation of <KSTREAM_JOINOTHER> and <KSTREAM-JOINTHIS> topics need to be created for join operation. These topics are created as kafka streams app executes, if topics creation is a controlled process, is there a way to bypass that while my application is running?
Meeiling Bradley | Lead BI Engineer | • EDABI | 33 South 6th Street, CC-2002 | Minneapolis, MN 55402
Re: Controlled topics creation in Kafka cluster, how does that affect
a Kafka Streams App that uses join?
Posted by "Matthias J. Sax" <ma...@confluent.io>.
You can figure out the names via
builder.build().describe()
The naming pattern is
<application.id>-<operatorName>-[repartition|changelog]
You will also need to create those topic with the correct number of
partitions (usually the number of input topic partitions).
With this information, you can pre-create the topics and Kafka Streams
will use them if the exists. Kafka Streams will also check the number of
partitions -- if it's different to what is required, it will fail and
tell you in the exception message how many partitions it expects.
-Matthias
On 9/5/18 8:39 AM, John Roesler wrote:
> Hi Meeiling,
>
> It sounds like "formal process" is more like "fill out a form beforehand"
> than just "access controlled".
>
> In that case, although it's somewhat of an internal detail, I think the
> easiest thing would be to figure out what the name of the internal topics
> will be and request their creation before you run the app.
>
> The names of the internal topics is predictable, based on the application
> id and the app topology, and it will be stable as long as you don't change
> the app.
>
> I think the simplest approach would be to run it locally and just see which
> topics get created.
>
> Does that help?
> -John
>
> On Tue, Sep 4, 2018 at 10:07 PM Meeiling.Bradley <
> Meeiling.Bradley@target.com> wrote:
>
>> Would I be able to run a Kafka streams application contains join
>> KStream-KStream that consumes and produces to a Kafka cluster that has a
>> formal process to provision topics creation? Since there are behind scene
>> creation of <KSTREAM_JOINOTHER> and <KSTREAM-JOINTHIS> topics need to be
>> created for join operation. These topics are created as kafka streams app
>> executes, if topics creation is a controlled process, is there a way to
>> bypass that while my application is running?
>>
>> Meeiling Bradley | Lead BI Engineer | • EDABI | 33 South 6th
>> Street, CC-2002 | Minneapolis, MN 55402
>>
>>
>>
>>
>
Re: Controlled topics creation in Kafka cluster, how does that affect
a Kafka Streams App that uses join?
Posted by John Roesler <jo...@confluent.io>.
Hi Meeiling,
It sounds like "formal process" is more like "fill out a form beforehand"
than just "access controlled".
In that case, although it's somewhat of an internal detail, I think the
easiest thing would be to figure out what the name of the internal topics
will be and request their creation before you run the app.
The names of the internal topics is predictable, based on the application
id and the app topology, and it will be stable as long as you don't change
the app.
I think the simplest approach would be to run it locally and just see which
topics get created.
Does that help?
-John
On Tue, Sep 4, 2018 at 10:07 PM Meeiling.Bradley <
Meeiling.Bradley@target.com> wrote:
> Would I be able to run a Kafka streams application contains join
> KStream-KStream that consumes and produces to a Kafka cluster that has a
> formal process to provision topics creation? Since there are behind scene
> creation of <KSTREAM_JOINOTHER> and <KSTREAM-JOINTHIS> topics need to be
> created for join operation. These topics are created as kafka streams app
> executes, if topics creation is a controlled process, is there a way to
> bypass that while my application is running?
>
> Meeiling Bradley | Lead BI Engineer | • EDABI | 33 South 6th
> Street, CC-2002 | Minneapolis, MN 55402
>
>
>
>