You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Roger Hoover <ro...@gmail.com> on 2014/12/18 01:51:25 UTC
How to get number of partitions for a downstream topic?
Hi all,
Is there an easy way to get the number of partitions for a downstream
topic? Or would I have to instantiate a Kafka producer myself in the
Task.init().
I appreciate any suggestions. Thanks,
Roger
Re: How to get number of partitions for a downstream topic?
Posted by Roger Hoover <ro...@gmail.com>.
Fantastic. Much appreciated, Chris.
On Thu, Dec 18, 2014 at 11:11 AM, Chris Riccomini <
criccomini@linkedin.com.invalid> wrote:
>
> Hey Roger,
>
> You shouldn't need to use Scala, though you can if you want. The
> SystemFactory and SystemAdmin APIs are both Java-based.
>
> Set<String> streamNames = new HashSet<String>();
> streamNames.add("my-topic");
> SystemAdmin kafkaSystemAdmin = new KafkaSystemFactory().getAdmin("kafka",
> config);
> Map<String, SystemStreamMetadata> metadata =
> SyskafkaSystemAdmin.getSystemStreamMetadata(streamNames);
>
> You can get the config object that you need from InitableTask.config().
>
> Cheers,
> Chris
>
> On 12/18/14 10:37 AM, "Roger Hoover" <ro...@gmail.com> wrote:
>
> >Looking around the code base, I think it's only available in Scala
> >(samza-core). That's ok. I can write the job in Scala. Need a little
> >help on how to instantiate the object though...
> >
> >On Thu, Dec 18, 2014 at 10:18 AM, Roger Hoover <ro...@gmail.com>
> >wrote:
> >>
> >> To do this, I think I need a SystemFactory object so that I can
> >> call getAdmin() on it to get the SystemAdmin object you mentioned. Can
> >>you
> >> give me a pointer on how to do that in the task init()?
> >>
> >> Thanks!
> >>
> >> On Wed, Dec 17, 2014 at 5:20 PM, Roger Hoover <ro...@gmail.com>
> >> wrote:
> >>>
> >>> Thanks, Chris!
> >>>
> >>> Sent from my iPhone
> >>>
> >>> > On Dec 17, 2014, at 5:10 PM, Chris Riccomini
> >>> <cr...@linkedin.com.INVALID> wrote:
> >>> >
> >>> > Hey Roger,
> >>> >
> >>> > You could instantiate a SystemAdmin for the topic in question. This
> >>> > interface has a getSystemStreamMetadata call that will give you
> >>> partition
> >>> > information for arbitrary topics and systems.
> >>> >
> >>> > Cheers,
> >>> > Chris
> >>> >
> >>> >> On 12/17/14 4:51 PM, "Roger Hoover" <ro...@gmail.com> wrote:
> >>> >>
> >>> >> Hi all,
> >>> >>
> >>> >> Is there an easy way to get the number of partitions for a
> >>>downstream
> >>> >> topic? Or would I have to instantiate a Kafka producer myself in
> >>>the
> >>> >> Task.init().
> >>> >>
> >>> >> I appreciate any suggestions. Thanks,
> >>> >>
> >>> >> Roger
> >>> >
> >>>
> >>
>
>
Re: How to get number of partitions for a downstream topic?
Posted by Chris Riccomini <cr...@linkedin.com.INVALID>.
Hey Roger,
You shouldn't need to use Scala, though you can if you want. The
SystemFactory and SystemAdmin APIs are both Java-based.
Set<String> streamNames = new HashSet<String>();
streamNames.add("my-topic");
SystemAdmin kafkaSystemAdmin = new KafkaSystemFactory().getAdmin("kafka",
config);
Map<String, SystemStreamMetadata> metadata =
SyskafkaSystemAdmin.getSystemStreamMetadata(streamNames);
You can get the config object that you need from InitableTask.config().
Cheers,
Chris
On 12/18/14 10:37 AM, "Roger Hoover" <ro...@gmail.com> wrote:
>Looking around the code base, I think it's only available in Scala
>(samza-core). That's ok. I can write the job in Scala. Need a little
>help on how to instantiate the object though...
>
>On Thu, Dec 18, 2014 at 10:18 AM, Roger Hoover <ro...@gmail.com>
>wrote:
>>
>> To do this, I think I need a SystemFactory object so that I can
>> call getAdmin() on it to get the SystemAdmin object you mentioned. Can
>>you
>> give me a pointer on how to do that in the task init()?
>>
>> Thanks!
>>
>> On Wed, Dec 17, 2014 at 5:20 PM, Roger Hoover <ro...@gmail.com>
>> wrote:
>>>
>>> Thanks, Chris!
>>>
>>> Sent from my iPhone
>>>
>>> > On Dec 17, 2014, at 5:10 PM, Chris Riccomini
>>> <cr...@linkedin.com.INVALID> wrote:
>>> >
>>> > Hey Roger,
>>> >
>>> > You could instantiate a SystemAdmin for the topic in question. This
>>> > interface has a getSystemStreamMetadata call that will give you
>>> partition
>>> > information for arbitrary topics and systems.
>>> >
>>> > Cheers,
>>> > Chris
>>> >
>>> >> On 12/17/14 4:51 PM, "Roger Hoover" <ro...@gmail.com> wrote:
>>> >>
>>> >> Hi all,
>>> >>
>>> >> Is there an easy way to get the number of partitions for a
>>>downstream
>>> >> topic? Or would I have to instantiate a Kafka producer myself in
>>>the
>>> >> Task.init().
>>> >>
>>> >> I appreciate any suggestions. Thanks,
>>> >>
>>> >> Roger
>>> >
>>>
>>
Re: How to get number of partitions for a downstream topic?
Posted by Roger Hoover <ro...@gmail.com>.
Looking around the code base, I think it's only available in Scala
(samza-core). That's ok. I can write the job in Scala. Need a little
help on how to instantiate the object though...
On Thu, Dec 18, 2014 at 10:18 AM, Roger Hoover <ro...@gmail.com>
wrote:
>
> To do this, I think I need a SystemFactory object so that I can
> call getAdmin() on it to get the SystemAdmin object you mentioned. Can you
> give me a pointer on how to do that in the task init()?
>
> Thanks!
>
> On Wed, Dec 17, 2014 at 5:20 PM, Roger Hoover <ro...@gmail.com>
> wrote:
>>
>> Thanks, Chris!
>>
>> Sent from my iPhone
>>
>> > On Dec 17, 2014, at 5:10 PM, Chris Riccomini
>> <cr...@linkedin.com.INVALID> wrote:
>> >
>> > Hey Roger,
>> >
>> > You could instantiate a SystemAdmin for the topic in question. This
>> > interface has a getSystemStreamMetadata call that will give you
>> partition
>> > information for arbitrary topics and systems.
>> >
>> > Cheers,
>> > Chris
>> >
>> >> On 12/17/14 4:51 PM, "Roger Hoover" <ro...@gmail.com> wrote:
>> >>
>> >> Hi all,
>> >>
>> >> Is there an easy way to get the number of partitions for a downstream
>> >> topic? Or would I have to instantiate a Kafka producer myself in the
>> >> Task.init().
>> >>
>> >> I appreciate any suggestions. Thanks,
>> >>
>> >> Roger
>> >
>>
>
Re: How to get number of partitions for a downstream topic?
Posted by Roger Hoover <ro...@gmail.com>.
To do this, I think I need a SystemFactory object so that I can
call getAdmin() on it to get the SystemAdmin object you mentioned. Can you
give me a pointer on how to do that in the task init()?
Thanks!
On Wed, Dec 17, 2014 at 5:20 PM, Roger Hoover <ro...@gmail.com>
wrote:
>
> Thanks, Chris!
>
> Sent from my iPhone
>
> > On Dec 17, 2014, at 5:10 PM, Chris Riccomini
> <cr...@linkedin.com.INVALID> wrote:
> >
> > Hey Roger,
> >
> > You could instantiate a SystemAdmin for the topic in question. This
> > interface has a getSystemStreamMetadata call that will give you partition
> > information for arbitrary topics and systems.
> >
> > Cheers,
> > Chris
> >
> >> On 12/17/14 4:51 PM, "Roger Hoover" <ro...@gmail.com> wrote:
> >>
> >> Hi all,
> >>
> >> Is there an easy way to get the number of partitions for a downstream
> >> topic? Or would I have to instantiate a Kafka producer myself in the
> >> Task.init().
> >>
> >> I appreciate any suggestions. Thanks,
> >>
> >> Roger
> >
>
Re: How to get number of partitions for a downstream topic?
Posted by Roger Hoover <ro...@gmail.com>.
Thanks, Chris!
Sent from my iPhone
> On Dec 17, 2014, at 5:10 PM, Chris Riccomini <cr...@linkedin.com.INVALID> wrote:
>
> Hey Roger,
>
> You could instantiate a SystemAdmin for the topic in question. This
> interface has a getSystemStreamMetadata call that will give you partition
> information for arbitrary topics and systems.
>
> Cheers,
> Chris
>
>> On 12/17/14 4:51 PM, "Roger Hoover" <ro...@gmail.com> wrote:
>>
>> Hi all,
>>
>> Is there an easy way to get the number of partitions for a downstream
>> topic? Or would I have to instantiate a Kafka producer myself in the
>> Task.init().
>>
>> I appreciate any suggestions. Thanks,
>>
>> Roger
>
Re: How to get number of partitions for a downstream topic?
Posted by Chris Riccomini <cr...@linkedin.com.INVALID>.
Hey Roger,
You could instantiate a SystemAdmin for the topic in question. This
interface has a getSystemStreamMetadata call that will give you partition
information for arbitrary topics and systems.
Cheers,
Chris
On 12/17/14 4:51 PM, "Roger Hoover" <ro...@gmail.com> wrote:
>Hi all,
>
>Is there an easy way to get the number of partitions for a downstream
>topic? Or would I have to instantiate a Kafka producer myself in the
>Task.init().
>
>I appreciate any suggestions. Thanks,
>
>Roger