You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Robert Wigginton <RW...@helixeducation.com.INVALID> on 2020/02/04 17:16:54 UTC
Samza consumer without Zookeeper
Hello,
We are currently evaluating moving from an on prem Kafka deployment to
Confluent Cloud. Confluent Cloud does not expose Zookeeper to user.
Is it possible to setup a Samza consumer without the zookeeper connect
option?
Thanks,
Robert
Re: Samza consumer without Zookeeper
Posted by Robert Wigginton <RW...@helixeducation.com.INVALID>.
Thanks Prateek and Yi.
We were able to find the reference in the application. After removing it, we are able to produce and consume without zookeeper.
On 2/10/20, 3:46 PM, "Prateek Maheshwari" <pr...@utexas.edu> wrote:
Robert, from the stacktrace this looks like something in the application
logic is looking for that key (i.e. in the HelixApp constructor).
Thanks,
Prateek
On Mon, Feb 10, 2020 at 1:14 PM Robert Wigginton
<RW...@helixeducation.com.invalid> wrote:
> Yi,
>
> I think I am getting ahead of myself. We are not getting past the job
> coordinator portion of the config. Running 1.3 and defining
> job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory
> in the properties file we still get the error below.
>
>
> Exception in thread "main" org.apache.samza.config.ConfigException:
> Missing key job.coordinator.zk.connect.
> at org.apache.samza.config.Config.getList(Config.java:168)
> at com.helixeducation.porter.HelixApp.<init>(HelixApp.java:29)
> at
> com.helixeducation.porter.HelixRocksDBApp.<init>(HelixRocksDBApp.java:16)
> at
> com.helixeducation.porter.inquirySubmission.InquirySubmissionApp.<init>(InquirySubmissionApp.java:25)
> at
> com.helixeducation.porter.inquirySubmission.InquirySubmissionTaskRunner.main(InquirySubmissionTaskRunner.java:19)
>
> On 2/5/20, 6:32 PM, "Yi Pan" <ni...@gmail.com> wrote:
>
> Hi, Robert,
>
> Thanks! I believe since Samza 1.2, the ZK dependency from Kafka
> consumer is
> removed from Samza. I briefly checked the code base from master and
> there
> is no reference to KafkaConsumerConfig.getZkConnect(). Do you see any
> error
> messages when you remove the systems.x.consumer.zookeeper.connect from
> your
> config?
>
> -Yi
>
> On Wed, Feb 5, 2020 at 3:12 PM Robert Wigginton
> <RW...@helixeducation.com.invalid> wrote:
>
> > Currently 1.0.0 but are in the process of upgrading to 1.3.0.
> >
> > On Wed, 2020-02-05 at 15:01 -0800, Yi Pan wrote:
> > > Hi, Robert,
> > >
> > > Which version of Samza are you using?
> > >
> > > -Yi
> > >
> > > On Tue, Feb 4, 2020 at 9:17 AM Robert Wigginton
> > > <RW...@helixeducation.com.invalid> wrote:
> > >
> > > > Hello,
> > > >
> > > > We are currently evaluating moving from an on prem Kafka
> deployment
> > > > to
> > > > Confluent Cloud. Confluent Cloud does not expose Zookeeper to
> > > > user.
> > > > Is it possible to setup a Samza consumer without the zookeeper
> > > > connect
> > > > option?
> > > >
> > > > Thanks,
> > > >
> > > > Robert
> > > >
> >
>
>
>
Re: Samza consumer without Zookeeper
Posted by Prateek Maheshwari <pr...@utexas.edu>.
Robert, from the stacktrace this looks like something in the application
logic is looking for that key (i.e. in the HelixApp constructor).
Thanks,
Prateek
On Mon, Feb 10, 2020 at 1:14 PM Robert Wigginton
<RW...@helixeducation.com.invalid> wrote:
> Yi,
>
> I think I am getting ahead of myself. We are not getting past the job
> coordinator portion of the config. Running 1.3 and defining
> job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory
> in the properties file we still get the error below.
>
>
> Exception in thread "main" org.apache.samza.config.ConfigException:
> Missing key job.coordinator.zk.connect.
> at org.apache.samza.config.Config.getList(Config.java:168)
> at com.helixeducation.porter.HelixApp.<init>(HelixApp.java:29)
> at
> com.helixeducation.porter.HelixRocksDBApp.<init>(HelixRocksDBApp.java:16)
> at
> com.helixeducation.porter.inquirySubmission.InquirySubmissionApp.<init>(InquirySubmissionApp.java:25)
> at
> com.helixeducation.porter.inquirySubmission.InquirySubmissionTaskRunner.main(InquirySubmissionTaskRunner.java:19)
>
> On 2/5/20, 6:32 PM, "Yi Pan" <ni...@gmail.com> wrote:
>
> Hi, Robert,
>
> Thanks! I believe since Samza 1.2, the ZK dependency from Kafka
> consumer is
> removed from Samza. I briefly checked the code base from master and
> there
> is no reference to KafkaConsumerConfig.getZkConnect(). Do you see any
> error
> messages when you remove the systems.x.consumer.zookeeper.connect from
> your
> config?
>
> -Yi
>
> On Wed, Feb 5, 2020 at 3:12 PM Robert Wigginton
> <RW...@helixeducation.com.invalid> wrote:
>
> > Currently 1.0.0 but are in the process of upgrading to 1.3.0.
> >
> > On Wed, 2020-02-05 at 15:01 -0800, Yi Pan wrote:
> > > Hi, Robert,
> > >
> > > Which version of Samza are you using?
> > >
> > > -Yi
> > >
> > > On Tue, Feb 4, 2020 at 9:17 AM Robert Wigginton
> > > <RW...@helixeducation.com.invalid> wrote:
> > >
> > > > Hello,
> > > >
> > > > We are currently evaluating moving from an on prem Kafka
> deployment
> > > > to
> > > > Confluent Cloud. Confluent Cloud does not expose Zookeeper to
> > > > user.
> > > > Is it possible to setup a Samza consumer without the zookeeper
> > > > connect
> > > > option?
> > > >
> > > > Thanks,
> > > >
> > > > Robert
> > > >
> >
>
>
>
Re: Samza consumer without Zookeeper
Posted by Robert Wigginton <RW...@helixeducation.com.INVALID>.
Yi,
I think I am getting ahead of myself. We are not getting past the job coordinator portion of the config. Running 1.3 and defining job.coordinator.factory=org.apache.samza.standalone.PassthroughJobCoordinatorFactory in the properties file we still get the error below.
Exception in thread "main" org.apache.samza.config.ConfigException: Missing key job.coordinator.zk.connect.
at org.apache.samza.config.Config.getList(Config.java:168)
at com.helixeducation.porter.HelixApp.<init>(HelixApp.java:29)
at com.helixeducation.porter.HelixRocksDBApp.<init>(HelixRocksDBApp.java:16)
at com.helixeducation.porter.inquirySubmission.InquirySubmissionApp.<init>(InquirySubmissionApp.java:25)
at com.helixeducation.porter.inquirySubmission.InquirySubmissionTaskRunner.main(InquirySubmissionTaskRunner.java:19)
On 2/5/20, 6:32 PM, "Yi Pan" <ni...@gmail.com> wrote:
Hi, Robert,
Thanks! I believe since Samza 1.2, the ZK dependency from Kafka consumer is
removed from Samza. I briefly checked the code base from master and there
is no reference to KafkaConsumerConfig.getZkConnect(). Do you see any error
messages when you remove the systems.x.consumer.zookeeper.connect from your
config?
-Yi
On Wed, Feb 5, 2020 at 3:12 PM Robert Wigginton
<RW...@helixeducation.com.invalid> wrote:
> Currently 1.0.0 but are in the process of upgrading to 1.3.0.
>
> On Wed, 2020-02-05 at 15:01 -0800, Yi Pan wrote:
> > Hi, Robert,
> >
> > Which version of Samza are you using?
> >
> > -Yi
> >
> > On Tue, Feb 4, 2020 at 9:17 AM Robert Wigginton
> > <RW...@helixeducation.com.invalid> wrote:
> >
> > > Hello,
> > >
> > > We are currently evaluating moving from an on prem Kafka deployment
> > > to
> > > Confluent Cloud. Confluent Cloud does not expose Zookeeper to
> > > user.
> > > Is it possible to setup a Samza consumer without the zookeeper
> > > connect
> > > option?
> > >
> > > Thanks,
> > >
> > > Robert
> > >
>
Re: Samza consumer without Zookeeper
Posted by Yi Pan <ni...@gmail.com>.
Hi, Robert,
Thanks! I believe since Samza 1.2, the ZK dependency from Kafka consumer is
removed from Samza. I briefly checked the code base from master and there
is no reference to KafkaConsumerConfig.getZkConnect(). Do you see any error
messages when you remove the systems.x.consumer.zookeeper.connect from your
config?
-Yi
On Wed, Feb 5, 2020 at 3:12 PM Robert Wigginton
<RW...@helixeducation.com.invalid> wrote:
> Currently 1.0.0 but are in the process of upgrading to 1.3.0.
>
> On Wed, 2020-02-05 at 15:01 -0800, Yi Pan wrote:
> > Hi, Robert,
> >
> > Which version of Samza are you using?
> >
> > -Yi
> >
> > On Tue, Feb 4, 2020 at 9:17 AM Robert Wigginton
> > <RW...@helixeducation.com.invalid> wrote:
> >
> > > Hello,
> > >
> > > We are currently evaluating moving from an on prem Kafka deployment
> > > to
> > > Confluent Cloud. Confluent Cloud does not expose Zookeeper to
> > > user.
> > > Is it possible to setup a Samza consumer without the zookeeper
> > > connect
> > > option?
> > >
> > > Thanks,
> > >
> > > Robert
> > >
>
Re: Samza consumer without Zookeeper
Posted by Robert Wigginton <RW...@helixeducation.com.INVALID>.
Currently 1.0.0 but are in the process of upgrading to 1.3.0.
On Wed, 2020-02-05 at 15:01 -0800, Yi Pan wrote:
> Hi, Robert,
>
> Which version of Samza are you using?
>
> -Yi
>
> On Tue, Feb 4, 2020 at 9:17 AM Robert Wigginton
> <RW...@helixeducation.com.invalid> wrote:
>
> > Hello,
> >
> > We are currently evaluating moving from an on prem Kafka deployment
> > to
> > Confluent Cloud. Confluent Cloud does not expose Zookeeper to
> > user.
> > Is it possible to setup a Samza consumer without the zookeeper
> > connect
> > option?
> >
> > Thanks,
> >
> > Robert
> >
Re: Samza consumer without Zookeeper
Posted by Yi Pan <ni...@gmail.com>.
Hi, Robert,
Which version of Samza are you using?
-Yi
On Tue, Feb 4, 2020 at 9:17 AM Robert Wigginton
<RW...@helixeducation.com.invalid> wrote:
> Hello,
>
> We are currently evaluating moving from an on prem Kafka deployment to
> Confluent Cloud. Confluent Cloud does not expose Zookeeper to user.
> Is it possible to setup a Samza consumer without the zookeeper connect
> option?
>
> Thanks,
>
> Robert
>