You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by "J.R. Pauley" <jr...@gmail.com> on 2017/08/08 12:37:23 UTC
standalone hello pulsar
Regarding the config files for zookeeper etc, are the defaults meant to
work or need changing?
I'm not locating any pulsar-standalone.log and checking ps no zookeeper is
started by default from bin/pulsar standalone.
I'm guessing I need to start zookeeper separately. No idea about bookkeeper
etc. The Gettign Started info is a bit sparse and contains a few 404's also.
My goal is to try out storm adaptors and I already have a running
storm/zookeeper, but not a good start when I can't get the Hello example to
produce a log.
Any quick tips?
Re: standalone hello pulsar
Posted by Matteo Merli <mm...@apache.org>.
You can try to debug the issue with simple command line tools, like
"pulsar-client" or "pulsar-perf".
Example and reference is here:
https://pulsar.incubator.apache.org/docs/latest/reference/CliTools/
On Fri, Aug 18, 2017 at 10:55 AM J.R. Pauley <jr...@gmail.com> wrote:
> Thanks. Only reason I ask is the exception I posted to Heron group. Trying
> to specify config URL http://localhost:6650 or pulsar://localhost:6650
> does not seem to do the trick
>
> On Fri, Aug 18, 2017 at 1:43 PM, Matteo Merli <mm...@apache.org> wrote:
>
>> You can certainly use the Pulsar standalone to feed streaming topologies.
>> The only constraints of the standalone are more around:
>> * Throughput (only 1 broker)
>> * Availability (if that machine/process goes down, the "service" is down)
>> * Durability (data is stored in a single node, lose one disk, lose the
>> data)
>>
>> Deploying a multi-node clusters with brokers and bookies is a bit more
>> involved, and probably not worth it to just do simple tests.
>> You can find docs for deploying on bare metal or Kubernetes in
>> https://pulsar.incubator.apache.org/docs/latest/deployment/InstanceSetup/
>>
>>
>> On Fri, Aug 18, 2017 at 9:38 AM J.R. Pauley <jr...@gmail.com> wrote:
>>
>>> is standalone suitable for use w/ Heron and PulsarSpout adaptor or does
>>> pulsar need to be started in broker mode? If the latter how do you specify
>>> the config file to use?
>>>
>>> --
>> Matteo Merli
>> <mm...@apache.org>
>>
>
> --
Matteo Merli
<mm...@apache.org>
Re: standalone hello pulsar
Posted by "J.R. Pauley" <jr...@gmail.com>.
Thanks. Only reason I ask is the exception I posted to Heron group. Trying
to specify config URL http://localhost:6650 or pulsar://localhost:6650 does
not seem to do the trick
On Fri, Aug 18, 2017 at 1:43 PM, Matteo Merli <mm...@apache.org> wrote:
> You can certainly use the Pulsar standalone to feed streaming topologies.
> The only constraints of the standalone are more around:
> * Throughput (only 1 broker)
> * Availability (if that machine/process goes down, the "service" is down)
> * Durability (data is stored in a single node, lose one disk, lose the
> data)
>
> Deploying a multi-node clusters with brokers and bookies is a bit more
> involved, and probably not worth it to just do simple tests.
> You can find docs for deploying on bare metal or Kubernetes in
> https://pulsar.incubator.apache.org/docs/latest/deployment/InstanceSetup/
>
> On Fri, Aug 18, 2017 at 9:38 AM J.R. Pauley <jr...@gmail.com> wrote:
>
>> is standalone suitable for use w/ Heron and PulsarSpout adaptor or does
>> pulsar need to be started in broker mode? If the latter how do you specify
>> the config file to use?
>>
>> --
> Matteo Merli
> <mm...@apache.org>
>
Re: standalone hello pulsar
Posted by Matteo Merli <mm...@apache.org>.
You can certainly use the Pulsar standalone to feed streaming topologies.
The only constraints of the standalone are more around:
* Throughput (only 1 broker)
* Availability (if that machine/process goes down, the "service" is down)
* Durability (data is stored in a single node, lose one disk, lose the
data)
Deploying a multi-node clusters with brokers and bookies is a bit more
involved, and probably not worth it to just do simple tests.
You can find docs for deploying on bare metal or Kubernetes in
https://pulsar.incubator.apache.org/docs/latest/deployment/InstanceSetup/
On Fri, Aug 18, 2017 at 9:38 AM J.R. Pauley <jr...@gmail.com> wrote:
> is standalone suitable for use w/ Heron and PulsarSpout adaptor or does
> pulsar need to be started in broker mode? If the latter how do you specify
> the config file to use?
>
> --
Matteo Merli
<mm...@apache.org>
Re: standalone hello pulsar
Posted by "J.R. Pauley" <jr...@gmail.com>.
is standalone suitable for use w/ Heron and PulsarSpout adaptor or does
pulsar need to be started in broker mode? If the latter how do you specify
the config file to use?
On Wed, Aug 9, 2017 at 2:47 PM, Matteo Merli <mm...@apache.org> wrote:
> So, for quick tests with the pulsar-standalone there's no need to worry
> about
> ZooKeeper, since it's providing everything out of the box.
>
> So, you need to deploy a ZK cluster or connect to an existing one only if
> you're trying
> to deploy a multi-node Pulsar cluster, with "real" brokers and bookies.
>
> From the perspective of Pulsar and Storm there is no requirement to share
> the same
> ZooKeeper cluster, though nothing prevents from doing it.
>
> I would say, that if the load and data set size on an existing ZK cluster
> are not that high,
> and you don't plan to create 100Ks of topics, there should be no problem
> in sharing
> the cluster.
>
> I guess the main challenges can arise if there are multiple teams
> interacting on the
> same ZK cluster with many different workloads. Some unexpected mistakes
> from one
> team could cause instabilities on the whole cluster. In practice, in a
> correctly setup
> hardware (multiple disks, enough mem and no swap) the ZK cluster can
> sustain
> very intensive workloads with super-high reliability.
>
> > Not too clear on how to get started with pulsar feeding into storm
> spout. Thanks
>
> I would just start with running the Pulsar standalone in one machine. Then
> follow
> the examples at https://pulsar.incubator.apache.org/docs/latest/
> adaptors/PulsarStorm/ for
> how to write a simple topology using the PulsarSpout adapter.
> You then just need to point the "serviceUrl" to the address of the machine
> where the
> Pulsar standalone service is running.
>
>
>
>
> On Wed, Aug 9, 2017 at 11:23 AM J.R. Pauley <jr...@gmail.com> wrote:
>
>> Hi. Thanks. Do you happen to know when using with Storm and trying out
>> PulsarSpout and Bolt, given that storm has its own zookeeper, what is best
>> practice for that? Do you want both storm and pulsar to have their separate
>> zookeeper, or share one with storm? If the latter where do I tell pulsar
>> not to start its own zookeeper. Not too clear on how to get started with
>> pulsar feeding into storm spout. Thanks
>>
> --
> Matteo Merli
> <mm...@apache.org>
>
Re: standalone hello pulsar
Posted by Matteo Merli <mm...@apache.org>.
So, for quick tests with the pulsar-standalone there's no need to worry
about
ZooKeeper, since it's providing everything out of the box.
So, you need to deploy a ZK cluster or connect to an existing one only if
you're trying
to deploy a multi-node Pulsar cluster, with "real" brokers and bookies.
From the perspective of Pulsar and Storm there is no requirement to share
the same
ZooKeeper cluster, though nothing prevents from doing it.
I would say, that if the load and data set size on an existing ZK cluster
are not that high,
and you don't plan to create 100Ks of topics, there should be no problem in
sharing
the cluster.
I guess the main challenges can arise if there are multiple teams
interacting on the
same ZK cluster with many different workloads. Some unexpected mistakes
from one
team could cause instabilities on the whole cluster. In practice, in a
correctly setup
hardware (multiple disks, enough mem and no swap) the ZK cluster can sustain
very intensive workloads with super-high reliability.
> Not too clear on how to get started with pulsar feeding into storm spout.
Thanks
I would just start with running the Pulsar standalone in one machine. Then
follow
the examples at
https://pulsar.incubator.apache.org/docs/latest/adaptors/PulsarStorm/ for
how to write a simple topology using the PulsarSpout adapter.
You then just need to point the "serviceUrl" to the address of the machine
where the
Pulsar standalone service is running.
On Wed, Aug 9, 2017 at 11:23 AM J.R. Pauley <jr...@gmail.com> wrote:
> Hi. Thanks. Do you happen to know when using with Storm and trying out
> PulsarSpout and Bolt, given that storm has its own zookeeper, what is best
> practice for that? Do you want both storm and pulsar to have their separate
> zookeeper, or share one with storm? If the latter where do I tell pulsar
> not to start its own zookeeper. Not too clear on how to get started with
> pulsar feeding into storm spout. Thanks
>
--
Matteo Merli
<mm...@apache.org>
Re: standalone hello pulsar
Posted by "J.R. Pauley" <jr...@gmail.com>.
Hi. Thanks. Do you happen to know when using with Storm and trying out
PulsarSpout and Bolt, given that storm has its own zookeeper, what is best
practice for that? Do you want both storm and pulsar to have their separate
zookeeper, or share one with storm? If the latter where do I tell pulsar
not to start its own zookeeper. Not too clear on how to get started with
pulsar feeding into storm spout. Thanks
On Tue, Aug 8, 2017 at 1:02 PM, Matteo Merli <mm...@apache.org> wrote:
> Hi James,
>
> the Pulsar standalone is a self-contained Pulsar instance and it starts
> everything it needs, within a single JVM, the broker, BookKeeper and
> ZooKeeper.
> It is meant to be used for testing or anywhere availability, replication
> and scalability are not a concern.
>
> By default the standalone logs are printed in the console.
>
> You can use:
> bin/pulsar-daemon start standalone
>
> to start it in background with logs redirected to a file. In any case,
> just modify the "conf/log4j.properties" for the logging configuration.
>
> Matteo
>
> On Tue, Aug 8, 2017 at 5:37 AM J.R. Pauley <jr...@gmail.com> wrote:
>
>> Regarding the config files for zookeeper etc, are the defaults meant to
>> work or need changing?
>>
>> I'm not locating any pulsar-standalone.log and checking ps no zookeeper
>> is started by default from bin/pulsar standalone.
>>
>> I'm guessing I need to start zookeeper separately. No idea about
>> bookkeeper etc. The Gettign Started info is a bit sparse and contains a few
>> 404's also.
>>
>> My goal is to try out storm adaptors and I already have a running
>> storm/zookeeper, but not a good start when I can't get the Hello example to
>> produce a log.
>>
>> Any quick tips?
>>
> --
> Matteo Merli
> <mm...@apache.org>
>
Re: standalone hello pulsar
Posted by Matteo Merli <mm...@apache.org>.
Hi James,
the Pulsar standalone is a self-contained Pulsar instance and it starts
everything it needs, within a single JVM, the broker, BookKeeper and
ZooKeeper.
It is meant to be used for testing or anywhere availability, replication
and scalability are not a concern.
By default the standalone logs are printed in the console.
You can use:
bin/pulsar-daemon start standalone
to start it in background with logs redirected to a file. In any case, just
modify the "conf/log4j.properties" for the logging configuration.
Matteo
On Tue, Aug 8, 2017 at 5:37 AM J.R. Pauley <jr...@gmail.com> wrote:
> Regarding the config files for zookeeper etc, are the defaults meant to
> work or need changing?
>
> I'm not locating any pulsar-standalone.log and checking ps no zookeeper is
> started by default from bin/pulsar standalone.
>
> I'm guessing I need to start zookeeper separately. No idea about
> bookkeeper etc. The Gettign Started info is a bit sparse and contains a few
> 404's also.
>
> My goal is to try out storm adaptors and I already have a running
> storm/zookeeper, but not a good start when I can't get the Hello example to
> produce a log.
>
> Any quick tips?
>
--
Matteo Merli
<mm...@apache.org>