You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Sa Li <sa...@gmail.com> on 2014/09/16 23:06:36 UTC
muliple-nodes kafka cluster
Hi, All
I have been using kafka cluster in single server with three brokers, but I
am thinking to build a larger kafka cluster, say 4 nodes (server), and 3
brokers in each node, so totally 12 brokers, would that be better than
single node cluster? Or single node will be fair enough, since web api may
push million rows into kafka cluster every day, I am kinda worry if the
cluster is capable to take such much data without losing data. If I am
using multiple-nodes, the tradeoff is the connection time among different
nodes?
thanks
Alec
Re: muliple-nodes kafka cluster
Posted by Gezim Musliaj <gm...@gmail.com>.
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is
missin
g, no dependency information available
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Storm .............................................. SUCCESS [
2.816 s]
[INFO] maven-shade-clojure-transformer .................... SUCCESS [
30.954 s]
[INFO] Storm Core ......................................... SUCCESS [01:34
min]
[INFO] storm-starter ...................................... FAILURE [
0.128 s]
[INFO] storm-kafka ........................................ SKIPPED
[INFO] storm-hdfs ......................................... SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 02:08 min
[INFO] Finished at: 2014-09-16T23:51:57+02:00
[INFO] Final Memory: 39M/411M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal on project storm-starter: Could not resolve
depen
dencies for project
org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT
: Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in
http://rep
o1.maven.org/maven2/ was cached in the local repository, resolution will
not be
reattempted until the update interval of central has elapsed or updates are
forc
ed -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please rea
d the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso
lutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR] mvn <goals> -rf :storm-starter
On Tue, Sep 16, 2014 at 11:46 PM, Harsha <st...@harsha.io> wrote:
> Hi Alec,
> Single node kafka cluster not recommended apart from using it for
> development. I highly recommend using multinode cluster and create a
> partitioned topic with replication. This not only makes it optimal to take
> in more data at faster rates also allows your cluster running if there is a
> node failure as the topic is replicated there wouldn't be huge data loss.
>
> " If I am using multiple-nodes, the tradeoff is the connection time among
> different nodes?"
> kafka producer api sends a message to broker either round-robin
> or based on partition function.
> please go through the kafka docs here
> http://kafka.apache.org/documentation.html for simple consumer and also
> how the replication works among multiple nodes.
>
> -Harsha
>
>
> On Tue, Sep 16, 2014, at 02:06 PM, Sa Li wrote:
>
> Hi, All
>
> I have been using kafka cluster in single server with three brokers, but I
> am thinking to build a larger kafka cluster, say 4 nodes (server), and 3
> brokers in each node, so totally 12 brokers, would that be better than
> single node cluster? Or single node will be fair enough, since web api may
> push million rows into kafka cluster every day, I am kinda worry if the
> cluster is capable to take such much data without losing data. If I am
> using multiple-nodes, the tradeoff is the connection time among different
> nodes?
>
>
> thanks
>
> Alec
>
>
>
Re: muliple-nodes kafka cluster
Posted by Harsha <st...@harsha.io>.
Hi Alec,
Single node kafka cluster not recommended apart from
using it for development. I highly recommend using multinode
cluster and create a partitioned topic with replication. This
not only makes it optimal to take in more data at faster rates
also allows your cluster running if there is a node failure as
the topic is replicated there wouldn't be huge data loss.
" If I am using multiple-nodes, the tradeoff is the connection
time among different nodes?"
kafka producer api sends a message to broker either
round-robin or based on partition function.
please go through the kafka docs
here [1]http://kafka.apache.org/documentation.html for simple
consumer and also how the replication works among multiple
nodes.
-Harsha
On Tue, Sep 16, 2014, at 02:06 PM, Sa Li wrote:
Hi, All
I have been using kafka cluster in single server with three
brokers, but I am thinking to build a larger kafka cluster, say
4 nodes (server), and 3 brokers in each node, so totally 12
brokers, would that be better than single node cluster? Or
single node will be fair enough, since web api may push million
rows into kafka cluster every day, I am kinda worry if the
cluster is capable to take such much data without losing data.
If I am using multiple-nodes, the tradeoff is the connection
time among different nodes?
thanks
Alec
References
1. http://kafka.apache.org/documentation.html