You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Sven Hodapp <sv...@scai.fraunhofer.de> on 2015/12/09 17:30:03 UTC
Multiple talbet servers on the same host
Hi there,
I've read in the Accumulo Book (p. 496) that it should be possible to start on a (fat) machine multiple tablet servers to scale (also) vertically. Sadly it's not described how to do it. Also I didn't find anything about this issue in the official documentation.
I thought it must be configured in ACCUMULO_HOME/conf/slaves.
So I've added here the same IP with distinct ports multiple times ... in my naive thinking.
But *no* additional tablet servers are started.
Or I am completely wrong here?
Regards,
Sven
--
Sven Hodapp, M.Sc.,
Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
Department of Bioinformatics
Schloss Birlinghoven, 53754 Sankt Augustin, Germany
sven.hodapp@scai.fraunhofer.de
www.scai.fraunhofer.de
Re: Multiple talbet servers on the same host
Posted by Josh Elser <jo...@gmail.com>.
Great! Glad to hear you got it working.
For reference, you could also set tserver.port.client=0 and have
consistent accumulo-site.xml files. Accumulo don't need static ports for
the TabletServer thrift server.
Thanks for sharing instructions. I'm sure this will be useful.
Sven Hodapp wrote:
> Hi Josh,
> HI Billie,
>
> now it works, thank you both!
>
> So in short:
>
> 1. Copy ACCUMULO_HOME to another distinct directory
> 2. export ACCUMULO_HOME=/path/to/new/directory
> 3. Edit conf/accumulo-site.xml in the new directory
>
> <property>
> <name>tserver.port.client</name>
> <value>29997</value> <!-- this should be a free port on the machine -->
> </property>
>
> <property>
> <name>replication.receipt.service.port</name>
> <value>0</value> <!-- 0 means random port -->
> </property>
>
> 4. bin/tup.sh to start the tablet servers
>
> Maybe you can recycle this for the user manual.
>
> Regards,
> Sven
>
Re: Multiple talbet servers on the same host
Posted by Sven Hodapp <sv...@scai.fraunhofer.de>.
Hi Josh,
HI Billie,
now it works, thank you both!
So in short:
1. Copy ACCUMULO_HOME to another distinct directory
2. export ACCUMULO_HOME=/path/to/new/directory
3. Edit conf/accumulo-site.xml in the new directory
<property>
<name>tserver.port.client</name>
<value>29997</value> <!-- this should be a free port on the machine -->
</property>
<property>
<name>replication.receipt.service.port</name>
<value>0</value> <!-- 0 means random port -->
</property>
4. bin/tup.sh to start the tablet servers
Maybe you can recycle this for the user manual.
Regards,
Sven
--
Sven Hodapp, M.Sc.,
Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
Department of Bioinformatics
Schloss Birlinghoven, 53754 Sankt Augustin, Germany
sven.hodapp@scai.fraunhofer.de
www.scai.fraunhofer.de
----- Ursprüngliche Mail -----
> Von: "Josh Elser" <jo...@gmail.com>
> An: "user" <us...@accumulo.apache.org>
> Gesendet: Donnerstag, 10. Dezember 2015 17:56:39
> Betreff: Re: Multiple talbet servers on the same host
> Sven Hodapp wrote:
>> I've read in the Accumulo Book (p. 496) that it should be possible to start on a
>> (fat) machine multiple tablet servers to scale (also) vertically. Sadly it's
>> not described how to do it. Also I didn't find anything about this issue in the
>> official documentation.
>
> I've just created https://issues.apache.org/jira/browse/ACCUMULO-4072
> for us to get some documentation into the Accumulo User Manual on the
> matter. Thanks for letting us know we were missing this.
Re: Multiple talbet servers on the same host
Posted by Josh Elser <jo...@gmail.com>.
Sven Hodapp wrote:
> I've read in the Accumulo Book (p. 496) that it should be possible to start on a (fat) machine multiple tablet servers to scale (also) vertically. Sadly it's not described how to do it. Also I didn't find anything about this issue in the official documentation.
I've just created https://issues.apache.org/jira/browse/ACCUMULO-4072
for us to get some documentation into the Accumulo User Manual on the
matter. Thanks for letting us know we were missing this.
Re: Multiple talbet servers on the same host
Posted by Billie Rinaldi <bi...@gmail.com>.
Ah, I forgot that the pid file support won't be added until 1.8. This
means that to use the start scripts or tup you'd have to have a distinct
ACCUMULO_HOME directory for the second instance (since it greps for that
directory name to determine if a process is already up). Anyway, if you
change the port property Josh specified, you should be able to start the
second tserver. I tried it myself to make sure this time. :-)
On Thu, Dec 10, 2015 at 5:50 AM, Sven Hodapp <sven.hodapp@scai.fraunhofer.de
> wrote:
> Hi Billie,
>
> it seems that in Accumulo 1.7 there is no ACCUMULO_PID_DIR anymore, it is
> generated in the start-server.sh script with ps.
>
> So I've tried this:
>
> $ export ACCUMULO_CONF_DIR="${ACCUMULO_CONF_DIR:$ACCUMULO_HOME/conf2}"
> $ bin/accumulo tserver --address serverIP
>
> This will start the tserver on port of conf2 (currently 29997), but tries
> to open another port which is in use:
>
> Unable to start TServer
> org.apache.thrift.transport.TTransportException: Could not create
> ServerSocket on address serverIP:10002.
>
> I think it is in use by the other tablet server...
>
> Regards,
> Sven
>
> --
> Sven Hodapp, M.Sc.,
> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> Department of Bioinformatics
> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> sven.hodapp@scai.fraunhofer.de
> www.scai.fraunhofer.de
>
> ----- Ursprüngliche Mail -----
> > Von: "Billie Rinaldi" <bi...@gmail.com>
> > An: "user" <us...@accumulo.apache.org>
> > Gesendet: Donnerstag, 10. Dezember 2015 13:33:47
> > Betreff: Re: Multiple talbet servers on the same host
>
> > Ah, that's my fault. I forgot another change you should make. In
> > conf2/accumulo-env.sh, change ACCUMULO_LOG_DIR and ACCUMULO_PID_DIR and
> see
> > if that helps.
> > On Dec 10, 2015 3:29 AM, "Sven Hodapp" <sv...@scai.fraunhofer.de>
> > wrote:
> >
> > Hi Billie,
> >
> > thanks for your reply! I've created a conf2 dir with tserver.port.client
> > set to another port. I've also made shure, that in the environment
> > ACCUMULO_CONF_DIR is set with conf2.
> > But it's said "tablet server already running".
> >
> > I'm using Accumulo 1.7.0, currently on a single node. But because the
> > machine is a big one (48 cores, 256 GB RAM) I'd like to test if multiple
> > tservers on the same machine bring some benefit.
> > Currently the single tablet server yields about 100k writes per second.
> Or
> > may we have a disk-bottleneck?
> >
> > Regards,
> > Sven
> >
> > --
> > Sven Hodapp, M.Sc.,
> > Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> > Department of Bioinformatics
> > Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> > sven.hodapp@scai.fraunhofer.de
> > www.scai.fraunhofer.de
> >
> > ----- Ursprüngliche Mail -----
> >> Von: "Billie Rinaldi" <bi...@gmail.com>
> >> An: "user" <us...@accumulo.apache.org>
> >> Gesendet: Mittwoch, 9. Dezember 2015 18:40:25
> >> Betreff: Re: Multiple talbet servers on the same host
> >
> >> Hi Sven,
> >>
> >> The tablet server port is configured in the accumulo-site.xml file.
> Ports
> >> should not appear in the slaves file. Thus, you would need a separate
> >> configuration directory for Accumulo to be able to start up a second set
> > of
> >> tablet servers on a different port. Let's say you copied your existing
> >> conf directory to a directory named conf2, with the tserver.port.client
> >> property set to a new value in the conf2/accumulo-site.xml file, and
> > pushed
> >> this conf2 directory out to all your nodes. Then you should be able to
> >> start the second set of tablet servers by running
> >> ACCUMULO_CONF_DIR=/path/to/conf2 $ACCUMULO_HOME/bin/tup.sh from one of
> the
> >> nodes.
> >>
> >> That's how you could do this, but I'd encourage you to evaluate whether
> > you
> >> actually need to do this based on your workload, hardware, version of
> >> Accumulo, etc.
> >>
> >> Billie
> >>
> >> On Wed, Dec 9, 2015 at 8:30 AM, Sven Hodapp <
> > sven.hodapp@scai.fraunhofer.de>
> >> wrote:
> >>
> >>> Hi there,
> >>>
> >>> I've read in the Accumulo Book (p. 496) that it should be possible to
> >>> start on a (fat) machine multiple tablet servers to scale (also)
> >>> vertically. Sadly it's not described how to do it. Also I didn't find
> >>> anything about this issue in the official documentation.
> >>>
> >>> I thought it must be configured in ACCUMULO_HOME/conf/slaves.
> >>> So I've added here the same IP with distinct ports multiple times ...
> in
> >>> my naive thinking.
> >>> But *no* additional tablet servers are started.
> >>>
> >>> Or I am completely wrong here?
> >>>
> >>> Regards,
> >>> Sven
> >>>
> >>> --
> >>> Sven Hodapp, M.Sc.,
> >>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> >>> Department of Bioinformatics
> >>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> >>> sven.hodapp@scai.fraunhofer.de
> > >> www.scai.fraunhofer.de
>
Re: Multiple talbet servers on the same host
Posted by Josh Elser <jo...@gmail.com>.
Ahh, there is another port to change for TabletServers. I forgot about
this when I read Billie's initial reply.
You can set replication.receipt.service.port=0 (if you don't care what
port it uses) or set a unique port in conf2. This port is for a service
running on each TabletServer which is part of the data-center
replication feature (new in 1.7).
Sven Hodapp wrote:
> Hi Billie,
>
> it seems that in Accumulo 1.7 there is no ACCUMULO_PID_DIR anymore, it is generated in the start-server.sh script with ps.
>
> So I've tried this:
>
> $ export ACCUMULO_CONF_DIR="${ACCUMULO_CONF_DIR:$ACCUMULO_HOME/conf2}"
> $ bin/accumulo tserver --address serverIP
>
> This will start the tserver on port of conf2 (currently 29997), but tries to open another port which is in use:
>
> Unable to start TServer
> org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address serverIP:10002.
>
> I think it is in use by the other tablet server...
>
> Regards,
> Sven
>
Re: Multiple talbet servers on the same host
Posted by Sven Hodapp <sv...@scai.fraunhofer.de>.
Hi Billie,
it seems that in Accumulo 1.7 there is no ACCUMULO_PID_DIR anymore, it is generated in the start-server.sh script with ps.
So I've tried this:
$ export ACCUMULO_CONF_DIR="${ACCUMULO_CONF_DIR:$ACCUMULO_HOME/conf2}"
$ bin/accumulo tserver --address serverIP
This will start the tserver on port of conf2 (currently 29997), but tries to open another port which is in use:
Unable to start TServer
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address serverIP:10002.
I think it is in use by the other tablet server...
Regards,
Sven
--
Sven Hodapp, M.Sc.,
Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
Department of Bioinformatics
Schloss Birlinghoven, 53754 Sankt Augustin, Germany
sven.hodapp@scai.fraunhofer.de
www.scai.fraunhofer.de
----- Ursprüngliche Mail -----
> Von: "Billie Rinaldi" <bi...@gmail.com>
> An: "user" <us...@accumulo.apache.org>
> Gesendet: Donnerstag, 10. Dezember 2015 13:33:47
> Betreff: Re: Multiple talbet servers on the same host
> Ah, that's my fault. I forgot another change you should make. In
> conf2/accumulo-env.sh, change ACCUMULO_LOG_DIR and ACCUMULO_PID_DIR and see
> if that helps.
> On Dec 10, 2015 3:29 AM, "Sven Hodapp" <sv...@scai.fraunhofer.de>
> wrote:
>
> Hi Billie,
>
> thanks for your reply! I've created a conf2 dir with tserver.port.client
> set to another port. I've also made shure, that in the environment
> ACCUMULO_CONF_DIR is set with conf2.
> But it's said "tablet server already running".
>
> I'm using Accumulo 1.7.0, currently on a single node. But because the
> machine is a big one (48 cores, 256 GB RAM) I'd like to test if multiple
> tservers on the same machine bring some benefit.
> Currently the single tablet server yields about 100k writes per second. Or
> may we have a disk-bottleneck?
>
> Regards,
> Sven
>
> --
> Sven Hodapp, M.Sc.,
> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> Department of Bioinformatics
> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> sven.hodapp@scai.fraunhofer.de
> www.scai.fraunhofer.de
>
> ----- Ursprüngliche Mail -----
>> Von: "Billie Rinaldi" <bi...@gmail.com>
>> An: "user" <us...@accumulo.apache.org>
>> Gesendet: Mittwoch, 9. Dezember 2015 18:40:25
>> Betreff: Re: Multiple talbet servers on the same host
>
>> Hi Sven,
>>
>> The tablet server port is configured in the accumulo-site.xml file. Ports
>> should not appear in the slaves file. Thus, you would need a separate
>> configuration directory for Accumulo to be able to start up a second set
> of
>> tablet servers on a different port. Let's say you copied your existing
>> conf directory to a directory named conf2, with the tserver.port.client
>> property set to a new value in the conf2/accumulo-site.xml file, and
> pushed
>> this conf2 directory out to all your nodes. Then you should be able to
>> start the second set of tablet servers by running
>> ACCUMULO_CONF_DIR=/path/to/conf2 $ACCUMULO_HOME/bin/tup.sh from one of the
>> nodes.
>>
>> That's how you could do this, but I'd encourage you to evaluate whether
> you
>> actually need to do this based on your workload, hardware, version of
>> Accumulo, etc.
>>
>> Billie
>>
>> On Wed, Dec 9, 2015 at 8:30 AM, Sven Hodapp <
> sven.hodapp@scai.fraunhofer.de>
>> wrote:
>>
>>> Hi there,
>>>
>>> I've read in the Accumulo Book (p. 496) that it should be possible to
>>> start on a (fat) machine multiple tablet servers to scale (also)
>>> vertically. Sadly it's not described how to do it. Also I didn't find
>>> anything about this issue in the official documentation.
>>>
>>> I thought it must be configured in ACCUMULO_HOME/conf/slaves.
>>> So I've added here the same IP with distinct ports multiple times ... in
>>> my naive thinking.
>>> But *no* additional tablet servers are started.
>>>
>>> Or I am completely wrong here?
>>>
>>> Regards,
>>> Sven
>>>
>>> --
>>> Sven Hodapp, M.Sc.,
>>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>>> Department of Bioinformatics
>>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>>> sven.hodapp@scai.fraunhofer.de
> >> www.scai.fraunhofer.de
Re: Multiple talbet servers on the same host
Posted by Billie Rinaldi <bi...@gmail.com>.
Ah, that's my fault. I forgot another change you should make. In
conf2/accumulo-env.sh, change ACCUMULO_LOG_DIR and ACCUMULO_PID_DIR and see
if that helps.
On Dec 10, 2015 3:29 AM, "Sven Hodapp" <sv...@scai.fraunhofer.de>
wrote:
Hi Billie,
thanks for your reply! I've created a conf2 dir with tserver.port.client
set to another port. I've also made shure, that in the environment
ACCUMULO_CONF_DIR is set with conf2.
But it's said "tablet server already running".
I'm using Accumulo 1.7.0, currently on a single node. But because the
machine is a big one (48 cores, 256 GB RAM) I'd like to test if multiple
tservers on the same machine bring some benefit.
Currently the single tablet server yields about 100k writes per second. Or
may we have a disk-bottleneck?
Regards,
Sven
--
Sven Hodapp, M.Sc.,
Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
Department of Bioinformatics
Schloss Birlinghoven, 53754 Sankt Augustin, Germany
sven.hodapp@scai.fraunhofer.de
www.scai.fraunhofer.de
----- Ursprüngliche Mail -----
> Von: "Billie Rinaldi" <bi...@gmail.com>
> An: "user" <us...@accumulo.apache.org>
> Gesendet: Mittwoch, 9. Dezember 2015 18:40:25
> Betreff: Re: Multiple talbet servers on the same host
> Hi Sven,
>
> The tablet server port is configured in the accumulo-site.xml file. Ports
> should not appear in the slaves file. Thus, you would need a separate
> configuration directory for Accumulo to be able to start up a second set
of
> tablet servers on a different port. Let's say you copied your existing
> conf directory to a directory named conf2, with the tserver.port.client
> property set to a new value in the conf2/accumulo-site.xml file, and
pushed
> this conf2 directory out to all your nodes. Then you should be able to
> start the second set of tablet servers by running
> ACCUMULO_CONF_DIR=/path/to/conf2 $ACCUMULO_HOME/bin/tup.sh from one of the
> nodes.
>
> That's how you could do this, but I'd encourage you to evaluate whether
you
> actually need to do this based on your workload, hardware, version of
> Accumulo, etc.
>
> Billie
>
> On Wed, Dec 9, 2015 at 8:30 AM, Sven Hodapp <
sven.hodapp@scai.fraunhofer.de>
> wrote:
>
>> Hi there,
>>
>> I've read in the Accumulo Book (p. 496) that it should be possible to
>> start on a (fat) machine multiple tablet servers to scale (also)
>> vertically. Sadly it's not described how to do it. Also I didn't find
>> anything about this issue in the official documentation.
>>
>> I thought it must be configured in ACCUMULO_HOME/conf/slaves.
>> So I've added here the same IP with distinct ports multiple times ... in
>> my naive thinking.
>> But *no* additional tablet servers are started.
>>
>> Or I am completely wrong here?
>>
>> Regards,
>> Sven
>>
>> --
>> Sven Hodapp, M.Sc.,
>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>> Department of Bioinformatics
>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>> sven.hodapp@scai.fraunhofer.de
>> www.scai.fraunhofer.de
Re: Multiple talbet servers on the same host
Posted by Sven Hodapp <sv...@scai.fraunhofer.de>.
Hi Billie,
thanks for your reply! I've created a conf2 dir with tserver.port.client set to another port. I've also made shure, that in the environment ACCUMULO_CONF_DIR is set with conf2.
But it's said "tablet server already running".
I'm using Accumulo 1.7.0, currently on a single node. But because the machine is a big one (48 cores, 256 GB RAM) I'd like to test if multiple tservers on the same machine bring some benefit.
Currently the single tablet server yields about 100k writes per second. Or may we have a disk-bottleneck?
Regards,
Sven
--
Sven Hodapp, M.Sc.,
Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
Department of Bioinformatics
Schloss Birlinghoven, 53754 Sankt Augustin, Germany
sven.hodapp@scai.fraunhofer.de
www.scai.fraunhofer.de
----- Ursprüngliche Mail -----
> Von: "Billie Rinaldi" <bi...@gmail.com>
> An: "user" <us...@accumulo.apache.org>
> Gesendet: Mittwoch, 9. Dezember 2015 18:40:25
> Betreff: Re: Multiple talbet servers on the same host
> Hi Sven,
>
> The tablet server port is configured in the accumulo-site.xml file. Ports
> should not appear in the slaves file. Thus, you would need a separate
> configuration directory for Accumulo to be able to start up a second set of
> tablet servers on a different port. Let's say you copied your existing
> conf directory to a directory named conf2, with the tserver.port.client
> property set to a new value in the conf2/accumulo-site.xml file, and pushed
> this conf2 directory out to all your nodes. Then you should be able to
> start the second set of tablet servers by running
> ACCUMULO_CONF_DIR=/path/to/conf2 $ACCUMULO_HOME/bin/tup.sh from one of the
> nodes.
>
> That's how you could do this, but I'd encourage you to evaluate whether you
> actually need to do this based on your workload, hardware, version of
> Accumulo, etc.
>
> Billie
>
> On Wed, Dec 9, 2015 at 8:30 AM, Sven Hodapp <sv...@scai.fraunhofer.de>
> wrote:
>
>> Hi there,
>>
>> I've read in the Accumulo Book (p. 496) that it should be possible to
>> start on a (fat) machine multiple tablet servers to scale (also)
>> vertically. Sadly it's not described how to do it. Also I didn't find
>> anything about this issue in the official documentation.
>>
>> I thought it must be configured in ACCUMULO_HOME/conf/slaves.
>> So I've added here the same IP with distinct ports multiple times ... in
>> my naive thinking.
>> But *no* additional tablet servers are started.
>>
>> Or I am completely wrong here?
>>
>> Regards,
>> Sven
>>
>> --
>> Sven Hodapp, M.Sc.,
>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>> Department of Bioinformatics
>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>> sven.hodapp@scai.fraunhofer.de
>> www.scai.fraunhofer.de
Re: Multiple talbet servers on the same host
Posted by Billie Rinaldi <bi...@gmail.com>.
Hi Sven,
The tablet server port is configured in the accumulo-site.xml file. Ports
should not appear in the slaves file. Thus, you would need a separate
configuration directory for Accumulo to be able to start up a second set of
tablet servers on a different port. Let's say you copied your existing
conf directory to a directory named conf2, with the tserver.port.client
property set to a new value in the conf2/accumulo-site.xml file, and pushed
this conf2 directory out to all your nodes. Then you should be able to
start the second set of tablet servers by running
ACCUMULO_CONF_DIR=/path/to/conf2 $ACCUMULO_HOME/bin/tup.sh from one of the
nodes.
That's how you could do this, but I'd encourage you to evaluate whether you
actually need to do this based on your workload, hardware, version of
Accumulo, etc.
Billie
On Wed, Dec 9, 2015 at 8:30 AM, Sven Hodapp <sv...@scai.fraunhofer.de>
wrote:
> Hi there,
>
> I've read in the Accumulo Book (p. 496) that it should be possible to
> start on a (fat) machine multiple tablet servers to scale (also)
> vertically. Sadly it's not described how to do it. Also I didn't find
> anything about this issue in the official documentation.
>
> I thought it must be configured in ACCUMULO_HOME/conf/slaves.
> So I've added here the same IP with distinct ports multiple times ... in
> my naive thinking.
> But *no* additional tablet servers are started.
>
> Or I am completely wrong here?
>
> Regards,
> Sven
>
> --
> Sven Hodapp, M.Sc.,
> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> Department of Bioinformatics
> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> sven.hodapp@scai.fraunhofer.de
> www.scai.fraunhofer.de
>