You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Srinath Reddy <ks...@gmail.com> on 2017/02/09 05:13:11 UTC
Error when running nodetool cleanup after adding a new node to a
cluster
Hi,
Trying to re-balacne a Cassandra cluster after adding a new node and I'm getting this error when running nodetool cleanup. The Cassandra cluster is running in a Kubernetes cluster.
Cassandra version is 2.2.8
nodetool cleanup
error: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server. See log for stacktrace.
-- StackTrace --
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server. See log for stacktrace.
at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:119)
at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.244.3.4 6.91 GB 256 60.8% bad1c6c6-8c2e-4f0c-9aea-0d63b451e7a1 rack1
UN 10.244.0.3 6.22 GB 256 60.2% 936cb0c0-d14f-4ddd-bfde-3865b922e267 rack1
UN 10.244.1.3 6.12 GB 256 59.4% 0cb43711-b155-449c-83ba-00ed2a97affe rack1
UN 10.244.4.3 632.43 MB 256 57.8% 55095c75-26df-4180-9004-9fabf88faacc rack1
UN 10.244.2.10 6.08 GB 256 61.8% 32e32bd2-364f-4b6f-b13a-8814164ed160 rack1
Any suggestions on what is needed to re-balance the cluster after adding the new node? I have run nodetool repair but not able to run nodetool cleanup.
Thanks.
Re: Error when running nodetool cleanup after adding a new node to a cluster
Posted by Jerold Kinder <jr...@gmail.com>.
While you folks are solving problems, I'm looking for expertise that sorely
needed at the Architect level that can consult and mentor. Sorry for the
interruption but our need is getting desperate. Thanks for your patience.
Regards
On Fri, Feb 10, 2017 at 2:14 AM, Srinath Reddy <ks...@gmail.com> wrote:
> The nodetool cleanup ran successfully after setting the CLASSPATH
> variable to the kubernetes-cassandra.jar.
>
> Thanks.
>
> On 09-Feb-2017, at 2:23 PM, Srinath Reddy <ks...@gmail.com> wrote:
>
> Alex,
>
> Thanks for reply. I will try the workaround and post an update.
>
> Regards,
>
> Srinath Reddy
>
> On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin <
> oleksandr.shulgin@zalando.de> wrote:
>
> On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy <ks...@gmail.com> wrote:
>
>> Hi,
>>
>> Trying to re-balacne a Cassandra cluster after adding a new node and I'm
>> getting this error when running nodetool cleanup. The Cassandra cluster
>> is running in a Kubernetes cluster.
>>
>> Cassandra version is 2.2.8
>>
>> nodetool cleanup
>> error: io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server. See log for
>> stacktrace.
>> -- StackTrace --
>> org.apache.cassandra.exceptions.ConfigurationException:
>> io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server. See log for
>> stacktrace.
>> at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(D
>> atabaseDescriptor.java:676)
>> at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(Data
>> baseDescriptor.java:119)
>> at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
>> at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(No
>> deProbe.java:262)
>> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
>> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
>> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>>
>
> Hi,
>
> From the above stacktrace it looks like you're hitting the following TODO
> item:
>
> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e
> 9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282
>
> That is, nodetool needs to know concurrent_compactors setting's value
> before starting cleanup, but doesn't use JMX and tries to parse the
> configuration file instead. That fails because your custom SeedProvider
> class is not on classpath for nodetool.
>
> A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be
> found by java when running nodetool script, see https://github.com/apache/
> cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108
>
> Proper fix: get rid of the TODO and really query the value using JMX,
> especially since the latest tick-tock release of Cassandra (3.10) added a
> way to modify it with JMX.
>
> --
> Alex
>
>
>
>
Re: Error when running nodetool cleanup after adding a new node to a
cluster
Posted by Srinath Reddy <ks...@gmail.com>.
The nodetool cleanup ran successfully after setting the CLASSPATH variable to the kubernetes-cassandra.jar.
Thanks.
> On 09-Feb-2017, at 2:23 PM, Srinath Reddy <ks...@gmail.com> wrote:
>
> Alex,
>
> Thanks for reply. I will try the workaround and post an update.
>
> Regards,
>
> Srinath Reddy
>
>> On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin <oleksandr.shulgin@zalando.de <ma...@zalando.de>> wrote:
>>
>> On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy <ksreddy@gmail.com <ma...@gmail.com>> wrote:
>> Hi,
>>
>> Trying to re-balacne a Cassandra cluster after adding a new node and I'm getting this error when running nodetool cleanup. The Cassandra cluster is running in a Kubernetes cluster.
>>
>> Cassandra version is 2.2.8
>>
>> nodetool cleanup
>> error: io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server. See log for stacktrace.
>> -- StackTrace --
>> org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server. See log for stacktrace.
>> at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
>> at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:119)
>> at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
>> at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
>> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
>> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
>> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>>
>> Hi,
>>
>> From the above stacktrace it looks like you're hitting the following TODO item:
>>
>> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282 <https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282>
>>
>> That is, nodetool needs to know concurrent_compactors setting's value before starting cleanup, but doesn't use JMX and tries to parse the configuration file instead. That fails because your custom SeedProvider class is not on classpath for nodetool.
>>
>> A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be found by java when running nodetool script, see https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108 <https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108>
>>
>> Proper fix: get rid of the TODO and really query the value using JMX, especially since the latest tick-tock release of Cassandra (3.10) added a way to modify it with JMX.
>>
>> --
>> Alex
>
Re: Error when running nodetool cleanup after adding a new node to a
cluster
Posted by Srinath Reddy <ks...@gmail.com>.
Alex,
Thanks for reply. I will try the workaround and post an update.
Regards,
Srinath Reddy
> On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin <ol...@zalando.de> wrote:
>
> On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy <ksreddy@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> Trying to re-balacne a Cassandra cluster after adding a new node and I'm getting this error when running nodetool cleanup. The Cassandra cluster is running in a Kubernetes cluster.
>
> Cassandra version is 2.2.8
>
> nodetool cleanup
> error: io.k8s.cassandra.KubernetesSeedProvider
> Fatal configuration error; unable to start server. See log for stacktrace.
> -- StackTrace --
> org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
> Fatal configuration error; unable to start server. See log for stacktrace.
> at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
> at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:119)
> at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
> at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>
> Hi,
>
> From the above stacktrace it looks like you're hitting the following TODO item:
>
> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282 <https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282>
>
> That is, nodetool needs to know concurrent_compactors setting's value before starting cleanup, but doesn't use JMX and tries to parse the configuration file instead. That fails because your custom SeedProvider class is not on classpath for nodetool.
>
> A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be found by java when running nodetool script, see https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108 <https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108>
>
> Proper fix: get rid of the TODO and really query the value using JMX, especially since the latest tick-tock release of Cassandra (3.10) added a way to modify it with JMX.
>
> --
> Alex
Re: Error when running nodetool cleanup after adding a new node to a cluster
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy <ks...@gmail.com> wrote:
> Hi,
>
> Trying to re-balacne a Cassandra cluster after adding a new node and I'm
> getting this error when running nodetool cleanup. The Cassandra cluster
> is running in a Kubernetes cluster.
>
> Cassandra version is 2.2.8
>
> nodetool cleanup
> error: io.k8s.cassandra.KubernetesSeedProvider
> Fatal configuration error; unable to start server. See log for stacktrace.
> -- StackTrace --
> org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.
> KubernetesSeedProvider
> Fatal configuration error; unable to start server. See log for stacktrace.
> at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(
> DatabaseDescriptor.java:676)
> at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(
> DatabaseDescriptor.java:119)
> at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
> at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(
> NodeProbe.java:262)
> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>
Hi,
From the above stacktrace it looks like you're hitting the following TODO
item:
https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282
That is, nodetool needs to know concurrent_compactors setting's value
before starting cleanup, but doesn't use JMX and tries to parse the
configuration file instead. That fails because your custom SeedProvider
class is not on classpath for nodetool.
A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be
found by java when running nodetool script, see
https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108
Proper fix: get rid of the TODO and really query the value using JMX,
especially since the latest tick-tock release of Cassandra (3.10) added a
way to modify it with JMX.
--
Alex
Re: Error when running nodetool cleanup after adding a new node to a
cluster
Posted by Srinath Reddy <ks...@gmail.com>.
Yes, I ran the nodetool cleanup on the other nodes and got the error.
Thanks.
> On 09-Feb-2017, at 11:12 AM, Harikrishnan Pillai <HP...@walmartlabs.com> wrote:
>
> The cleanup has to run on other nodes
>
> Sent from my iPhone
>
> On Feb 8, 2017, at 9:14 PM, Srinath Reddy <ksreddy@gmail.com <ma...@gmail.com>> wrote:
>
>> Hi,
>>
>> Trying to re-balacne a Cassandra cluster after adding a new node and I'm getting this error when running nodetool cleanup. The Cassandra cluster is running in a Kubernetes cluster.
>>
>> Cassandra version is 2.2.8
>>
>> nodetool cleanup
>> error: io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server. See log for stacktrace.
>> -- StackTrace --
>> org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server. See log for stacktrace.
>> at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
>> at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:119)
>> at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
>> at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
>> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
>> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
>> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>>
>> nodetool status
>> Datacenter: datacenter1
>> =======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID Rack
>> UN 10.244.3.4 6.91 GB 256 60.8% bad1c6c6-8c2e-4f0c-9aea-0d63b451e7a1 rack1
>> UN 10.244.0.3 6.22 GB 256 60.2% 936cb0c0-d14f-4ddd-bfde-3865b922e267 rack1
>> UN 10.244.1.3 6.12 GB 256 59.4% 0cb43711-b155-449c-83ba-00ed2a97affe rack1
>> UN 10.244.4.3 632.43 MB 256 57.8% 55095c75-26df-4180-9004-9fabf88faacc rack1
>> UN 10.244.2.10 6.08 GB 256 61.8% 32e32bd2-364f-4b6f-b13a-8814164ed160 rack1
>>
>>
>> Any suggestions on what is needed to re-balance the cluster after adding the new node? I have run nodetool repair but not able to run nodetool cleanup.
>>
>> Thanks.
>>
>>
Re: Error when running nodetool cleanup after adding a new node to a
cluster
Posted by Harikrishnan Pillai <HP...@walmartlabs.com>.
The cleanup has to run on other nodes
Sent from my iPhone
On Feb 8, 2017, at 9:14 PM, Srinath Reddy <ks...@gmail.com>> wrote:
Hi,
Trying to re-balacne a Cassandra cluster after adding a new node and I'm getting this error when running nodetool cleanup. The Cassandra cluster is running in a Kubernetes cluster.
Cassandra version is 2.2.8
nodetool cleanup
error: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server. See log for stacktrace.
-- StackTrace --
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server. See log for stacktrace.
at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:119)
at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.244.3.4 6.91 GB 256 60.8% bad1c6c6-8c2e-4f0c-9aea-0d63b451e7a1 rack1
UN 10.244.0.3 6.22 GB 256 60.2% 936cb0c0-d14f-4ddd-bfde-3865b922e267 rack1
UN 10.244.1.3 6.12 GB 256 59.4% 0cb43711-b155-449c-83ba-00ed2a97affe rack1
UN 10.244.4.3 632.43 MB 256 57.8% 55095c75-26df-4180-9004-9fabf88faacc rack1
UN 10.244.2.10 6.08 GB 256 61.8% 32e32bd2-364f-4b6f-b13a-8814164ed160 rack1
Any suggestions on what is needed to re-balance the cluster after adding the new node? I have run nodetool repair but not able to run nodetool cleanup.
Thanks.