You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by Jesus arteche <ch...@gmail.com> on 2014/06/27 12:39:36 UTC
Error on Wide Area Cassandra Cluster on Softlayer example
Hey guys,
I'm trying to run the WideArea Cassandra cluster, deploying over 2
different DC on SoftLayer. The machines spun up in SoftLayer, but I got
this error :
*Failure running task post-start (IthjWEwo)
<http://5.153.54.164:8081/#/v1/applications/uoQlEEkk/entities/cnzhYNme/activities/subtask/IthjWEwo>:
*Software
process entity CassandraNodeImpl{id=cnzhYNme} did not pass is-running check
within the required 3m limit (3m 5s elapsed)
Any idea?
Re: Error on Wide Area Cassandra Cluster on Softlayer example
Posted by Andrew Kennedy <an...@cloudsoftcorp.com>.
Jesus, Hi.
Actually, I forgot about the other config key, for the download location
URL; it seems that not every old version is available on the Apache mirror
download sites. So, you need to have this, as well as the 'version' key,
added to your configuration:
key=mirrorUrl
value=http://archive.apache.org/dist/cassandra
My config section now has these values:
"config": {
"cassandra.cluster.initialSize": "2",
"version": "1.2.12",
"mirrorUrl": "http://archive.apache.org/dist/cassandra"
}
Obviously we are going to have to fix the underlying issue with the Snitch
API as well, and I will get back to you on that shortly.
Cheers,
Andrew.
--
-- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;
On 29 June 2014 19:49, Jesus arteche <ch...@gmail.com> wrote:
> hey Andrew,
>
> I´m launching the cluster from the web console, I added in the add
> configuration dialog
>
> key=version
> value=1.2.12
>
> But I got this error:
>
> Execution failed, invalid result 9 for installing
> CassandraNodeImpl{id=FuG0vn9f}
>
> Am I doing it right?
>
>
> On Sat, Jun 28, 2014 at 4:50 PM, Andrew Kennedy <
> andrew.kennedy@cloudsoftcorp.com> wrote:
>
>> I raised a new JIRA issue BROOKLYN-11 to track this.
>>
>> - https://issues.apache.org/jira/browse/BROOKLYN-11
>>
>> Aled, if you think you know how to fix the snitch code can you make a
>> comment or assign yourself to the issue, please?
>>
>> Andrew.
>>
>> --
>> -- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;
>>
>>
>> On 28 June 2014 16:23, Andrew Kennedy <an...@cloudsoftcorp.com>
>> wrote:
>>
>>> Hi.
>>>
>>> I'm not sure how you're deploying this (YAML blueprint, or via the
>>> Catalog view in the Brooklyn console?) but if you add a 'version: 1.2.12'
>>> configuration entry to the YAML spec or add it in the Catalog configuration
>>> dialog, that should change the Cassandra version back to the last that
>>> contained the method in question. How did you start the cluster? I'm going
>>> to give it a try myself, as well, and report back on the best way to do
>>> this.
>>>
>>> The problem is that I'm not a Cassandra expert, so I'm not sure what the
>>> effect of the API changes after Cassandra 1.2.13 are.We are using 1.2.16 in
>>> the CassandraNode entity explicitly, but I don't know how to re-write the
>>> snitch code to be compatible with this version, which is what will be
>>> required to fix this. I have copied in @aledsage, who may be able to help.
>>>
>>> I see from comments in our Cassandra entities that a switch to the 2.x
>>> API has been proposed as well, but I don't think this has been progressed
>>> very far yet.
>>>
>>> Andrew.
>>>
>>> --
>>> -- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;
>>>
>>>
>>> On 28 June 2014 15:29, Jesus arteche <ch...@gmail.com> wrote:
>>>
>>>> well...does it mean that if I use the previous Brooklyn version
>>>> 6...will it
>>>> work?
>>>>
>>>>
>>>> On Sat, Jun 28, 2014 at 3:25 PM, Jesus arteche <ch...@gmail.com>
>>>> wrote:
>>>>
>>>> > dont have a clue about how to do it...if you guide me ...I can do it
>>>> >
>>>> >
>>>> > On Sat, Jun 28, 2014 at 2:47 PM, Andrew Kennedy <
>>>> > andrew.kennedy@cloudsoftcorp.com> wrote:
>>>> >
>>>> >> On 28 June 2014 14:24, Jesus arteche <ch...@gmail.com> wrote:
>>>> >>
>>>> >> > Caused by: java.lang.NoSuchMethodError:
>>>> >> > org.apache.cassandra.locator.TokenMetadata.invalidateCaches()V
>>>> >> >
>>>> >>
>>>> >> Well, it looks like this is the culprit. It seems that the
>>>> >> `invalidateCaches()` method was suddenly removed in version 1.2.13
>>>> without
>>>> >> ever being deprecated, cassandra-all-1.2.12 is the last version that
>>>> >> contains it:
>>>> >>
>>>> >> -
>>>> >>
>>>> >>
>>>> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/1.2.12/org/apache/cassandra/locator/TokenMetadata.java#TokenMetadata.invalidateCaches%28%29
>>>> >>
>>>> >> So, we'll need to take a look at our Cassandra dependencies and fix
>>>> this.
>>>> >> Have you created an issue in JIRA about this yet?
>>>> >>
>>>> >> Andrew.
>>>> >> --
>>>> >> -- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/
>>>> ;
>>>> >>
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
Re: Error on Wide Area Cassandra Cluster on Softlayer example
Posted by Andrew Kennedy <an...@cloudsoftcorp.com>.
Hi.
I'm not sure how you're deploying this (YAML blueprint, or via the Catalog
view in the Brooklyn console?) but if you add a 'version: 1.2.12'
configuration entry to the YAML spec or add it in the Catalog configuration
dialog, that should change the Cassandra version back to the last that
contained the method in question. How did you start the cluster? I'm going
to give it a try myself, as well, and report back on the best way to do
this.
The problem is that I'm not a Cassandra expert, so I'm not sure what the
effect of the API changes after Cassandra 1.2.13 are.We are using 1.2.16 in
the CassandraNode entity explicitly, but I don't know how to re-write the
snitch code to be compatible with this version, which is what will be
required to fix this. I have copied in @aledsage, who may be able to help.
I see from comments in our Cassandra entities that a switch to the 2.x API
has been proposed as well, but I don't think this has been progressed very
far yet.
Andrew.
--
-- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;
On 28 June 2014 15:29, Jesus arteche <ch...@gmail.com> wrote:
> well...does it mean that if I use the previous Brooklyn version 6...will it
> work?
>
>
> On Sat, Jun 28, 2014 at 3:25 PM, Jesus arteche <ch...@gmail.com>
> wrote:
>
> > dont have a clue about how to do it...if you guide me ...I can do it
> >
> >
> > On Sat, Jun 28, 2014 at 2:47 PM, Andrew Kennedy <
> > andrew.kennedy@cloudsoftcorp.com> wrote:
> >
> >> On 28 June 2014 14:24, Jesus arteche <ch...@gmail.com> wrote:
> >>
> >> > Caused by: java.lang.NoSuchMethodError:
> >> > org.apache.cassandra.locator.TokenMetadata.invalidateCaches()V
> >> >
> >>
> >> Well, it looks like this is the culprit. It seems that the
> >> `invalidateCaches()` method was suddenly removed in version 1.2.13
> without
> >> ever being deprecated, cassandra-all-1.2.12 is the last version that
> >> contains it:
> >>
> >> -
> >>
> >>
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/1.2.12/org/apache/cassandra/locator/TokenMetadata.java#TokenMetadata.invalidateCaches%28%29
> >>
> >> So, we'll need to take a look at our Cassandra dependencies and fix
> this.
> >> Have you created an issue in JIRA about this yet?
> >>
> >> Andrew.
> >> --
> >> -- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;
> >>
> >
> >
>
Re: Error on Wide Area Cassandra Cluster on Softlayer example
Posted by Jesus arteche <ch...@gmail.com>.
well...does it mean that if I use the previous Brooklyn version 6...will it
work?
On Sat, Jun 28, 2014 at 3:25 PM, Jesus arteche <ch...@gmail.com>
wrote:
> dont have a clue about how to do it...if you guide me ...I can do it
>
>
> On Sat, Jun 28, 2014 at 2:47 PM, Andrew Kennedy <
> andrew.kennedy@cloudsoftcorp.com> wrote:
>
>> On 28 June 2014 14:24, Jesus arteche <ch...@gmail.com> wrote:
>>
>> > Caused by: java.lang.NoSuchMethodError:
>> > org.apache.cassandra.locator.TokenMetadata.invalidateCaches()V
>> >
>>
>> Well, it looks like this is the culprit. It seems that the
>> `invalidateCaches()` method was suddenly removed in version 1.2.13 without
>> ever being deprecated, cassandra-all-1.2.12 is the last version that
>> contains it:
>>
>> -
>>
>> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/1.2.12/org/apache/cassandra/locator/TokenMetadata.java#TokenMetadata.invalidateCaches%28%29
>>
>> So, we'll need to take a look at our Cassandra dependencies and fix this.
>> Have you created an issue in JIRA about this yet?
>>
>> Andrew.
>> --
>> -- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;
>>
>
>
Re: Error on Wide Area Cassandra Cluster on Softlayer example
Posted by Jesus arteche <ch...@gmail.com>.
dont have a clue about how to do it...if you guide me ...I can do it
On Sat, Jun 28, 2014 at 2:47 PM, Andrew Kennedy <
andrew.kennedy@cloudsoftcorp.com> wrote:
> On 28 June 2014 14:24, Jesus arteche <ch...@gmail.com> wrote:
>
> > Caused by: java.lang.NoSuchMethodError:
> > org.apache.cassandra.locator.TokenMetadata.invalidateCaches()V
> >
>
> Well, it looks like this is the culprit. It seems that the
> `invalidateCaches()` method was suddenly removed in version 1.2.13 without
> ever being deprecated, cassandra-all-1.2.12 is the last version that
> contains it:
>
> -
>
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/1.2.12/org/apache/cassandra/locator/TokenMetadata.java#TokenMetadata.invalidateCaches%28%29
>
> So, we'll need to take a look at our Cassandra dependencies and fix this.
> Have you created an issue in JIRA about this yet?
>
> Andrew.
> --
> -- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;
>
Re: Error on Wide Area Cassandra Cluster on Softlayer example
Posted by Andrew Kennedy <an...@cloudsoftcorp.com>.
On 28 June 2014 14:24, Jesus arteche <ch...@gmail.com> wrote:
> Caused by: java.lang.NoSuchMethodError:
> org.apache.cassandra.locator.TokenMetadata.invalidateCaches()V
>
Well, it looks like this is the culprit. It seems that the
`invalidateCaches()` method was suddenly removed in version 1.2.13 without
ever being deprecated, cassandra-all-1.2.12 is the last version that
contains it:
-
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/1.2.12/org/apache/cassandra/locator/TokenMetadata.java#TokenMetadata.invalidateCaches%28%29
So, we'll need to take a look at our Cassandra dependencies and fix this.
Have you created an issue in JIRA about this yet?
Andrew.
--
-- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;
Re: Error on Wide Area Cassandra Cluster on Softlayer example
Posted by Jesus arteche <ch...@gmail.com>.
thanks Aled,
I got this from the Cassandra log in the node (well, all nodes failed :( ,
so I guess the error is the same on all):
INFO [main] 2014-06-26 18:41:31,577 CassandraDaemon.java (line 119) Logging
initialized
INFO [main] 2014-06-26 18:41:31,596 CassandraDaemon.java (line 144) JVM
vendor/version: OpenJDK 64-Bit Server VM/1.7.0_55
WARN [main] 2014-06-26 18:41:31,597 CassandraDaemon.java (line 149)
OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO [main] 2014-06-26 18:41:31,597 CassandraDaemon.java (line 182) Heap
size: 508559360/509607936
INFO [main] 2014-06-26 18:41:31,597 CassandraDaemon.java (line 183)
Classpath:
/root/brooklyn-managed-processes/apps/uoQlEEkk/entities/CassandraNode_d3g5BrUs/conf:./bin/../build$
INFO [main] 2014-06-26 18:41:31,599 CLibrary.java (line 65) JNA not found.
Native methods will be disabled.
INFO [main] 2014-06-26 18:41:31,613 DatabaseDescriptor.java (line 141)
Loading settings from
file:/root/brooklyn-managed-processes/apps/uoQlEEkk/entities/CassandraNode_d3g5BrUs/$
INFO [main] 2014-06-26 18:41:32,239 DatabaseDescriptor.java (line 159)
Data files directories:
[/root/brooklyn-managed-processes/apps/uoQlEEkk/entities/CassandraNode_d3g5BrUs/da$
INFO [main] 2014-06-26 18:41:32,240 DatabaseDescriptor.java (line 160)
Commit log directory:
/root/brooklyn-managed-processes/apps/uoQlEEkk/entities/CassandraNode_d3g5BrUs/commi$
INFO [main] 2014-06-26 18:41:32,240 DatabaseDescriptor.java (line 200)
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2014-06-26 18:41:32,240 DatabaseDescriptor.java (line 214)
disk_failure_policy is stop
INFO [main] 2014-06-26 18:41:32,247 DatabaseDescriptor.java (line 290)
Global memtable threshold is enabled at 162MB
ERROR [main] 2014-06-26 18:41:32,411 DatabaseDescriptor.java (line 528)
Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Error instantiating
snitch class
'brooklyn.entity.nosql.cassandra.customsnitch.MultiCloudSnitch'.
at
org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:465)
at
org.apache.cassandra.config.DatabaseDescriptor.createEndpointSnitch(DatabaseDescriptor.java:544)
at
org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:361)
at
org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:132)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:447)
... 6 more
Caused by: java.lang.NoSuchMethodError:
org.apache.cassandra.locator.TokenMetadata.invalidateCaches()V
at
brooklyn.entity.nosql.cassandra.customsnitch.MultiCloudSnitch.reloadConfiguration(MultiCloudSnitch.java:138)
at
brooklyn.entity.nosql.cassandra.customsnitch.MultiCloudSnitch.<init>(MultiCloudSnitch.java:78)
... 11 more
Thanks in advance
On Fri, Jun 27, 2014 at 3:17 PM, Aled Sage <al...@gmail.com> wrote:
> Hi,
>
> The service-up is based on JMX connectivity to the cassandra node.
>
> You need to drill into the startup tasks, and the debug log, to see why it
> might have failed.
>
> In the Brooklyn web-console, you can select the failed CassandraNode
> entity in the tree (under "applications"), then click on the Activity tab
> and select the "start" row. You can then drill into the tasks performed to
> see the actual stdout/stdin/stderr of the ssh command for starting the
> Cassandra Node.
>
> In the brooklyn.debug.log file, you can look for any exceptions (as well
> as the stdout/stderr of the commands executed).
>
> If the failed CassandraNode has gone from the web-console then you'll want
> to turn on "quarantine", which tells Brooklyn to not terminate the node(s)
> that fail to start - see DynamicCluster.QUARANTINE_FAILED_ENTITIES.
> Add the configuration "dynamiccluster.quarantineFailedEntities" set to
> true in your blueprint.
>
> If it is not obvious from stdout/stdin/stderr what is wrong, then you'll
> want to login to the CassandraNode VM to see what is in the cassandra log
> file.
> The normal "cassandra.log" file will tell you why that particular node
> didn't startup.
> There is a "log.location" sensor on the CassandraNode that should give you
> the path of the log file on the VM.
>
> Aled
>
>
>
> On 27/06/2014 11:39, Jesus arteche wrote:
>
>> Hey guys,
>>
>> I'm trying to run the WideArea Cassandra cluster, deploying over 2
>> different DC on SoftLayer. The machines spun up in SoftLayer, but I got
>> this error :
>>
>> *Failure running task post-start (IthjWEwo)
>> <http://5.153.54.164:8081/#/v1/applications/uoQlEEkk/
>> entities/cnzhYNme/activities/subtask/IthjWEwo>:
>> *Software
>>
>> process entity CassandraNodeImpl{id=cnzhYNme} did not pass is-running
>> check
>> within the required 3m limit (3m 5s elapsed)
>>
>> Any idea?
>>
>>
>
Re: Error on Wide Area Cassandra Cluster on Softlayer example
Posted by Aled Sage <al...@gmail.com>.
Hi,
The service-up is based on JMX connectivity to the cassandra node.
You need to drill into the startup tasks, and the debug log, to see why
it might have failed.
In the Brooklyn web-console, you can select the failed CassandraNode
entity in the tree (under "applications"), then click on the Activity
tab and select the "start" row. You can then drill into the tasks
performed to see the actual stdout/stdin/stderr of the ssh command for
starting the Cassandra Node.
In the brooklyn.debug.log file, you can look for any exceptions (as well
as the stdout/stderr of the commands executed).
If the failed CassandraNode has gone from the web-console then you'll
want to turn on "quarantine", which tells Brooklyn to not terminate the
node(s) that fail to start - see DynamicCluster.QUARANTINE_FAILED_ENTITIES.
Add the configuration "dynamiccluster.quarantineFailedEntities" set to
true in your blueprint.
If it is not obvious from stdout/stdin/stderr what is wrong, then you'll
want to login to the CassandraNode VM to see what is in the cassandra
log file.
The normal "cassandra.log" file will tell you why that particular node
didn't startup.
There is a "log.location" sensor on the CassandraNode that should give
you the path of the log file on the VM.
Aled
On 27/06/2014 11:39, Jesus arteche wrote:
> Hey guys,
>
> I'm trying to run the WideArea Cassandra cluster, deploying over 2
> different DC on SoftLayer. The machines spun up in SoftLayer, but I got
> this error :
>
> *Failure running task post-start (IthjWEwo)
> <http://5.153.54.164:8081/#/v1/applications/uoQlEEkk/entities/cnzhYNme/activities/subtask/IthjWEwo>:
> *Software
> process entity CassandraNodeImpl{id=cnzhYNme} did not pass is-running check
> within the required 3m limit (3m 5s elapsed)
>
> Any idea?
>