You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "roman.drapeko@baesystems.com" <ro...@baesystems.com> on 2015/06/02 19:36:35 UTC
AWS cluster script - abnormal behavior
Hi there,
I was going to do some performance tests to evaluate the technology on AWS.
Followed https://blogs.aws.amazon.com/bigdata/post/Tx15973X6QHUM43/Running-Apache-Accumulo-on-Amazon-EMR
The exact command I am using:
aws emr create-cluster --name Accumulo --no-auto-terminate --bootstrap-actions Path=s3://elasticmapreduce.bootstrapactions/accumulo/1.6.1/install-accumulo_mj,Name=Install_Accumulo --ami-version 3.3.1 --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge InstanceGroupType=CORE,InstanceCount=3,InstanceType=m3.xlarge --ec2-attributes KeyName=elastic_search_test --use-default-roles
After that I can access monitor web-interface and see that everything good, however I am not able to access shell with root/secret and any other possible combination. I receive - "Invalid credentials". What is really interesting - 1 attempt out of 10 was successful and everything worked as expected. This fact really confuses me.
./stop-all.sh does not work without ctrl+c
When I do ./start-all.sh I can see many logs similar to the below.
Script that is used behind the scenes is this one: https://github.com/awslabs/emr-bootstrap-actions/blob/master/accumulo/1.6.1/install-accumulo_mj
Any idea why I observe this behaviour?
Many thanks,
Roman
==> tracer_ip-172-31-47-228.eu-west-1.compute.internal.debug.log <==
2015-06-02 17:25:13,829 [tracer.TraceServer] INFO : Waiting to checking/create the trace table.
org.apache.accumulo.core.client.AccumuloSecurityException: Error BAD_CREDENTIALS for user root - Username or Password is Invalid
at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:63)
at org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:70)
at org.apache.accumulo.server.client.HdfsZooInstance.getConnector(HdfsZooInstance.java:154)
at org.apache.accumulo.tracer.TraceServer.<init>(TraceServer.java:199)
at org.apache.accumulo.tracer.TraceServer.main(TraceServer.java:294)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.accumulo.start.Main$1.run(Main.java:141)
at java.lang.Thread.run(Thread.java:745)
Caused by: ThriftSecurityException(user:root, code:BAD_CREDENTIALS)
at org.apache.accumulo.core.client.impl.thrift.ClientService$authenticate_result$authenticate_resultStandardScheme.read(ClientService.java:13855)
at org.apache.accumulo.core.client.impl.thrift.ClientService$authen
Please consider the environment before printing this email. This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately. Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies under the control of BAE Systems Applied Intelligence Limited, details of which can be found at http://www.baesystems.com/Businesses/index.htm.
Re: AWS cluster script - abnormal behavior
Posted by Josh Elser <jo...@gmail.com>.
Roman,
Cool to hear you're trying to take Accumulo for a spin.
The tracer failures are likely due to incorrect configuration. The
tracer uses a normal user account to connect and write traces to Accumulo.
This configuration is done by setting trace.user and
trace.token.property.password in accumulo-site.xml. This user account
needs to exist and have permissions to write to the value of trace.table
("trace" by default).
I'm not sure what to tell you about being unable to log in from your
client as I have no idea what this EMR installation is doing. The fact
that 1 out of 10 was actually successful sounds like there is an issue
in their implementation. If you can provide server logs, we might be
able to help you get to the bottom of it.
./stop-all.sh will wait for the instance to gracefully stop. If Accumulo
is not healthy, as it sounds like it isn't if you can't properly
authenticate yourself, this will likely take a long time (if ever
complete). The ctrl-c just short-circuits this wait and takes down the
system hard. What does the monitor say it's doing before you ctrl-c the
stop-all.sh?
Does anyone from Amazon watch this list? Does anyone know who actually
worked on this at Amazon that might be able to help?
- Josh
roman.drapeko@baesystems.com wrote:
> Hi there,
>
> I was going to do some performance tests to evaluate the technology on AWS.
>
> Followed
> https://blogs.aws.amazon.com/bigdata/post/Tx15973X6QHUM43/Running-Apache-Accumulo-on-Amazon-EMR
>
>
> The exact command I am using:
>
> aws emr create-cluster --name Accumulo --no-auto-terminate
> --bootstrap-actions
> Path=s3://elasticmapreduce.bootstrapactions/accumulo/1.6.1/install-accumulo_mj,Name=Install_Accumulo
> --ami-version 3.3.1 --instance-groups
> InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge
> InstanceGroupType=CORE,InstanceCount=3,InstanceType=m3.xlarge
> --ec2-attributes KeyName=elastic_search_test --use-default-roles
>
> After that I can access monitor web-interface and see that everything
> good, however I am not able to access shell with root/secret and any
> other possible combination. I receive – “Invalid credentials”. What is
> really interesting – 1 attempt out of 10 was successful and everything
> worked as expected. This fact really confuses me.
>
> ./stop-all.sh does not work without ctrl+c
>
> When I do ./start-all.sh I can see many logs similar to the below.
>
> Script that is used behind the scenes is this one:
> https://github.com/awslabs/emr-bootstrap-actions/blob/master/accumulo/1.6.1/install-accumulo_mj
>
> Any idea why I observe this behaviour?
>
> Many thanks,
>
> Roman
>
> ==> tracer_ip-172-31-47-228.eu-west-1.compute.internal.debug.log <==
>
> 2015-06-02 17:25:13,829 [tracer.TraceServer] INFO : Waiting to
> checking/create the trace table.
>
> org.apache.accumulo.core.client.AccumuloSecurityException: Error
> BAD_CREDENTIALS for user root - Username or Password is Invalid
>
> at
> org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:63)
>
> at
> org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:70)
>
> at
> org.apache.accumulo.server.client.HdfsZooInstance.getConnector(HdfsZooInstance.java:154)
>
> at org.apache.accumulo.tracer.TraceServer.<init>(TraceServer.java:199)
>
> at org.apache.accumulo.tracer.TraceServer.main(TraceServer.java:294)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.accumulo.start.Main$1.run(Main.java:141)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: ThriftSecurityException(user:root, code:BAD_CREDENTIALS)
>
> at
> org.apache.accumulo.core.client.impl.thrift.ClientService$authenticate_result$authenticate_resultStandardScheme.read(ClientService.java:13855)
>
> at org.apache.accumulo.core.client.impl.thrift.ClientService$authen
>
> Please consider the environment before printing this email. This message
> should be regarded as confidential. If you have received this email in
> error please notify the sender and destroy it immediately. Statements of
> intent shall only become binding when confirmed in hard copy by an
> authorised signatory. The contents of this email may relate to dealings
> with other companies under the control of BAE Systems Applied
> Intelligence Limited, details of which can be found at
> http://www.baesystems.com/Businesses/index.htm.
Re: AWS cluster script - abnormal behavior
Posted by Josef Roehrl - PHEMI <jr...@phemi.com>.
Hi All,
We've done quite a bit of Accumulo on AWS.
The first problem you see is that the users can't be created if HDFS is
not up on time. We've seen this more often when creating the trace user
rather than root.
If you can post output from the script, that would be helpful.
=josef
On 15-06-02 10:36 AM, roman.drapeko@baesystems.com wrote:
>
> Hi there,
>
> I was going to do some performance tests to evaluate the technology on
> AWS.
>
> Followed
> https://blogs.aws.amazon.com/bigdata/post/Tx15973X6QHUM43/Running-Apache-Accumulo-on-Amazon-EMR
>
>
> The exact command I am using:
>
> aws emr create-cluster --name Accumulo --no-auto-terminate
> --bootstrap-actions
> Path=s3://elasticmapreduce.bootstrapactions/accumulo/1.6.1/install-accumulo_mj,Name=Install_Accumulo
> --ami-version 3.3.1 --instance-groups
> InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge
> InstanceGroupType=CORE,InstanceCount=3,InstanceType=m3.xlarge
> --ec2-attributes KeyName=elastic_search_test --use-default-roles
>
> After that I can access monitor web-interface and see that everything
> good, however I am not able to access shell with root/secret and any
> other possible combination. I receive – “Invalid credentials”. What is
> really interesting – 1 attempt out of 10 was successful and everything
> worked as expected. This fact really confuses me.
>
> ./stop-all.sh does not work without ctrl+c
>
> When I do ./start-all.sh I can see many logs similar to the below.
>
> Script that is used behind the scenes is this one:
> https://github.com/awslabs/emr-bootstrap-actions/blob/master/accumulo/1.6.1/install-accumulo_mj
>
> Any idea why I observe this behaviour?
>
> Many thanks,
>
> Roman
>
> ==> tracer_ip-172-31-47-228.eu-west-1.compute.internal.debug.log <==
>
> 2015-06-02 17:25:13,829 [tracer.TraceServer] INFO : Waiting to
> checking/create the trace table.
>
> org.apache.accumulo.core.client.AccumuloSecurityException: Error
> BAD_CREDENTIALS for user root - Username or Password is Invalid
>
> at
> org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:63)
>
> at
> org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:70)
>
> at
> org.apache.accumulo.server.client.HdfsZooInstance.getConnector(HdfsZooInstance.java:154)
>
> at org.apache.accumulo.tracer.TraceServer.<init>(TraceServer.java:199)
>
> at org.apache.accumulo.tracer.TraceServer.main(TraceServer.java:294)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.accumulo.start.Main$1.run(Main.java:141)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: ThriftSecurityException(user:root, code:BAD_CREDENTIALS)
>
> at
> org.apache.accumulo.core.client.impl.thrift.ClientService$authenticate_result$authenticate_resultStandardScheme.read(ClientService.java:13855)
>
> at org.apache.accumulo.core.client.impl.thrift.ClientService$authen
>
> Please consider the environment before printing this email. This
> message should be regarded as confidential. If you have received this
> email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard
> copy by an authorised signatory. The contents of this email may relate
> to dealings with other companies under the control of BAE Systems
> Applied Intelligence Limited, details of which can be found at
> http://www.baesystems.com/Businesses/index.htm.
--
>
> Josef Roehrl
> Software Developer
> *PHEMI Health Systems*
> 180-887 Great Northern Way
> Vancouver, BC V5T 4T5
> 604-336-1119 <tel:604-336-1119>
> Website <http://www.phemi.com/> Twitter
> <https://twitter.com/PHEMISystems> Linkedin
> <http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId%3A1403279580554%2Ctas%3Aphemi%20hea%2Cidx%3A1-1-1>
>