You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Xu (Simon) Chen" <xc...@gmail.com> on 2015/06/11 23:00:36 UTC
same token couldn't authenticate twice?
Folks,
I am working on geomesa+accumulo+spark integration. For some reason, I
found that the same token cannot be used to authenticate twice.
The workflow is that geomesa would try to create a hadoop rdd, during
which it tries to create an AccumuloDataStore:
https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
During this process, a ZooKeeperInstance is created:
https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
I modified geomesa such that it would use kerberos to authenticate
here. This step works fine.
But next, geomesa calls ConfigurationBase.setConnectorInfo:
https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
This is using the same token and the same zookeeper URI, for some
reason it is stuck on spark-shell, and the following is outputted on
tserver side:
2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: Error
occurred during processing of message.
java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 11 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 17 more
Any idea why?
Thanks.
-Simon
Re: same token couldn't authenticate twice?
Posted by "Xu (Simon) Chen" <xc...@gmail.com>.
Jim,
The commit you mentioned is not in the branch you created for me
(rc7_a1.7_h2.5)...
-Simon
On Thu, Jun 11, 2015 at 8:09 PM, James Hughes <jn...@virginia.edu> wrote:
> Simon,
>
> Hmm... it may be worth reverting the changes to the GeoMesaSpark source from
> this commit:
> https://github.com/locationtech/geomesa/commit/1d0178e399ce5f66f0bcd94b071bf0634fe31e8d.
>
> That'd give a chance to make sure the problem is with the AccumuloInput
> rather than something GeoMesa is doing with the input format. As note,
> GeoMesa has some code to help gather splits similar to what Eugene proposed
> recently. I don't know if that made it into 1.7.0 or not.
>
> Cheers,
>
> Jim
>
> On Thu, Jun 11, 2015 at 7:51 PM, Xu (Simon) Chen <xc...@gmail.com> wrote:
>>
>> Thanks Josh...
>>
>> I tested this in scala REPL, and called DataStoreFinder.getDataStore()
>> multiple times, each time it seems to be reusing the same
>> KerberosToken object, and it works fine each time.
>>
>> So my problem only happens when the token is used in accumulo's mapred
>> package. Weird..
>>
>> -Simon
>>
>>
>> On Thu, Jun 11, 2015 at 5:29 PM, Josh Elser <jo...@gmail.com> wrote:
>> > Simon,
>> >
>> > Can you reproduce this in plain-jane Java code? I don't know enough
>> > about
>> > spark/scala, much less what Geomesa is actually do, to know what the
>> > issue
>> > is.
>> >
>> > Also, which token are you referring to: A KerberosToken or a
>> > DelegationToken? Either of them should be usable as many times as you'd
>> > like
>> > (given the underlying credentials are still available for KT or the DT
>> > token
>> > hasn't yet expired).
>> >
>> >
>> > Xu (Simon) Chen wrote:
>> >>
>> >> Folks,
>> >>
>> >> I am working on geomesa+accumulo+spark integration. For some reason, I
>> >> found that the same token cannot be used to authenticate twice.
>> >>
>> >> The workflow is that geomesa would try to create a hadoop rdd, during
>> >> which it tries to create an AccumuloDataStore:
>> >>
>> >>
>> >> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>> >>
>> >> During this process, a ZooKeeperInstance is created:
>> >>
>> >>
>> >> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>> >> I modified geomesa such that it would use kerberos to authenticate
>> >> here. This step works fine.
>> >>
>> >> But next, geomesa calls ConfigurationBase.setConnectorInfo:
>> >>
>> >>
>> >> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>> >>
>> >> This is using the same token and the same zookeeper URI, for some
>> >> reason it is stuck on spark-shell, and the following is outputted on
>> >> tserver side:
>> >>
>> >> 2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: Error
>> >> occurred during processing of message.
>> >> java.lang.RuntimeException:
>> >> org.apache.thrift.transport.TTransportException:
>> >> java.net.SocketTimeoutException: Read timed out
>> >> at
>> >>
>> >> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>> >> at
>> >>
>> >> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>> >> at
>> >>
>> >> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>> >> at java.security.AccessController.doPrivileged(Native Method)
>> >> at javax.security.auth.Subject.doAs(Subject.java:356)
>> >> at
>> >>
>> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>> >> at
>> >>
>> >> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>> >> at
>> >>
>> >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>> >> at
>> >>
>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >> at
>> >>
>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >> at
>> >>
>> >> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>> >> at java.lang.Thread.run(Thread.java:745)
>> >> Caused by: org.apache.thrift.transport.TTransportException:
>> >> java.net.SocketTimeoutException: Read timed out
>> >> at
>> >>
>> >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>> >> at
>> >> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> >> at
>> >>
>> >> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>> >> at
>> >>
>> >> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>> >> at
>> >>
>> >> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>> >> at
>> >>
>> >> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>> >> at
>> >>
>> >> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>> >> ... 11 more
>> >> Caused by: java.net.SocketTimeoutException: Read timed out
>> >> at java.net.SocketInputStream.socketRead0(Native Method)
>> >> at java.net.SocketInputStream.read(SocketInputStream.java:152)
>> >> at java.net.SocketInputStream.read(SocketInputStream.java:122)
>> >> at
>> >> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> >> at
>> >> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> >> at
>> >>
>> >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>> >> ... 17 more
>> >>
>> >> Any idea why?
>> >>
>> >> Thanks.
>> >> -Simon
>
>
Re: same token couldn't authenticate twice?
Posted by James Hughes <jn...@virginia.edu>.
Simon,
Hmm... it may be worth reverting the changes to the GeoMesaSpark source
from this commit:
https://github.com/locationtech/geomesa/commit/1d0178e399ce5f66f0bcd94b071bf0634fe31e8d.
That'd give a chance to make sure the problem is with the AccumuloInput
rather than something GeoMesa is doing with the input format. As note,
GeoMesa has some code to help gather splits similar to what Eugene proposed
recently. I don't know if that made it into 1.7.0 or not.
Cheers,
Jim
On Thu, Jun 11, 2015 at 7:51 PM, Xu (Simon) Chen <xc...@gmail.com> wrote:
> Thanks Josh...
>
> I tested this in scala REPL, and called DataStoreFinder.getDataStore()
> multiple times, each time it seems to be reusing the same
> KerberosToken object, and it works fine each time.
>
> So my problem only happens when the token is used in accumulo's mapred
> package. Weird..
>
> -Simon
>
>
> On Thu, Jun 11, 2015 at 5:29 PM, Josh Elser <jo...@gmail.com> wrote:
> > Simon,
> >
> > Can you reproduce this in plain-jane Java code? I don't know enough about
> > spark/scala, much less what Geomesa is actually do, to know what the
> issue
> > is.
> >
> > Also, which token are you referring to: A KerberosToken or a
> > DelegationToken? Either of them should be usable as many times as you'd
> like
> > (given the underlying credentials are still available for KT or the DT
> token
> > hasn't yet expired).
> >
> >
> > Xu (Simon) Chen wrote:
> >>
> >> Folks,
> >>
> >> I am working on geomesa+accumulo+spark integration. For some reason, I
> >> found that the same token cannot be used to authenticate twice.
> >>
> >> The workflow is that geomesa would try to create a hadoop rdd, during
> >> which it tries to create an AccumuloDataStore:
> >>
> >>
> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
> >>
> >> During this process, a ZooKeeperInstance is created:
> >>
> >>
> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
> >> I modified geomesa such that it would use kerberos to authenticate
> >> here. This step works fine.
> >>
> >> But next, geomesa calls ConfigurationBase.setConnectorInfo:
> >>
> >>
> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
> >>
> >> This is using the same token and the same zookeeper URI, for some
> >> reason it is stuck on spark-shell, and the following is outputted on
> >> tserver side:
> >>
> >> 2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: Error
> >> occurred during processing of message.
> >> java.lang.RuntimeException:
> >> org.apache.thrift.transport.TTransportException:
> >> java.net.SocketTimeoutException: Read timed out
> >> at
> >>
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> >> at
> >>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> >> at
> >>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:356)
> >> at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
> >> at
> >>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> >> at
> >>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at
> >>
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> >> at java.lang.Thread.run(Thread.java:745)
> >> Caused by: org.apache.thrift.transport.TTransportException:
> >> java.net.SocketTimeoutException: Read timed out
> >> at
> >>
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> >> at
> >> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> >> at
> >>
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
> >> at
> >>
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
> >> at
> >> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> >> at
> >>
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> >> at
> >>
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> >> ... 11 more
> >> Caused by: java.net.SocketTimeoutException: Read timed out
> >> at java.net.SocketInputStream.socketRead0(Native Method)
> >> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> >> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> >> at
> >> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> >> at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> >> at
> >>
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> >> ... 17 more
> >>
> >> Any idea why?
> >>
> >> Thanks.
> >> -Simon
>
Re: same token couldn't authenticate twice?
Posted by "Xu (Simon) Chen" <xc...@gmail.com>.
I did something like AccumuloInputFormat.setZooKeeperInstance(Job,
ClientConfiguration.loadDefault().withZKHosts(zk).withInstance(name).withSasl(true))
So, this explicitly instruct to turn on SASL, and it's working past
where I was stuck on. Now I seem to have a different problem :-( I'll
look into it and report back later.
Thanks Josh!
-Simon
On Fri, Jun 12, 2015 at 1:47 PM, Josh Elser <jo...@gmail.com> wrote:
> Generally, it's not a good idea to assume that you can always locate the
> correct ClientConfiguration from the local filesystem.
>
> For example, YARN nodes might not have Accumulo installed, or might even
> point to the wrong Accumulo instance. The Job's Configuration serves as the
> de-facto place for _all_ information that your Job needs to perform its
> work.
>
> Can you try calling AccumuloInputFormat.setZooKeeperInstance(Job,
> ClientConfiguration) instead?
>
> Great work tracking this down, Simon!
>
>
> Xu (Simon) Chen wrote:
>>
>> Ah, I found the problem...
>>
>> In the hadoop chain of events, this is called eventually, because
>> clientConfigString is not null (containing instance.name and
>> instance.zookeeper.host):
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java#L382
>>
>> The deserialize function unfortunately doesn't load the default
>> options, therefore left out the sasl thing from ~/.accumulo/config
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/ClientConfiguration.java#L235
>>
>> Would it be reasonable for deserialize to load the default settings?
>>
>> -Simon
>>
>>
>> On Fri, Jun 12, 2015 at 11:34 AM, Josh Elser<jo...@gmail.com> wrote:
>>>
>>> Just be careful with the mapreduce classes. I wouldn't be surprised if we
>>> try to avoid any locally installed client.conf in MapReduce (using only
>>> the
>>> ClientConfiguration stored inside the Job).
>>>
>>> Will wait to hear back from you :)
>>>
>>> Xu (Simon) Chen wrote:
>>>>
>>>> Emm.. I have ~/.accumulo/config with "instance.rpc.sasl.enabled=true".
>>>> That property is indeed populated to ClientConfiguration the first time
>>>> - that's why I said the token worked initially.
>>>>
>>>> Apparently, in the Hadoop portion that property is not set, as I added
>>>> some debug message to ZooKeeperInstance class. I think that's likely the
>>>> issue.
>>>>
>>>> So the zookeeper instance is created in the following sequence:
>>>>
>>>>
>>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L341
>>>>
>>>>
>>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java#L671
>>>>
>>>>
>>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java#L361
>>>>
>>>> The getClientConfiguration function calls getDefaultSearchPath()
>>>> eventually, so my ~/.accumulo/config should be searched. I think we are
>>>> close to the root cause... Will update when I find out more.
>>>>
>>>> Thanks!
>>>> -Simon
>>>>
>>>> On Thu, Jun 11, 2015 at 11:28 PM, Josh Elser<jo...@gmail.com>
>>>> wrote:
>>>> > Are you sure that the spark tasks have the proper
>>>> ClientConfiguration? They
>>>> > need to have instance.rpc.sasl.enabled. I believe you should be
>>>> able
>>>> to set
>>>> > this via the AccumuloInputFormat
>>>> >
>>>> > You can turn up logging org.apache.accumulo.core.client=TRACE
>>>> and/or
>>>> set the
>>>> > system property -Dsun.security.krb5.debug=true to get some more
>>>> information
>>>> > as to why the authentication is failing.
>>>> >
>>>> >
>>>> > Xu (Simon) Chen wrote:
>>>> >>
>>>> >> Josh,
>>>> >>
>>>> >> I am using this function:
>>>> >>
>>>> >>
>>>> >>
>>>>
>>>>
>>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L106
>>>> >>
>>>> >> If I pass in a KerberosToken, it's stuck at line 111; if I pass in
>>>> a
>>>> >> delegation token, the setConnectorInfo function finishes fine.
>>>> >>
>>>> >> But when I do something like queryRDD.count, spark eventually
>>>> calls
>>>> >> HadoopRDD.getPartitions, which calls the following and get stuck
>>>> in
>>>> >> the last authenticate() function:
>>>> >>
>>>> >>
>>>>
>>>>
>>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L621
>>>> >>
>>>> >>
>>>>
>>>>
>>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L348
>>>> >>
>>>> >>
>>>>
>>>>
>>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/ZooKeeperInstance.java#L248
>>>> >>
>>>> >>
>>>>
>>>>
>>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
>>>> >>
>>>> >> Which essentially the same place where it would be stuck with
>>>> >> KerberosToken.
>>>> >>
>>>> >> -Simon
>>>> >>
>>>> >> On Thu, Jun 11, 2015 at 9:41 PM, Josh Elser<jo...@gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>> What are the Accumulo methods that you are calling and what is
>>>> the
>>>> error
>>>> >>> you
>>>> >>> are seeing?
>>>> >>>
>>>> >>> A KerberosToken cannot be used in a MapReduce job which is why a
>>>> >>> DelegationToken is automatically retrieved. You should still be
>>>> able
>>>> to
>>>> >>> provide your own DelegationToken -- if that doesn't work, that's
>>>> a
>>>> bug.
>>>> >>>
>>>> >>> Xu (Simon) Chen wrote:
>>>> >>>>
>>>> >>>> I actually added a flag such that I can pass in either a
>>>> KerberosToken
>>>> >>>> or a DelegationTokenImpl to accumulo.
>>>> >>>>
>>>> >>>> Actually when a KerberosToken is passed in, accumulo converts it
>>>> to
>>>> a
>>>> >>>> DelegationToken - the conversion is where I am having trouble. I
>>>> tried
>>>> >>>> passing in a delegation token directly to bypass the conversion,
>>>> but
>>>> a
>>>> >>>> similar problem happens, that I am stuck at authenticate on the
>>>> client
>>>> >>>> side, and server side outputs the same output...
>>>> >>>>
>>>> >>>> On Thursday, June 11, 2015, Josh Elser<josh.elser@gmail.com
>>>> >>>> <ma...@gmail.com>> wrote:
>>>> >>>>
>>>> >>>> Keep in mind that the authentication path for
>>>> DelegationToken
>>>> >>>> (mapreduce) and KerberosToken are completely different.
>>>> >>>>
>>>> >>>> Since most mapreduce jobs have multiple mappers (or
>>>> reducers),
>>>> I
>>>> >>>> expect we would have run into the case that the same
>>>> >>>> DelegationToken
>>>> >>>> was used multiple times. It would still be good to narrow
>>>> down the
>>>> >>>> scope of the problem.
>>>> >>>>
>>>> >>>> Xu (Simon) Chen wrote:
>>>> >>>>
>>>> >>>> Thanks Josh...
>>>> >>>>
>>>> >>>> I tested this in scala REPL, and called
>>>> >>>> DataStoreFinder.getDataStore()
>>>> >>>> multiple times, each time it seems to be reusing the
>>>> same
>>>> >>>> KerberosToken object, and it works fine each time.
>>>> >>>>
>>>> >>>> So my problem only happens when the token is used in
>>>> accumulo's
>>>> >>>> mapred
>>>> >>>> package. Weird..
>>>> >>>>
>>>> >>>> -Simon
>>>> >>>>
>>>> >>>>
>>>> >>>> On Thu, Jun 11, 2015 at 5:29 PM, Josh
>>>> >>>> Elser<jo...@gmail.com> wrote:
>>>> >>>>
>>>> >>>> Simon,
>>>> >>>>
>>>> >>>> Can you reproduce this in plain-jane Java code? I
>>>> don't
>>>> >>>> know
>>>> >>>> enough about
>>>> >>>> spark/scala, much less what Geomesa is actually do,
>>>> to know
>>>> >>>> what the issue
>>>> >>>> is.
>>>> >>>>
>>>> >>>> Also, which token are you referring to: A
>>>> KerberosToken or
>>>> >>>> a
>>>> >>>> DelegationToken? Either of them should be usable as
>>>> many
>>>> >>>> times as you'd like
>>>> >>>> (given the underlying credentials are still
>>>> available
>>>> for
>>>> >>>> KT
>>>> >>>> or the DT token
>>>> >>>> hasn't yet expired).
>>>> >>>>
>>>> >>>>
>>>> >>>> Xu (Simon) Chen wrote:
>>>> >>>>
>>>> >>>> Folks,
>>>> >>>>
>>>> >>>> I am working on geomesa+accumulo+spark
>>>> integration. For
>>>> >>>> some reason, I
>>>> >>>> found that the same token cannot be used to
>>>> >>>> authenticate
>>>> >>>> twice.
>>>> >>>>
>>>> >>>> The workflow is that geomesa would try to
>>>> create a
>>>> >>>> hadoop rdd, during
>>>> >>>> which it tries to create an AccumuloDataStore:
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>>>> >>>>
>>>> >>>> During this process, a ZooKeeperInstance is
>>>> created:
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>>>> >>>> I modified geomesa such that it would use
>>>> kerberos
>>>> to
>>>> >>>> authenticate
>>>> >>>> here. This step works fine.
>>>> >>>>
>>>> >>>> But next, geomesa calls
>>>> >>>> ConfigurationBase.setConnectorInfo:
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>>>> >>>>
>>>> >>>> This is using the same token and the same
>>>> zookeeper
>>>> >>>> URI,
>>>> >>>> for some
>>>> >>>> reason it is stuck on spark-shell, and the
>>>> following is
>>>> >>>> outputted on
>>>> >>>> tserver side:
>>>> >>>>
>>>> >>>> 2015-06-06 18:58:19,616
>>>> [server.TThreadPoolServer]
>>>> >>>> ERROR: Error
>>>> >>>> occurred during processing of message.
>>>> >>>> java.lang.RuntimeException:
>>>> >>>>
>>>> org.apache.thrift.transport.TTransportException:
>>>> >>>> java.net
>>>> <http://java.net><http://java.net>.SocketTimeoutException: Read
>>>>
>>>> >>>>
>>>> >>>> timed out
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>>>> >>>> at
>>>> >>>>
>>>> java.security.AccessController.doPrivileged(Native
>>>> >>>> Method)
>>>> >>>> at
>>>> >>>>
>>>> javax.security.auth.Subject.doAs(Subject.java:356)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>>> >>>> at
>>>> java.lang.Thread.run(Thread.java:745)
>>>> >>>> Caused by:
>>>> >>>> org.apache.thrift.transport.TTransportException:
>>>> >>>> java.net
>>>> <http://java.net><http://java.net>.SocketTimeoutException: Read
>>>>
>>>> >>>> timed out
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>>> >>>> ... 11 more
>>>> >>>> Caused by: java.net<http://java.net>
>>>> >>>> <http://java.net>.SocketTimeoutException: Read timed
>>>> >>>> out
>>>> >>>> at
>>>> >>>> java.net.SocketInputStream.socketRead0(Native
>>>> Method)
>>>> >>>> at
>>>> >>>>
>>>> >>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
>>>> >>>> at
>>>> >>>>
>>>> >>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
>>>> >>>> at
>>>> >>>>
>>>> >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> >>>> at
>>>> >>>>
>>>> >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> >>>> at
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>>
>>>>
>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>>> >>>> ... 17 more
>>>> >>>>
>>>> >>>> Any idea why?
>>>> >>>>
>>>> >>>> Thanks.
>>>> >>>> -Simon
>>>> >>>>
>>>> >
>>>>
>
Re: same token couldn't authenticate twice?
Posted by Josh Elser <jo...@gmail.com>.
Generally, it's not a good idea to assume that you can always locate the
correct ClientConfiguration from the local filesystem.
For example, YARN nodes might not have Accumulo installed, or might even
point to the wrong Accumulo instance. The Job's Configuration serves as
the de-facto place for _all_ information that your Job needs to perform
its work.
Can you try calling AccumuloInputFormat.setZooKeeperInstance(Job,
ClientConfiguration) instead?
Great work tracking this down, Simon!
Xu (Simon) Chen wrote:
> Ah, I found the problem...
>
> In the hadoop chain of events, this is called eventually, because
> clientConfigString is not null (containing instance.name and
> instance.zookeeper.host):
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java#L382
>
> The deserialize function unfortunately doesn't load the default
> options, therefore left out the sasl thing from ~/.accumulo/config
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/ClientConfiguration.java#L235
>
> Would it be reasonable for deserialize to load the default settings?
>
> -Simon
>
>
> On Fri, Jun 12, 2015 at 11:34 AM, Josh Elser<jo...@gmail.com> wrote:
>> Just be careful with the mapreduce classes. I wouldn't be surprised if we
>> try to avoid any locally installed client.conf in MapReduce (using only the
>> ClientConfiguration stored inside the Job).
>>
>> Will wait to hear back from you :)
>>
>> Xu (Simon) Chen wrote:
>>> Emm.. I have ~/.accumulo/config with "instance.rpc.sasl.enabled=true".
>>> That property is indeed populated to ClientConfiguration the first time
>>> - that's why I said the token worked initially.
>>>
>>> Apparently, in the Hadoop portion that property is not set, as I added
>>> some debug message to ZooKeeperInstance class. I think that's likely the
>>> issue.
>>>
>>> So the zookeeper instance is created in the following sequence:
>>>
>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L341
>>>
>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java#L671
>>>
>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java#L361
>>>
>>> The getClientConfiguration function calls getDefaultSearchPath()
>>> eventually, so my ~/.accumulo/config should be searched. I think we are
>>> close to the root cause... Will update when I find out more.
>>>
>>> Thanks!
>>> -Simon
>>>
>>> On Thu, Jun 11, 2015 at 11:28 PM, Josh Elser<jo...@gmail.com> wrote:
>>> > Are you sure that the spark tasks have the proper
>>> ClientConfiguration? They
>>> > need to have instance.rpc.sasl.enabled. I believe you should be able
>>> to set
>>> > this via the AccumuloInputFormat
>>> >
>>> > You can turn up logging org.apache.accumulo.core.client=TRACE and/or
>>> set the
>>> > system property -Dsun.security.krb5.debug=true to get some more
>>> information
>>> > as to why the authentication is failing.
>>> >
>>> >
>>> > Xu (Simon) Chen wrote:
>>> >>
>>> >> Josh,
>>> >>
>>> >> I am using this function:
>>> >>
>>> >>
>>> >>
>>>
>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L106
>>> >>
>>> >> If I pass in a KerberosToken, it's stuck at line 111; if I pass in a
>>> >> delegation token, the setConnectorInfo function finishes fine.
>>> >>
>>> >> But when I do something like queryRDD.count, spark eventually calls
>>> >> HadoopRDD.getPartitions, which calls the following and get stuck in
>>> >> the last authenticate() function:
>>> >>
>>> >>
>>>
>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L621
>>> >>
>>> >>
>>>
>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L348
>>> >>
>>> >>
>>>
>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/ZooKeeperInstance.java#L248
>>> >>
>>> >>
>>>
>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
>>> >>
>>> >> Which essentially the same place where it would be stuck with
>>> >> KerberosToken.
>>> >>
>>> >> -Simon
>>> >>
>>> >> On Thu, Jun 11, 2015 at 9:41 PM, Josh Elser<jo...@gmail.com>
>>> wrote:
>>> >>>
>>> >>> What are the Accumulo methods that you are calling and what is the
>>> error
>>> >>> you
>>> >>> are seeing?
>>> >>>
>>> >>> A KerberosToken cannot be used in a MapReduce job which is why a
>>> >>> DelegationToken is automatically retrieved. You should still be able
>>> to
>>> >>> provide your own DelegationToken -- if that doesn't work, that's a
>>> bug.
>>> >>>
>>> >>> Xu (Simon) Chen wrote:
>>> >>>>
>>> >>>> I actually added a flag such that I can pass in either a
>>> KerberosToken
>>> >>>> or a DelegationTokenImpl to accumulo.
>>> >>>>
>>> >>>> Actually when a KerberosToken is passed in, accumulo converts it to
>>> a
>>> >>>> DelegationToken - the conversion is where I am having trouble. I
>>> tried
>>> >>>> passing in a delegation token directly to bypass the conversion, but
>>> a
>>> >>>> similar problem happens, that I am stuck at authenticate on the
>>> client
>>> >>>> side, and server side outputs the same output...
>>> >>>>
>>> >>>> On Thursday, June 11, 2015, Josh Elser<josh.elser@gmail.com
>>> >>>> <ma...@gmail.com>> wrote:
>>> >>>>
>>> >>>> Keep in mind that the authentication path for DelegationToken
>>> >>>> (mapreduce) and KerberosToken are completely different.
>>> >>>>
>>> >>>> Since most mapreduce jobs have multiple mappers (or reducers),
>>> I
>>> >>>> expect we would have run into the case that the same
>>> >>>> DelegationToken
>>> >>>> was used multiple times. It would still be good to narrow
>>> down the
>>> >>>> scope of the problem.
>>> >>>>
>>> >>>> Xu (Simon) Chen wrote:
>>> >>>>
>>> >>>> Thanks Josh...
>>> >>>>
>>> >>>> I tested this in scala REPL, and called
>>> >>>> DataStoreFinder.getDataStore()
>>> >>>> multiple times, each time it seems to be reusing the same
>>> >>>> KerberosToken object, and it works fine each time.
>>> >>>>
>>> >>>> So my problem only happens when the token is used in
>>> accumulo's
>>> >>>> mapred
>>> >>>> package. Weird..
>>> >>>>
>>> >>>> -Simon
>>> >>>>
>>> >>>>
>>> >>>> On Thu, Jun 11, 2015 at 5:29 PM, Josh
>>> >>>> Elser<jo...@gmail.com> wrote:
>>> >>>>
>>> >>>> Simon,
>>> >>>>
>>> >>>> Can you reproduce this in plain-jane Java code? I don't
>>> >>>> know
>>> >>>> enough about
>>> >>>> spark/scala, much less what Geomesa is actually do,
>>> to know
>>> >>>> what the issue
>>> >>>> is.
>>> >>>>
>>> >>>> Also, which token are you referring to: A
>>> KerberosToken or
>>> >>>> a
>>> >>>> DelegationToken? Either of them should be usable as
>>> many
>>> >>>> times as you'd like
>>> >>>> (given the underlying credentials are still available
>>> for
>>> >>>> KT
>>> >>>> or the DT token
>>> >>>> hasn't yet expired).
>>> >>>>
>>> >>>>
>>> >>>> Xu (Simon) Chen wrote:
>>> >>>>
>>> >>>> Folks,
>>> >>>>
>>> >>>> I am working on geomesa+accumulo+spark
>>> integration. For
>>> >>>> some reason, I
>>> >>>> found that the same token cannot be used to
>>> >>>> authenticate
>>> >>>> twice.
>>> >>>>
>>> >>>> The workflow is that geomesa would try to create a
>>> >>>> hadoop rdd, during
>>> >>>> which it tries to create an AccumuloDataStore:
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>>> >>>>
>>> >>>> During this process, a ZooKeeperInstance is
>>> created:
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>>> >>>> I modified geomesa such that it would use kerberos
>>> to
>>> >>>> authenticate
>>> >>>> here. This step works fine.
>>> >>>>
>>> >>>> But next, geomesa calls
>>> >>>> ConfigurationBase.setConnectorInfo:
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>>> >>>>
>>> >>>> This is using the same token and the same zookeeper
>>> >>>> URI,
>>> >>>> for some
>>> >>>> reason it is stuck on spark-shell, and the
>>> following is
>>> >>>> outputted on
>>> >>>> tserver side:
>>> >>>>
>>> >>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
>>> >>>> ERROR: Error
>>> >>>> occurred during processing of message.
>>> >>>> java.lang.RuntimeException:
>>> >>>> org.apache.thrift.transport.TTransportException:
>>> >>>> java.net
>>> <http://java.net><http://java.net>.SocketTimeoutException: Read
>>>
>>> >>>>
>>> >>>> timed out
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>>> >>>> at
>>> >>>> java.security.AccessController.doPrivileged(Native
>>> >>>> Method)
>>> >>>> at
>>> >>>> javax.security.auth.Subject.doAs(Subject.java:356)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>> >>>> at java.lang.Thread.run(Thread.java:745)
>>> >>>> Caused by:
>>> >>>> org.apache.thrift.transport.TTransportException:
>>> >>>> java.net
>>> <http://java.net><http://java.net>.SocketTimeoutException: Read
>>>
>>> >>>> timed out
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>> >>>> at
>>> >>>>
>>> >>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>> >>>> ... 11 more
>>> >>>> Caused by: java.net<http://java.net>
>>> >>>> <http://java.net>.SocketTimeoutException: Read timed
>>> >>>> out
>>> >>>> at
>>> >>>> java.net.SocketInputStream.socketRead0(Native
>>> Method)
>>> >>>> at
>>> >>>>
>>> >>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
>>> >>>> at
>>> >>>>
>>> >>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
>>> >>>> at
>>> >>>>
>>> >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> >>>> at
>>> >>>>
>>> >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> >>>> at
>>> >>>>
>>> >>>>
>>> >>>>
>>>
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>> >>>> ... 17 more
>>> >>>>
>>> >>>> Any idea why?
>>> >>>>
>>> >>>> Thanks.
>>> >>>> -Simon
>>> >>>>
>>> >
>>>
Re: same token couldn't authenticate twice?
Posted by "Xu (Simon) Chen" <xc...@gmail.com>.
Ah, I found the problem...
In the hadoop chain of events, this is called eventually, because
clientConfigString is not null (containing instance.name and
instance.zookeeper.host):
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java#L382
The deserialize function unfortunately doesn't load the default
options, therefore left out the sasl thing from ~/.accumulo/config
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/ClientConfiguration.java#L235
Would it be reasonable for deserialize to load the default settings?
-Simon
On Fri, Jun 12, 2015 at 11:34 AM, Josh Elser <jo...@gmail.com> wrote:
> Just be careful with the mapreduce classes. I wouldn't be surprised if we
> try to avoid any locally installed client.conf in MapReduce (using only the
> ClientConfiguration stored inside the Job).
>
> Will wait to hear back from you :)
>
> Xu (Simon) Chen wrote:
>>
>> Emm.. I have ~/.accumulo/config with "instance.rpc.sasl.enabled=true".
>> That property is indeed populated to ClientConfiguration the first time
>> - that's why I said the token worked initially.
>>
>> Apparently, in the Hadoop portion that property is not set, as I added
>> some debug message to ZooKeeperInstance class. I think that's likely the
>> issue.
>>
>> So the zookeeper instance is created in the following sequence:
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L341
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java#L671
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java#L361
>>
>> The getClientConfiguration function calls getDefaultSearchPath()
>> eventually, so my ~/.accumulo/config should be searched. I think we are
>> close to the root cause... Will update when I find out more.
>>
>> Thanks!
>> -Simon
>>
>> On Thu, Jun 11, 2015 at 11:28 PM, Josh Elser <jo...@gmail.com> wrote:
>> > Are you sure that the spark tasks have the proper
>> ClientConfiguration? They
>> > need to have instance.rpc.sasl.enabled. I believe you should be able
>> to set
>> > this via the AccumuloInputFormat
>> >
>> > You can turn up logging org.apache.accumulo.core.client=TRACE and/or
>> set the
>> > system property -Dsun.security.krb5.debug=true to get some more
>> information
>> > as to why the authentication is failing.
>> >
>> >
>> > Xu (Simon) Chen wrote:
>> >>
>> >> Josh,
>> >>
>> >> I am using this function:
>> >>
>> >>
>> >>
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L106
>> >>
>> >> If I pass in a KerberosToken, it's stuck at line 111; if I pass in a
>> >> delegation token, the setConnectorInfo function finishes fine.
>> >>
>> >> But when I do something like queryRDD.count, spark eventually calls
>> >> HadoopRDD.getPartitions, which calls the following and get stuck in
>> >> the last authenticate() function:
>> >>
>> >>
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L621
>> >>
>> >>
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L348
>> >>
>> >>
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/ZooKeeperInstance.java#L248
>> >>
>> >>
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
>> >>
>> >> Which essentially the same place where it would be stuck with
>> >> KerberosToken.
>> >>
>> >> -Simon
>> >>
>> >> On Thu, Jun 11, 2015 at 9:41 PM, Josh Elser<jo...@gmail.com>
>> wrote:
>> >>>
>> >>> What are the Accumulo methods that you are calling and what is the
>> error
>> >>> you
>> >>> are seeing?
>> >>>
>> >>> A KerberosToken cannot be used in a MapReduce job which is why a
>> >>> DelegationToken is automatically retrieved. You should still be able
>> to
>> >>> provide your own DelegationToken -- if that doesn't work, that's a
>> bug.
>> >>>
>> >>> Xu (Simon) Chen wrote:
>> >>>>
>> >>>> I actually added a flag such that I can pass in either a
>> KerberosToken
>> >>>> or a DelegationTokenImpl to accumulo.
>> >>>>
>> >>>> Actually when a KerberosToken is passed in, accumulo converts it to
>> a
>> >>>> DelegationToken - the conversion is where I am having trouble. I
>> tried
>> >>>> passing in a delegation token directly to bypass the conversion, but
>> a
>> >>>> similar problem happens, that I am stuck at authenticate on the
>> client
>> >>>> side, and server side outputs the same output...
>> >>>>
>> >>>> On Thursday, June 11, 2015, Josh Elser<josh.elser@gmail.com
>> >>>> <ma...@gmail.com>> wrote:
>> >>>>
>> >>>> Keep in mind that the authentication path for DelegationToken
>> >>>> (mapreduce) and KerberosToken are completely different.
>> >>>>
>> >>>> Since most mapreduce jobs have multiple mappers (or reducers),
>> I
>> >>>> expect we would have run into the case that the same
>> >>>> DelegationToken
>> >>>> was used multiple times. It would still be good to narrow
>> down the
>> >>>> scope of the problem.
>> >>>>
>> >>>> Xu (Simon) Chen wrote:
>> >>>>
>> >>>> Thanks Josh...
>> >>>>
>> >>>> I tested this in scala REPL, and called
>> >>>> DataStoreFinder.getDataStore()
>> >>>> multiple times, each time it seems to be reusing the same
>> >>>> KerberosToken object, and it works fine each time.
>> >>>>
>> >>>> So my problem only happens when the token is used in
>> accumulo's
>> >>>> mapred
>> >>>> package. Weird..
>> >>>>
>> >>>> -Simon
>> >>>>
>> >>>>
>> >>>> On Thu, Jun 11, 2015 at 5:29 PM, Josh
>> >>>> Elser<jo...@gmail.com> wrote:
>> >>>>
>> >>>> Simon,
>> >>>>
>> >>>> Can you reproduce this in plain-jane Java code? I don't
>> >>>> know
>> >>>> enough about
>> >>>> spark/scala, much less what Geomesa is actually do,
>> to know
>> >>>> what the issue
>> >>>> is.
>> >>>>
>> >>>> Also, which token are you referring to: A
>> KerberosToken or
>> >>>> a
>> >>>> DelegationToken? Either of them should be usable as
>> many
>> >>>> times as you'd like
>> >>>> (given the underlying credentials are still available
>> for
>> >>>> KT
>> >>>> or the DT token
>> >>>> hasn't yet expired).
>> >>>>
>> >>>>
>> >>>> Xu (Simon) Chen wrote:
>> >>>>
>> >>>> Folks,
>> >>>>
>> >>>> I am working on geomesa+accumulo+spark
>> integration. For
>> >>>> some reason, I
>> >>>> found that the same token cannot be used to
>> >>>> authenticate
>> >>>> twice.
>> >>>>
>> >>>> The workflow is that geomesa would try to create a
>> >>>> hadoop rdd, during
>> >>>> which it tries to create an AccumuloDataStore:
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>>
>> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>> >>>>
>> >>>> During this process, a ZooKeeperInstance is
>> created:
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>>
>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>> >>>> I modified geomesa such that it would use kerberos
>> to
>> >>>> authenticate
>> >>>> here. This step works fine.
>> >>>>
>> >>>> But next, geomesa calls
>> >>>> ConfigurationBase.setConnectorInfo:
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>>
>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>> >>>>
>> >>>> This is using the same token and the same zookeeper
>> >>>> URI,
>> >>>> for some
>> >>>> reason it is stuck on spark-shell, and the
>> following is
>> >>>> outputted on
>> >>>> tserver side:
>> >>>>
>> >>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
>> >>>> ERROR: Error
>> >>>> occurred during processing of message.
>> >>>> java.lang.RuntimeException:
>> >>>> org.apache.thrift.transport.TTransportException:
>> >>>> java.net
>> <http://java.net><http://java.net>.SocketTimeoutException: Read
>>
>> >>>>
>> >>>> timed out
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>> >>>> at
>> >>>> java.security.AccessController.doPrivileged(Native
>> >>>> Method)
>> >>>> at
>> >>>> javax.security.auth.Subject.doAs(Subject.java:356)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>> >>>> at java.lang.Thread.run(Thread.java:745)
>> >>>> Caused by:
>> >>>> org.apache.thrift.transport.TTransportException:
>> >>>> java.net
>> <http://java.net><http://java.net>.SocketTimeoutException: Read
>>
>> >>>> timed out
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>> >>>> at
>> >>>>
>> >>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>> >>>> at
>> >>>>
>> >>>>
>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>> >>>> ... 11 more
>> >>>> Caused by: java.net <http://java.net>
>> >>>> <http://java.net>.SocketTimeoutException: Read timed
>> >>>> out
>> >>>> at
>> >>>> java.net.SocketInputStream.socketRead0(Native
>> Method)
>> >>>> at
>> >>>>
>> >>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
>> >>>> at
>> >>>>
>> >>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
>> >>>> at
>> >>>>
>> >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> >>>> at
>> >>>>
>> >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> >>>> at
>> >>>>
>> >>>>
>> >>>>
>>
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>> >>>> ... 17 more
>> >>>>
>> >>>> Any idea why?
>> >>>>
>> >>>> Thanks.
>> >>>> -Simon
>> >>>>
>> >
>>
>
Re: same token couldn't authenticate twice?
Posted by Josh Elser <jo...@gmail.com>.
Just be careful with the mapreduce classes. I wouldn't be surprised if
we try to avoid any locally installed client.conf in MapReduce (using
only the ClientConfiguration stored inside the Job).
Will wait to hear back from you :)
Xu (Simon) Chen wrote:
> Emm.. I have ~/.accumulo/config with "instance.rpc.sasl.enabled=true".
> That property is indeed populated to ClientConfiguration the first time
> - that's why I said the token worked initially.
>
> Apparently, in the Hadoop portion that property is not set, as I added
> some debug message to ZooKeeperInstance class. I think that's likely the
> issue.
>
> So the zookeeper instance is created in the following sequence:
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L341
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java#L671
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java#L361
>
> The getClientConfiguration function calls getDefaultSearchPath()
> eventually, so my ~/.accumulo/config should be searched. I think we are
> close to the root cause... Will update when I find out more.
>
> Thanks!
> -Simon
>
> On Thu, Jun 11, 2015 at 11:28 PM, Josh Elser <jo...@gmail.com> wrote:
> > Are you sure that the spark tasks have the proper
> ClientConfiguration? They
> > need to have instance.rpc.sasl.enabled. I believe you should be able
> to set
> > this via the AccumuloInputFormat
> >
> > You can turn up logging org.apache.accumulo.core.client=TRACE and/or
> set the
> > system property -Dsun.security.krb5.debug=true to get some more
> information
> > as to why the authentication is failing.
> >
> >
> > Xu (Simon) Chen wrote:
> >>
> >> Josh,
> >>
> >> I am using this function:
> >>
> >>
> >>
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L106
> >>
> >> If I pass in a KerberosToken, it's stuck at line 111; if I pass in a
> >> delegation token, the setConnectorInfo function finishes fine.
> >>
> >> But when I do something like queryRDD.count, spark eventually calls
> >> HadoopRDD.getPartitions, which calls the following and get stuck in
> >> the last authenticate() function:
> >>
> >>
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L621
> >>
> >>
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L348
> >>
> >>
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/ZooKeeperInstance.java#L248
> >>
> >>
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
> >>
> >> Which essentially the same place where it would be stuck with
> >> KerberosToken.
> >>
> >> -Simon
> >>
> >> On Thu, Jun 11, 2015 at 9:41 PM, Josh Elser<jo...@gmail.com>
> wrote:
> >>>
> >>> What are the Accumulo methods that you are calling and what is the
> error
> >>> you
> >>> are seeing?
> >>>
> >>> A KerberosToken cannot be used in a MapReduce job which is why a
> >>> DelegationToken is automatically retrieved. You should still be able to
> >>> provide your own DelegationToken -- if that doesn't work, that's a bug.
> >>>
> >>> Xu (Simon) Chen wrote:
> >>>>
> >>>> I actually added a flag such that I can pass in either a KerberosToken
> >>>> or a DelegationTokenImpl to accumulo.
> >>>>
> >>>> Actually when a KerberosToken is passed in, accumulo converts it to a
> >>>> DelegationToken - the conversion is where I am having trouble. I tried
> >>>> passing in a delegation token directly to bypass the conversion, but a
> >>>> similar problem happens, that I am stuck at authenticate on the client
> >>>> side, and server side outputs the same output...
> >>>>
> >>>> On Thursday, June 11, 2015, Josh Elser<josh.elser@gmail.com
> >>>> <ma...@gmail.com>> wrote:
> >>>>
> >>>> Keep in mind that the authentication path for DelegationToken
> >>>> (mapreduce) and KerberosToken are completely different.
> >>>>
> >>>> Since most mapreduce jobs have multiple mappers (or reducers), I
> >>>> expect we would have run into the case that the same
> >>>> DelegationToken
> >>>> was used multiple times. It would still be good to narrow
> down the
> >>>> scope of the problem.
> >>>>
> >>>> Xu (Simon) Chen wrote:
> >>>>
> >>>> Thanks Josh...
> >>>>
> >>>> I tested this in scala REPL, and called
> >>>> DataStoreFinder.getDataStore()
> >>>> multiple times, each time it seems to be reusing the same
> >>>> KerberosToken object, and it works fine each time.
> >>>>
> >>>> So my problem only happens when the token is used in
> accumulo's
> >>>> mapred
> >>>> package. Weird..
> >>>>
> >>>> -Simon
> >>>>
> >>>>
> >>>> On Thu, Jun 11, 2015 at 5:29 PM, Josh
> >>>> Elser<jo...@gmail.com> wrote:
> >>>>
> >>>> Simon,
> >>>>
> >>>> Can you reproduce this in plain-jane Java code? I don't
> >>>> know
> >>>> enough about
> >>>> spark/scala, much less what Geomesa is actually do,
> to know
> >>>> what the issue
> >>>> is.
> >>>>
> >>>> Also, which token are you referring to: A
> KerberosToken or
> >>>> a
> >>>> DelegationToken? Either of them should be usable as many
> >>>> times as you'd like
> >>>> (given the underlying credentials are still available for
> >>>> KT
> >>>> or the DT token
> >>>> hasn't yet expired).
> >>>>
> >>>>
> >>>> Xu (Simon) Chen wrote:
> >>>>
> >>>> Folks,
> >>>>
> >>>> I am working on geomesa+accumulo+spark
> integration. For
> >>>> some reason, I
> >>>> found that the same token cannot be used to
> >>>> authenticate
> >>>> twice.
> >>>>
> >>>> The workflow is that geomesa would try to create a
> >>>> hadoop rdd, during
> >>>> which it tries to create an AccumuloDataStore:
> >>>>
> >>>>
> >>>>
> >>>>
> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
> >>>>
> >>>> During this process, a ZooKeeperInstance is created:
> >>>>
> >>>>
> >>>>
> >>>>
> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
> >>>> I modified geomesa such that it would use kerberos to
> >>>> authenticate
> >>>> here. This step works fine.
> >>>>
> >>>> But next, geomesa calls
> >>>> ConfigurationBase.setConnectorInfo:
> >>>>
> >>>>
> >>>>
> >>>>
> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
> >>>>
> >>>> This is using the same token and the same zookeeper
> >>>> URI,
> >>>> for some
> >>>> reason it is stuck on spark-shell, and the
> following is
> >>>> outputted on
> >>>> tserver side:
> >>>>
> >>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
> >>>> ERROR: Error
> >>>> occurred during processing of message.
> >>>> java.lang.RuntimeException:
> >>>> org.apache.thrift.transport.TTransportException:
> >>>> java.net
> <http://java.net><http://java.net>.SocketTimeoutException: Read
> >>>>
> >>>> timed out
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> >>>> at
> >>>> java.security.AccessController.doPrivileged(Native
> >>>> Method)
> >>>> at
> >>>> javax.security.auth.Subject.doAs(Subject.java:356)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> >>>> at
> >>>>
> >>>>
> >>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>>>
> >>>>
> >>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> Caused by:
> >>>> org.apache.thrift.transport.TTransportException:
> >>>> java.net
> <http://java.net><http://java.net>.SocketTimeoutException: Read
> >>>> timed out
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> >>>> at
> >>>>
> >>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
> >>>> at
> >>>>
> >>>>
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> >>>> ... 11 more
> >>>> Caused by: java.net <http://java.net>
> >>>> <http://java.net>.SocketTimeoutException: Read timed
> >>>> out
> >>>> at
> >>>> java.net.SocketInputStream.socketRead0(Native Method)
> >>>> at
> >>>>
> >>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
> >>>> at
> >>>>
> >>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
> >>>> at
> >>>>
> >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> >>>> at
> >>>>
> >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> >>>> at
> >>>>
> >>>>
> >>>>
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> >>>> ... 17 more
> >>>>
> >>>> Any idea why?
> >>>>
> >>>> Thanks.
> >>>> -Simon
> >>>>
> >
>
same token couldn't authenticate twice?
Posted by "Xu (Simon) Chen" <xc...@gmail.com>.
Emm.. I have ~/.accumulo/config with "instance.rpc.sasl.enabled=true". That
property is indeed populated to ClientConfiguration the first time - that's
why I said the token worked initially.
Apparently, in the Hadoop portion that property is not set, as I added some
debug message to ZooKeeperInstance class. I think that's likely the issue.
So the zookeeper instance is created in the following sequence:
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L341
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java#L671
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java#L361
The getClientConfiguration function calls getDefaultSearchPath()
eventually, so my ~/.accumulo/config should be searched. I think we are
close to the root cause... Will update when I find out more.
Thanks!
-Simon
On Thu, Jun 11, 2015 at 11:28 PM, Josh Elser <jo...@gmail.com> wrote:
> Are you sure that the spark tasks have the proper ClientConfiguration?
They
> need to have instance.rpc.sasl.enabled. I believe you should be able to
set
> this via the AccumuloInputFormat
>
> You can turn up logging org.apache.accumulo.core.client=TRACE and/or set
the
> system property -Dsun.security.krb5.debug=true to get some more
information
> as to why the authentication is failing.
>
>
> Xu (Simon) Chen wrote:
>>
>> Josh,
>>
>> I am using this function:
>>
>>
>>
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L106
>>
>> If I pass in a KerberosToken, it's stuck at line 111; if I pass in a
>> delegation token, the setConnectorInfo function finishes fine.
>>
>> But when I do something like queryRDD.count, spark eventually calls
>> HadoopRDD.getPartitions, which calls the following and get stuck in
>> the last authenticate() function:
>>
>>
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L621
>>
>>
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L348
>>
>>
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/ZooKeeperInstance.java#L248
>>
>>
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
>>
>> Which essentially the same place where it would be stuck with
>> KerberosToken.
>>
>> -Simon
>>
>> On Thu, Jun 11, 2015 at 9:41 PM, Josh Elser<jo...@gmail.com> wrote:
>>>
>>> What are the Accumulo methods that you are calling and what is the error
>>> you
>>> are seeing?
>>>
>>> A KerberosToken cannot be used in a MapReduce job which is why a
>>> DelegationToken is automatically retrieved. You should still be able to
>>> provide your own DelegationToken -- if that doesn't work, that's a bug.
>>>
>>> Xu (Simon) Chen wrote:
>>>>
>>>> I actually added a flag such that I can pass in either a KerberosToken
>>>> or a DelegationTokenImpl to accumulo.
>>>>
>>>> Actually when a KerberosToken is passed in, accumulo converts it to a
>>>> DelegationToken - the conversion is where I am having trouble. I tried
>>>> passing in a delegation token directly to bypass the conversion, but a
>>>> similar problem happens, that I am stuck at authenticate on the client
>>>> side, and server side outputs the same output...
>>>>
>>>> On Thursday, June 11, 2015, Josh Elser<josh.elser@gmail.com
>>>> <ma...@gmail.com>> wrote:
>>>>
>>>> Keep in mind that the authentication path for DelegationToken
>>>> (mapreduce) and KerberosToken are completely different.
>>>>
>>>> Since most mapreduce jobs have multiple mappers (or reducers), I
>>>> expect we would have run into the case that the same
>>>> DelegationToken
>>>> was used multiple times. It would still be good to narrow down the
>>>> scope of the problem.
>>>>
>>>> Xu (Simon) Chen wrote:
>>>>
>>>> Thanks Josh...
>>>>
>>>> I tested this in scala REPL, and called
>>>> DataStoreFinder.getDataStore()
>>>> multiple times, each time it seems to be reusing the same
>>>> KerberosToken object, and it works fine each time.
>>>>
>>>> So my problem only happens when the token is used in
accumulo's
>>>> mapred
>>>> package. Weird..
>>>>
>>>> -Simon
>>>>
>>>>
>>>> On Thu, Jun 11, 2015 at 5:29 PM, Josh
>>>> Elser<jo...@gmail.com> wrote:
>>>>
>>>> Simon,
>>>>
>>>> Can you reproduce this in plain-jane Java code? I don't
>>>> know
>>>> enough about
>>>> spark/scala, much less what Geomesa is actually do, to
know
>>>> what the issue
>>>> is.
>>>>
>>>> Also, which token are you referring to: A KerberosToken or
>>>> a
>>>> DelegationToken? Either of them should be usable as many
>>>> times as you'd like
>>>> (given the underlying credentials are still available for
>>>> KT
>>>> or the DT token
>>>> hasn't yet expired).
>>>>
>>>>
>>>> Xu (Simon) Chen wrote:
>>>>
>>>> Folks,
>>>>
>>>> I am working on geomesa+accumulo+spark integration.
For
>>>> some reason, I
>>>> found that the same token cannot be used to
>>>> authenticate
>>>> twice.
>>>>
>>>> The workflow is that geomesa would try to create a
>>>> hadoop rdd, during
>>>> which it tries to create an AccumuloDataStore:
>>>>
>>>>
>>>>
>>>>
https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>>>>
>>>> During this process, a ZooKeeperInstance is created:
>>>>
>>>>
>>>>
>>>>
https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>>>> I modified geomesa such that it would use kerberos to
>>>> authenticate
>>>> here. This step works fine.
>>>>
>>>> But next, geomesa calls
>>>> ConfigurationBase.setConnectorInfo:
>>>>
>>>>
>>>>
>>>>
https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>>>>
>>>> This is using the same token and the same zookeeper
>>>> URI,
>>>> for some
>>>> reason it is stuck on spark-shell, and the following
is
>>>> outputted on
>>>> tserver side:
>>>>
>>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
>>>> ERROR: Error
>>>> occurred during processing of message.
>>>> java.lang.RuntimeException:
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net<http://java.net>.SocketTimeoutException: Read
>>>>
>>>> timed out
>>>> at
>>>>
>>>>
>>>>
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>>> at
>>>>
>>>>
>>>>
org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>>>> at
>>>>
>>>>
>>>>
org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>>>> at
>>>> java.security.AccessController.doPrivileged(Native
>>>> Method)
>>>> at
>>>> javax.security.auth.Subject.doAs(Subject.java:356)
>>>> at
>>>>
>>>>
>>>>
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>>>> at
>>>>
>>>>
>>>>
org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>>>> at
>>>>
>>>>
>>>>
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>>>> at
>>>>
>>>>
>>>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>>
>>>>
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at
>>>>
>>>>
>>>>
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by:
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net<http://java.net>.SocketTimeoutException: Read
>>>> timed out
>>>> at
>>>>
>>>>
>>>>
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>>> at
>>>>
>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>>> at
>>>>
>>>>
>>>>
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>>>> at
>>>>
>>>>
>>>>
org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>>> at
>>>>
>>>>
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>>> at
>>>>
>>>>
>>>>
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>>> at
>>>>
>>>>
>>>>
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>>> ... 11 more
>>>> Caused by: java.net
>>>> <http://java.net>.SocketTimeoutException: Read timed
>>>> out
>>>> at
>>>> java.net.SocketInputStream.socketRead0(Native Method)
>>>> at
>>>>
>>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
>>>> at
>>>>
>>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
>>>> at
>>>>
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>>
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> at
>>>>
>>>>
>>>>
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>>> ... 17 more
>>>>
>>>> Any idea why?
>>>>
>>>> Thanks.
>>>> -Simon
>>>>
>
Re: same token couldn't authenticate twice?
Posted by Josh Elser <jo...@gmail.com>.
Are you sure that the spark tasks have the proper ClientConfiguration?
They need to have instance.rpc.sasl.enabled. I believe you should be
able to set this via the AccumuloInputFormat
You can turn up logging org.apache.accumulo.core.client=TRACE and/or set
the system property -Dsun.security.krb5.debug=true to get some more
information as to why the authentication is failing.
Xu (Simon) Chen wrote:
> Josh,
>
> I am using this function:
>
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L106
>
> If I pass in a KerberosToken, it's stuck at line 111; if I pass in a
> delegation token, the setConnectorInfo function finishes fine.
>
> But when I do something like queryRDD.count, spark eventually calls
> HadoopRDD.getPartitions, which calls the following and get stuck in
> the last authenticate() function:
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L621
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L348
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/ZooKeeperInstance.java#L248
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
>
> Which essentially the same place where it would be stuck with KerberosToken.
>
> -Simon
>
> On Thu, Jun 11, 2015 at 9:41 PM, Josh Elser<jo...@gmail.com> wrote:
>> What are the Accumulo methods that you are calling and what is the error you
>> are seeing?
>>
>> A KerberosToken cannot be used in a MapReduce job which is why a
>> DelegationToken is automatically retrieved. You should still be able to
>> provide your own DelegationToken -- if that doesn't work, that's a bug.
>>
>> Xu (Simon) Chen wrote:
>>> I actually added a flag such that I can pass in either a KerberosToken
>>> or a DelegationTokenImpl to accumulo.
>>>
>>> Actually when a KerberosToken is passed in, accumulo converts it to a
>>> DelegationToken - the conversion is where I am having trouble. I tried
>>> passing in a delegation token directly to bypass the conversion, but a
>>> similar problem happens, that I am stuck at authenticate on the client
>>> side, and server side outputs the same output...
>>>
>>> On Thursday, June 11, 2015, Josh Elser<josh.elser@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Keep in mind that the authentication path for DelegationToken
>>> (mapreduce) and KerberosToken are completely different.
>>>
>>> Since most mapreduce jobs have multiple mappers (or reducers), I
>>> expect we would have run into the case that the same DelegationToken
>>> was used multiple times. It would still be good to narrow down the
>>> scope of the problem.
>>>
>>> Xu (Simon) Chen wrote:
>>>
>>> Thanks Josh...
>>>
>>> I tested this in scala REPL, and called
>>> DataStoreFinder.getDataStore()
>>> multiple times, each time it seems to be reusing the same
>>> KerberosToken object, and it works fine each time.
>>>
>>> So my problem only happens when the token is used in accumulo's
>>> mapred
>>> package. Weird..
>>>
>>> -Simon
>>>
>>>
>>> On Thu, Jun 11, 2015 at 5:29 PM, Josh
>>> Elser<jo...@gmail.com> wrote:
>>>
>>> Simon,
>>>
>>> Can you reproduce this in plain-jane Java code? I don't know
>>> enough about
>>> spark/scala, much less what Geomesa is actually do, to know
>>> what the issue
>>> is.
>>>
>>> Also, which token are you referring to: A KerberosToken or a
>>> DelegationToken? Either of them should be usable as many
>>> times as you'd like
>>> (given the underlying credentials are still available for KT
>>> or the DT token
>>> hasn't yet expired).
>>>
>>>
>>> Xu (Simon) Chen wrote:
>>>
>>> Folks,
>>>
>>> I am working on geomesa+accumulo+spark integration. For
>>> some reason, I
>>> found that the same token cannot be used to authenticate
>>> twice.
>>>
>>> The workflow is that geomesa would try to create a
>>> hadoop rdd, during
>>> which it tries to create an AccumuloDataStore:
>>>
>>>
>>> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>>>
>>> During this process, a ZooKeeperInstance is created:
>>>
>>>
>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>>> I modified geomesa such that it would use kerberos to
>>> authenticate
>>> here. This step works fine.
>>>
>>> But next, geomesa calls
>>> ConfigurationBase.setConnectorInfo:
>>>
>>>
>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>>>
>>> This is using the same token and the same zookeeper URI,
>>> for some
>>> reason it is stuck on spark-shell, and the following is
>>> outputted on
>>> tserver side:
>>>
>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
>>> ERROR: Error
>>> occurred during processing of message.
>>> java.lang.RuntimeException:
>>> org.apache.thrift.transport.TTransportException:
>>> java.net<http://java.net>.SocketTimeoutException: Read
>>>
>>> timed out
>>> at
>>>
>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>> at
>>>
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>>> at
>>>
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>>> at
>>> java.security.AccessController.doPrivileged(Native Method)
>>> at
>>> javax.security.auth.Subject.doAs(Subject.java:356)
>>> at
>>>
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>>> at
>>>
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>>> at
>>>
>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at
>>>
>>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by:
>>> org.apache.thrift.transport.TTransportException:
>>> java.net<http://java.net>.SocketTimeoutException: Read
>>> timed out
>>> at
>>>
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>> at
>>>
>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>> at
>>>
>>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>>> at
>>>
>>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>> at
>>>
>>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>> at
>>>
>>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>> at
>>>
>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>> ... 11 more
>>> Caused by: java.net
>>> <http://java.net>.SocketTimeoutException: Read timed out
>>> at
>>> java.net.SocketInputStream.socketRead0(Native Method)
>>> at
>>>
>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
>>> at
>>>
>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
>>> at
>>>
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> at
>>>
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> at
>>>
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>> ... 17 more
>>>
>>> Any idea why?
>>>
>>> Thanks.
>>> -Simon
>>>
Re: same token couldn't authenticate twice?
Posted by "Xu (Simon) Chen" <xc...@gmail.com>.
Josh,
I am using this function:
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L106
If I pass in a KerberosToken, it's stuck at line 111; if I pass in a
delegation token, the setConnectorInfo function finishes fine.
But when I do something like queryRDD.count, spark eventually calls
HadoopRDD.getPartitions, which calls the following and get stuck in
the last authenticate() function:
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L621
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java#L348
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/ZooKeeperInstance.java#L248
https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
Which essentially the same place where it would be stuck with KerberosToken.
-Simon
On Thu, Jun 11, 2015 at 9:41 PM, Josh Elser <jo...@gmail.com> wrote:
> What are the Accumulo methods that you are calling and what is the error you
> are seeing?
>
> A KerberosToken cannot be used in a MapReduce job which is why a
> DelegationToken is automatically retrieved. You should still be able to
> provide your own DelegationToken -- if that doesn't work, that's a bug.
>
> Xu (Simon) Chen wrote:
>>
>> I actually added a flag such that I can pass in either a KerberosToken
>> or a DelegationTokenImpl to accumulo.
>>
>> Actually when a KerberosToken is passed in, accumulo converts it to a
>> DelegationToken - the conversion is where I am having trouble. I tried
>> passing in a delegation token directly to bypass the conversion, but a
>> similar problem happens, that I am stuck at authenticate on the client
>> side, and server side outputs the same output...
>>
>> On Thursday, June 11, 2015, Josh Elser <josh.elser@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Keep in mind that the authentication path for DelegationToken
>> (mapreduce) and KerberosToken are completely different.
>>
>> Since most mapreduce jobs have multiple mappers (or reducers), I
>> expect we would have run into the case that the same DelegationToken
>> was used multiple times. It would still be good to narrow down the
>> scope of the problem.
>>
>> Xu (Simon) Chen wrote:
>>
>> Thanks Josh...
>>
>> I tested this in scala REPL, and called
>> DataStoreFinder.getDataStore()
>> multiple times, each time it seems to be reusing the same
>> KerberosToken object, and it works fine each time.
>>
>> So my problem only happens when the token is used in accumulo's
>> mapred
>> package. Weird..
>>
>> -Simon
>>
>>
>> On Thu, Jun 11, 2015 at 5:29 PM, Josh
>> Elser<jo...@gmail.com> wrote:
>>
>> Simon,
>>
>> Can you reproduce this in plain-jane Java code? I don't know
>> enough about
>> spark/scala, much less what Geomesa is actually do, to know
>> what the issue
>> is.
>>
>> Also, which token are you referring to: A KerberosToken or a
>> DelegationToken? Either of them should be usable as many
>> times as you'd like
>> (given the underlying credentials are still available for KT
>> or the DT token
>> hasn't yet expired).
>>
>>
>> Xu (Simon) Chen wrote:
>>
>> Folks,
>>
>> I am working on geomesa+accumulo+spark integration. For
>> some reason, I
>> found that the same token cannot be used to authenticate
>> twice.
>>
>> The workflow is that geomesa would try to create a
>> hadoop rdd, during
>> which it tries to create an AccumuloDataStore:
>>
>>
>> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>>
>> During this process, a ZooKeeperInstance is created:
>>
>>
>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>> I modified geomesa such that it would use kerberos to
>> authenticate
>> here. This step works fine.
>>
>> But next, geomesa calls
>> ConfigurationBase.setConnectorInfo:
>>
>>
>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>>
>> This is using the same token and the same zookeeper URI,
>> for some
>> reason it is stuck on spark-shell, and the following is
>> outputted on
>> tserver side:
>>
>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
>> ERROR: Error
>> occurred during processing of message.
>> java.lang.RuntimeException:
>> org.apache.thrift.transport.TTransportException:
>> java.net <http://java.net>.SocketTimeoutException: Read
>>
>> timed out
>> at
>>
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>> at
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>> at
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>> at
>> java.security.AccessController.doPrivileged(Native Method)
>> at
>> javax.security.auth.Subject.doAs(Subject.java:356)
>> at
>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>> at
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>> at
>>
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at
>>
>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by:
>> org.apache.thrift.transport.TTransportException:
>> java.net <http://java.net>.SocketTimeoutException: Read
>> timed out
>> at
>>
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>> at
>>
>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> at
>>
>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>> at
>>
>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>> at
>>
>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>> at
>>
>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>> at
>>
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>> ... 11 more
>> Caused by: java.net
>> <http://java.net>.SocketTimeoutException: Read timed out
>> at
>> java.net.SocketInputStream.socketRead0(Native Method)
>> at
>>
>> java.net.SocketInputStream.read(SocketInputStream.java:152)
>> at
>>
>> java.net.SocketInputStream.read(SocketInputStream.java:122)
>> at
>>
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at
>>
>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> at
>>
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>> ... 17 more
>>
>> Any idea why?
>>
>> Thanks.
>> -Simon
>>
>
Re: same token couldn't authenticate twice?
Posted by Josh Elser <jo...@gmail.com>.
What are the Accumulo methods that you are calling and what is the error
you are seeing?
A KerberosToken cannot be used in a MapReduce job which is why a
DelegationToken is automatically retrieved. You should still be able to
provide your own DelegationToken -- if that doesn't work, that's a bug.
Xu (Simon) Chen wrote:
> I actually added a flag such that I can pass in either a KerberosToken
> or a DelegationTokenImpl to accumulo.
>
> Actually when a KerberosToken is passed in, accumulo converts it to a
> DelegationToken - the conversion is where I am having trouble. I tried
> passing in a delegation token directly to bypass the conversion, but a
> similar problem happens, that I am stuck at authenticate on the client
> side, and server side outputs the same output...
>
> On Thursday, June 11, 2015, Josh Elser <josh.elser@gmail.com
> <ma...@gmail.com>> wrote:
>
> Keep in mind that the authentication path for DelegationToken
> (mapreduce) and KerberosToken are completely different.
>
> Since most mapreduce jobs have multiple mappers (or reducers), I
> expect we would have run into the case that the same DelegationToken
> was used multiple times. It would still be good to narrow down the
> scope of the problem.
>
> Xu (Simon) Chen wrote:
>
> Thanks Josh...
>
> I tested this in scala REPL, and called
> DataStoreFinder.getDataStore()
> multiple times, each time it seems to be reusing the same
> KerberosToken object, and it works fine each time.
>
> So my problem only happens when the token is used in accumulo's
> mapred
> package. Weird..
>
> -Simon
>
>
> On Thu, Jun 11, 2015 at 5:29 PM, Josh
> Elser<jo...@gmail.com> wrote:
>
> Simon,
>
> Can you reproduce this in plain-jane Java code? I don't know
> enough about
> spark/scala, much less what Geomesa is actually do, to know
> what the issue
> is.
>
> Also, which token are you referring to: A KerberosToken or a
> DelegationToken? Either of them should be usable as many
> times as you'd like
> (given the underlying credentials are still available for KT
> or the DT token
> hasn't yet expired).
>
>
> Xu (Simon) Chen wrote:
>
> Folks,
>
> I am working on geomesa+accumulo+spark integration. For
> some reason, I
> found that the same token cannot be used to authenticate
> twice.
>
> The workflow is that geomesa would try to create a
> hadoop rdd, during
> which it tries to create an AccumuloDataStore:
>
> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>
> During this process, a ZooKeeperInstance is created:
>
> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
> I modified geomesa such that it would use kerberos to
> authenticate
> here. This step works fine.
>
> But next, geomesa calls ConfigurationBase.setConnectorInfo:
>
> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>
> This is using the same token and the same zookeeper URI,
> for some
> reason it is stuck on spark-shell, and the following is
> outputted on
> tserver side:
>
> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
> ERROR: Error
> occurred during processing of message.
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException:
> java.net <http://java.net>.SocketTimeoutException: Read
> timed out
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> at
> java.security.AccessController.doPrivileged(Native Method)
> at
> javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException:
> java.net <http://java.net>.SocketTimeoutException: Read
> timed out
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> ... 11 more
> Caused by: java.net
> <http://java.net>.SocketTimeoutException: Read timed out
> at
> java.net.SocketInputStream.socketRead0(Native Method)
> at
> java.net.SocketInputStream.read(SocketInputStream.java:152)
> at
> java.net.SocketInputStream.read(SocketInputStream.java:122)
> at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> ... 17 more
>
> Any idea why?
>
> Thanks.
> -Simon
>
Re: same token couldn't authenticate twice?
Posted by "Xu (Simon) Chen" <xc...@gmail.com>.
I actually added a flag such that I can pass in either a KerberosToken or a
DelegationTokenImpl to accumulo.
Actually when a KerberosToken is passed in, accumulo converts it to a
DelegationToken - the conversion is where I am having trouble. I tried
passing in a delegation token directly to bypass the conversion, but a
similar problem happens, that I am stuck at authenticate on the client
side, and server side outputs the same output...
On Thursday, June 11, 2015, Josh Elser <jo...@gmail.com> wrote:
> Keep in mind that the authentication path for DelegationToken (mapreduce)
> and KerberosToken are completely different.
>
> Since most mapreduce jobs have multiple mappers (or reducers), I expect we
> would have run into the case that the same DelegationToken was used
> multiple times. It would still be good to narrow down the scope of the
> problem.
>
> Xu (Simon) Chen wrote:
>
>> Thanks Josh...
>>
>> I tested this in scala REPL, and called DataStoreFinder.getDataStore()
>> multiple times, each time it seems to be reusing the same
>> KerberosToken object, and it works fine each time.
>>
>> So my problem only happens when the token is used in accumulo's mapred
>> package. Weird..
>>
>> -Simon
>>
>>
>> On Thu, Jun 11, 2015 at 5:29 PM, Josh Elser<jo...@gmail.com> wrote:
>>
>>> Simon,
>>>
>>> Can you reproduce this in plain-jane Java code? I don't know enough about
>>> spark/scala, much less what Geomesa is actually do, to know what the
>>> issue
>>> is.
>>>
>>> Also, which token are you referring to: A KerberosToken or a
>>> DelegationToken? Either of them should be usable as many times as you'd
>>> like
>>> (given the underlying credentials are still available for KT or the DT
>>> token
>>> hasn't yet expired).
>>>
>>>
>>> Xu (Simon) Chen wrote:
>>>
>>>> Folks,
>>>>
>>>> I am working on geomesa+accumulo+spark integration. For some reason, I
>>>> found that the same token cannot be used to authenticate twice.
>>>>
>>>> The workflow is that geomesa would try to create a hadoop rdd, during
>>>> which it tries to create an AccumuloDataStore:
>>>>
>>>>
>>>> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>>>>
>>>> During this process, a ZooKeeperInstance is created:
>>>>
>>>>
>>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>>>> I modified geomesa such that it would use kerberos to authenticate
>>>> here. This step works fine.
>>>>
>>>> But next, geomesa calls ConfigurationBase.setConnectorInfo:
>>>>
>>>>
>>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>>>>
>>>> This is using the same token and the same zookeeper URI, for some
>>>> reason it is stuck on spark-shell, and the following is outputted on
>>>> tserver side:
>>>>
>>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: Error
>>>> occurred during processing of message.
>>>> java.lang.RuntimeException:
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.SocketTimeoutException: Read timed out
>>>> at
>>>>
>>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>>> at
>>>>
>>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>>>> at
>>>>
>>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:356)
>>>> at
>>>>
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>>>> at
>>>>
>>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>>>> at
>>>>
>>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at
>>>>
>>>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>> java.net.SocketTimeoutException: Read timed out
>>>> at
>>>>
>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>>> at
>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>>> at
>>>>
>>>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>>>> at
>>>>
>>>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>>> at
>>>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>>> at
>>>>
>>>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>>> at
>>>>
>>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>>> ... 11 more
>>>> Caused by: java.net.SocketTimeoutException: Read timed out
>>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>>> at java.net.SocketInputStream.read(SocketInputStream.java:152)
>>>> at java.net.SocketInputStream.read(SocketInputStream.java:122)
>>>> at
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>> at
>>>>
>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>>> ... 17 more
>>>>
>>>> Any idea why?
>>>>
>>>> Thanks.
>>>> -Simon
>>>>
>>>
Re: same token couldn't authenticate twice?
Posted by Josh Elser <jo...@gmail.com>.
Keep in mind that the authentication path for DelegationToken
(mapreduce) and KerberosToken are completely different.
Since most mapreduce jobs have multiple mappers (or reducers), I expect
we would have run into the case that the same DelegationToken was used
multiple times. It would still be good to narrow down the scope of the
problem.
Xu (Simon) Chen wrote:
> Thanks Josh...
>
> I tested this in scala REPL, and called DataStoreFinder.getDataStore()
> multiple times, each time it seems to be reusing the same
> KerberosToken object, and it works fine each time.
>
> So my problem only happens when the token is used in accumulo's mapred
> package. Weird..
>
> -Simon
>
>
> On Thu, Jun 11, 2015 at 5:29 PM, Josh Elser<jo...@gmail.com> wrote:
>> Simon,
>>
>> Can you reproduce this in plain-jane Java code? I don't know enough about
>> spark/scala, much less what Geomesa is actually do, to know what the issue
>> is.
>>
>> Also, which token are you referring to: A KerberosToken or a
>> DelegationToken? Either of them should be usable as many times as you'd like
>> (given the underlying credentials are still available for KT or the DT token
>> hasn't yet expired).
>>
>>
>> Xu (Simon) Chen wrote:
>>> Folks,
>>>
>>> I am working on geomesa+accumulo+spark integration. For some reason, I
>>> found that the same token cannot be used to authenticate twice.
>>>
>>> The workflow is that geomesa would try to create a hadoop rdd, during
>>> which it tries to create an AccumuloDataStore:
>>>
>>> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>>>
>>> During this process, a ZooKeeperInstance is created:
>>>
>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>>> I modified geomesa such that it would use kerberos to authenticate
>>> here. This step works fine.
>>>
>>> But next, geomesa calls ConfigurationBase.setConnectorInfo:
>>>
>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>>>
>>> This is using the same token and the same zookeeper URI, for some
>>> reason it is stuck on spark-shell, and the following is outputted on
>>> tserver side:
>>>
>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: Error
>>> occurred during processing of message.
>>> java.lang.RuntimeException:
>>> org.apache.thrift.transport.TTransportException:
>>> java.net.SocketTimeoutException: Read timed out
>>> at
>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>> at
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>>> at
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:356)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>>> at
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>>> at
>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at
>>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.apache.thrift.transport.TTransportException:
>>> java.net.SocketTimeoutException: Read timed out
>>> at
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>> at
>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>> at
>>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>>> at
>>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>> at
>>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>> at
>>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>> at
>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>> ... 11 more
>>> Caused by: java.net.SocketTimeoutException: Read timed out
>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>> at java.net.SocketInputStream.read(SocketInputStream.java:152)
>>> at java.net.SocketInputStream.read(SocketInputStream.java:122)
>>> at
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>> at
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>> ... 17 more
>>>
>>> Any idea why?
>>>
>>> Thanks.
>>> -Simon
Re: same token couldn't authenticate twice?
Posted by "Xu (Simon) Chen" <xc...@gmail.com>.
Thanks Josh...
I tested this in scala REPL, and called DataStoreFinder.getDataStore()
multiple times, each time it seems to be reusing the same
KerberosToken object, and it works fine each time.
So my problem only happens when the token is used in accumulo's mapred
package. Weird..
-Simon
On Thu, Jun 11, 2015 at 5:29 PM, Josh Elser <jo...@gmail.com> wrote:
> Simon,
>
> Can you reproduce this in plain-jane Java code? I don't know enough about
> spark/scala, much less what Geomesa is actually do, to know what the issue
> is.
>
> Also, which token are you referring to: A KerberosToken or a
> DelegationToken? Either of them should be usable as many times as you'd like
> (given the underlying credentials are still available for KT or the DT token
> hasn't yet expired).
>
>
> Xu (Simon) Chen wrote:
>>
>> Folks,
>>
>> I am working on geomesa+accumulo+spark integration. For some reason, I
>> found that the same token cannot be used to authenticate twice.
>>
>> The workflow is that geomesa would try to create a hadoop rdd, during
>> which it tries to create an AccumuloDataStore:
>>
>> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>>
>> During this process, a ZooKeeperInstance is created:
>>
>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>> I modified geomesa such that it would use kerberos to authenticate
>> here. This step works fine.
>>
>> But next, geomesa calls ConfigurationBase.setConnectorInfo:
>>
>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>>
>> This is using the same token and the same zookeeper URI, for some
>> reason it is stuck on spark-shell, and the following is outputted on
>> tserver side:
>>
>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: Error
>> occurred during processing of message.
>> java.lang.RuntimeException:
>> org.apache.thrift.transport.TTransportException:
>> java.net.SocketTimeoutException: Read timed out
>> at
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>> at
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>> at
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:356)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>> at
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>> at
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at
>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.thrift.transport.TTransportException:
>> java.net.SocketTimeoutException: Read timed out
>> at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>> at
>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> at
>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>> at
>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>> at
>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>> at
>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>> at
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>> ... 11 more
>> Caused by: java.net.SocketTimeoutException: Read timed out
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(SocketInputStream.java:152)
>> at java.net.SocketInputStream.read(SocketInputStream.java:122)
>> at
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>> ... 17 more
>>
>> Any idea why?
>>
>> Thanks.
>> -Simon
Re: same token couldn't authenticate twice?
Posted by Josh Elser <jo...@gmail.com>.
Simon,
Can you reproduce this in plain-jane Java code? I don't know enough
about spark/scala, much less what Geomesa is actually do, to know what
the issue is.
Also, which token are you referring to: A KerberosToken or a
DelegationToken? Either of them should be usable as many times as you'd
like (given the underlying credentials are still available for KT or the
DT token hasn't yet expired).
Xu (Simon) Chen wrote:
> Folks,
>
> I am working on geomesa+accumulo+spark integration. For some reason, I
> found that the same token cannot be used to authenticate twice.
>
> The workflow is that geomesa would try to create a hadoop rdd, during
> which it tries to create an AccumuloDataStore:
> https://github.com/locationtech/geomesa/blob/master/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L81
>
> During this process, a ZooKeeperInstance is created:
> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
> I modified geomesa such that it would use kerberos to authenticate
> here. This step works fine.
>
> But next, geomesa calls ConfigurationBase.setConnectorInfo:
> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69
>
> This is using the same token and the same zookeeper URI, for some
> reason it is stuck on spark-shell, and the following is outputted on
> tserver side:
>
> 2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: Error
> occurred during processing of message.
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
> at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
> at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
> at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> ... 11 more
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> ... 17 more
>
> Any idea why?
>
> Thanks.
> -Simon