You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2019/04/21 10:30:00 UTC
Writing to Aerospike from Spark with bulk write with user
authentication fails
Aerospike Enterprise version if anyone has worked with user authentication!
As far as I know one can create a client with aerospike authentication as
follows that works for single put
import com.aerospike.spark.sql._
import com.aerospike.client.Bin
import com.aerospike.client.Key
import com.aerospike.client.Value
import com.aerospike.client.AerospikeClient
import com.aerospike.client.Host
import com.aerospike.client.policy.ClientPolicy
import com.aerospike.client.policy.WritePolicy
var hosts = {
new Host("rhes75", 3000)
}
var dbHost = "rhes75"
var dbPort = "3000"
var dbConnection = "xyz"
var namespace = "trading"
var dbPassword = "xxxxxx"
var dbSet = "testSet"
val policy = new ClientPolicy()
policy.user = dbConnection
policy.password = dbPassword
val client = new AerospikeClient(policy, hosts)
val TEST_COUNT = 100
for (i <- 1 to TEST_COUNT) {
val key = new Key(namespace, namespace, dbSet, i)
client.put(null, key,
new Bin("one", i),
new Bin("two", "two:"+i),
new Bin("three", i.toDouble)
)
}
However, I have no way passing of user credential when I am doing bulk
write using a dataframe
df.write. mode(SaveMode.Overwrite).
format("com.aerospike.spark.sql"). option("aerospike.namespace",
namespace). option("aerospike.set", dbSet).
option("aerospike.updateByKey", "id").
option("aerospike.keyColumn", "__id").
option("aerospike.batchMax", 5000). option("aerospike.keyPath",
"/etc/aerospike/features.conf"). save()
when I turn off security in conf file in aerospike this works fine.
Otherwise I get the following error:
Caused by: com.aerospike.client.AerospikeException$Connection: Error -8:
Failed to connect to host(s): rhes75 3000 Error 80: not authenticated
Any ideas how this can be done with user authentication
Thanks,
Mich
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: Writing to Aerospike from Spark with bulk write with user
authentication fails
Posted by Mich Talebzadeh <mi...@gmail.com>.
Just as an add on I see this in aerospike log
Apr 21 2019 17:52:24 GMT: INFO (security): (security.c:5483) permitted |
client: 50.140.197.220:33466 | authenticated user: mich | action: login |
detail: user=mich
*Apr 21 2019 17:52:25 GMT: INFO (security): (security.c:5483) not
authenticated | client: 50.140.197.220:33468 <http://50.140.197.220:33468>
| authenticated user: | action: info request | detail: *
So the first one with user=mich is authenticated.
however, the second one looking for bulk-write is missing the authenticated
user. So there must be a way of passing the username?
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sun, 21 Apr 2019 at 11:30, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Aerospike Enterprise version if anyone has worked with user authentication!
>
> As far as I know one can create a client with aerospike authentication as
> follows that works for single put
>
> import com.aerospike.spark.sql._
> import com.aerospike.client.Bin
> import com.aerospike.client.Key
> import com.aerospike.client.Value
> import com.aerospike.client.AerospikeClient
> import com.aerospike.client.Host
> import com.aerospike.client.policy.ClientPolicy
> import com.aerospike.client.policy.WritePolicy
>
> var hosts = {
> new Host("rhes75", 3000)
> }
>
> var dbHost = "rhes75"
> var dbPort = "3000"
> var dbConnection = "xyz"
> var namespace = "trading"
> var dbPassword = "xxxxxx"
> var dbSet = "testSet"
> val policy = new ClientPolicy()
> policy.user = dbConnection
> policy.password = dbPassword
> val client = new AerospikeClient(policy, hosts)
> val TEST_COUNT = 100
> for (i <- 1 to TEST_COUNT) {
> val key = new Key(namespace, namespace, dbSet, i)
> client.put(null, key,
> new Bin("one", i),
> new Bin("two", "two:"+i),
> new Bin("three", i.toDouble)
> )
> }
> However, I have no way passing of user credential when I am doing bulk
> write using a dataframe
>
> df.write. mode(SaveMode.Overwrite). format("com.aerospike.spark.sql"). option("aerospike.namespace", namespace). option("aerospike.set", dbSet). option("aerospike.updateByKey", "id"). option("aerospike.keyColumn", "__id"). option("aerospike.batchMax", 5000). option("aerospike.keyPath", "/etc/aerospike/features.conf"). save()
>
> when I turn off security in conf file in aerospike this works fine.
> Otherwise I get the following error:
>
> Caused by: com.aerospike.client.AerospikeException$Connection: Error -8:
> Failed to connect to host(s): rhes75 3000 Error 80: not authenticated
>
> Any ideas how this can be done with user authentication
>
> Thanks,
>
> Mich
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>