You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Nixon Rodrigues (Jira)" <ji...@apache.org> on 2020/05/04 12:06:00 UTC
[jira] [Updated] (ATLAS-3779) Inmemory JAASConfig issue in Atlas
[ https://issues.apache.org/jira/browse/ATLAS-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nixon Rodrigues updated ATLAS-3779:
-----------------------------------
Description:
Spark uses Kafka as source and sink in secure cluster. The test creates a JAAS file like this:
{code:java}
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
keyTab="/xxx/keytabs/systest.keytab"
useTicketCache=false
serviceName="kafka"
principal="systest@GCE.EXAMPLE.COM";
};
{code}
As one can see serviceName is set properly.
Then the test pass the JAAS file to Spark's driver + executor as well:
{code:java}
"--conf \"spark.driver.extraJavaOptions=-Djava.security.auth.login.config=./kafka_source_jaas.conf..."
"--conf \"spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_source_jaas.conf..."
{code}
Later on SAC + atlas makes some magic in the background with the Jvm JAAS configuration. As a result Spark is not able to create consumer for processing data:
{code:java}
Caused by: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
{code}
When I've turned off SAC then all the problem gone away.
Atlas replaces the JVM global JAAS configuration with InMemoryJAASConfiguration once Atlas configuration is initialized. InMemoryJAASConfiguration has an old JAAS config as "parent" but Atlas config takes precedence which is unexpected.
We never want to let Atlas to overwrite existing JAAS configuration if there's a conflict. (I believe most endpoints using Atlas client as a library would agree with this.) This may be achieved via swapping precedence for "parent" vs "Atlas config" in InMemoryJAASConfiguration, but I have no idea the change would be safe to Atlas side. In any way, Atlas should at least provide a config to let "parent" take precedence for the conflict.
was:
Spark uses Kafka as source and sink in secure cluster. The test creates a JAAS file like this:
{code:java}
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
keyTab="/cdep/keytabs/systest.keytab"
useTicketCache=false
serviceName="kafka"
principal="systest@GCE.CLOUDERA.COM";
};
{code}
As one can see serviceName is set properly.
Then the test pass the JAAS file to Spark's driver + executor as well:
{code:java}
"--conf \"spark.driver.extraJavaOptions=-Djava.security.auth.login.config=./kafka_source_jaas.conf..."
"--conf \"spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_source_jaas.conf..."
{code}
Later on SAC + atlas makes some magic in the background with the Jvm JAAS configuration. As a result Spark is not able to create consumer for processing data:
{code:java}
Caused by: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
{code}
When I've turned off SAC then all the problem gone away.
Atlas replaces the JVM global JAAS configuration with InMemoryJAASConfiguration once Atlas configuration is initialized. InMemoryJAASConfiguration has an old JAAS config as "parent" but Atlas config takes precedence which is unexpected.
We never want to let Atlas to overwrite existing JAAS configuration if there's a conflict. (I believe most endpoints using Atlas client as a library would agree with this.) This may be achieved via swapping precedence for "parent" vs "Atlas config" in InMemoryJAASConfiguration, but I have no idea the change would be safe to Atlas side. In any way, Atlas should at least provide a config to let "parent" take precedence for the conflict.
> Inmemory JAASConfig issue in Atlas
> ----------------------------------
>
> Key: ATLAS-3779
> URL: https://issues.apache.org/jira/browse/ATLAS-3779
> Project: Atlas
> Issue Type: Bug
> Reporter: Mayank Jain
> Priority: Major
>
> Spark uses Kafka as source and sink in secure cluster. The test creates a JAAS file like this:
> {code:java}
> KafkaClient {
> com.sun.security.auth.module.Krb5LoginModule required
> debug=true
> useKeyTab=true
> storeKey=true
> keyTab="/xxx/keytabs/systest.keytab"
> useTicketCache=false
> serviceName="kafka"
> principal="systest@GCE.EXAMPLE.COM";
> };
> {code}
> As one can see serviceName is set properly.
> Then the test pass the JAAS file to Spark's driver + executor as well:
> {code:java}
> "--conf \"spark.driver.extraJavaOptions=-Djava.security.auth.login.config=./kafka_source_jaas.conf..."
> "--conf \"spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_source_jaas.conf..."
> {code}
> Later on SAC + atlas makes some magic in the background with the Jvm JAAS configuration. As a result Spark is not able to create consumer for processing data:
> {code:java}
> Caused by: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config
> {code}
> When I've turned off SAC then all the problem gone away.
> Atlas replaces the JVM global JAAS configuration with InMemoryJAASConfiguration once Atlas configuration is initialized. InMemoryJAASConfiguration has an old JAAS config as "parent" but Atlas config takes precedence which is unexpected.
> We never want to let Atlas to overwrite existing JAAS configuration if there's a conflict. (I believe most endpoints using Atlas client as a library would agree with this.) This may be achieved via swapping precedence for "parent" vs "Atlas config" in InMemoryJAASConfiguration, but I have no idea the change would be safe to Atlas side. In any way, Atlas should at least provide a config to let "parent" take precedence for the conflict.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)