You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Alok Lal <al...@hortonworks.com> on 2016/01/08 09:21:04 UTC
Review Request 42063: Enable tagsync to run in secure mode
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42063/
-----------------------------------------------------------
Review request for ranger, Abhay Kulkarni and Madhan Neethiraj.
Bugs: RANGER-801
https://issues.apache.org/jira/browse/RANGER-801
Repository: ranger
Description
-------
Enable tagsync to run in secure mode. Please ignore prior review request for this change.
- Since kafka clients only work with jass files, this change does authentication only using jass files. Thanks @Abhay for that feedback during my offline discussion.
- service command passes the jass file argument during startup.
Diffs
-----
src/main/assembly/tagsync.xml 8adc5cc
tagsync/conf.dist/ranger-tagsync-env-setup-hadoop-home.sh c171d2a
tagsync/scripts/install.properties b5ad580
tagsync/scripts/ranger-tagsync-services.sh ca82ead
tagsync/scripts/setup.py f7455b8
tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java e1b5130
tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java 7bae973
Diff: https://reviews.apache.org/r/42063/diff/
Testing
-------
- Modified the tagsync code by hand to write to hdfs in a secure cluster.
- Current junits all work.
Thanks,
Alok Lal
Re: Review Request 42063: Enable tagsync to run in secure mode
Posted by Alok Lal <al...@hortonworks.com>.
> On Jan. 8, 2016, 9:13 a.m., Abhay Kulkarni wrote:
> > tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java, line 89
> > <https://reviews.apache.org/r/42063/diff/1/?file=1187240#file1187240line89>
> >
> > "Server"->"ranger-tagsync" ?
I had offline conversation about this with @Bosco. Looks like the names need to be what Zookeeper and Kafka clients expect. I have made this value configurable, though, so it can be changed, if needed.
> On Jan. 8, 2016, 9:13 a.m., Abhay Kulkarni wrote:
> > tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java, line 112
> > <https://reviews.apache.org/r/42063/diff/1/?file=1187240#file1187240line112>
> >
> > What if the Kerberos login fails for some reason?
The called function catches Throwable. So the main loop should work. If login fails and cluster is secure then kafka connection will fail. But there shoudl be error messages in log file even when running at INFO level.
- Alok
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42063/#review113491
-----------------------------------------------------------
On Jan. 8, 2016, 12:21 a.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42063/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 12:21 a.m.)
>
>
> Review request for ranger, Abhay Kulkarni and Madhan Neethiraj.
>
>
> Bugs: RANGER-801
> https://issues.apache.org/jira/browse/RANGER-801
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Enable tagsync to run in secure mode. Please ignore prior review request for this change.
> - Since kafka clients only work with jass files, this change does authentication only using jass files. Thanks @Abhay for that feedback during my offline discussion.
> - service command passes the jass file argument during startup.
>
>
> Diffs
> -----
>
> src/main/assembly/tagsync.xml 8adc5cc
> tagsync/conf.dist/ranger-tagsync-env-setup-hadoop-home.sh c171d2a
> tagsync/scripts/install.properties b5ad580
> tagsync/scripts/ranger-tagsync-services.sh ca82ead
> tagsync/scripts/setup.py f7455b8
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java e1b5130
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java 7bae973
>
> Diff: https://reviews.apache.org/r/42063/diff/
>
>
> Testing
> -------
>
> - Modified the tagsync code by hand to write to hdfs in a secure cluster.
> - Current junits all work.
>
>
> Thanks,
>
> Alok Lal
>
>
Re: Review Request 42063: Enable tagsync to run in secure mode
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42063/#review113491
-----------------------------------------------------------
tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java (line 89)
<https://reviews.apache.org/r/42063/#comment174231>
"Server"->"ranger-tagsync" ?
tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java (line 112)
<https://reviews.apache.org/r/42063/#comment174232>
What if the Kerberos login fails for some reason?
- Abhay Kulkarni
On Jan. 8, 2016, 8:21 a.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42063/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 8:21 a.m.)
>
>
> Review request for ranger, Abhay Kulkarni and Madhan Neethiraj.
>
>
> Bugs: RANGER-801
> https://issues.apache.org/jira/browse/RANGER-801
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Enable tagsync to run in secure mode. Please ignore prior review request for this change.
> - Since kafka clients only work with jass files, this change does authentication only using jass files. Thanks @Abhay for that feedback during my offline discussion.
> - service command passes the jass file argument during startup.
>
>
> Diffs
> -----
>
> src/main/assembly/tagsync.xml 8adc5cc
> tagsync/conf.dist/ranger-tagsync-env-setup-hadoop-home.sh c171d2a
> tagsync/scripts/install.properties b5ad580
> tagsync/scripts/ranger-tagsync-services.sh ca82ead
> tagsync/scripts/setup.py f7455b8
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java e1b5130
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java 7bae973
>
> Diff: https://reviews.apache.org/r/42063/diff/
>
>
> Testing
> -------
>
> - Modified the tagsync code by hand to write to hdfs in a secure cluster.
> - Current junits all work.
>
>
> Thanks,
>
> Alok Lal
>
>
Re: Review Request 42063: Enable tagsync to run in secure mode
Posted by Madhan Neethiraj <ma...@apache.org>.
> On Jan. 8, 2016, 4:12 p.m., Madhan Neethiraj wrote:
> > src/main/assembly/tagsync.xml, line 72
> > <https://reviews.apache.org/r/42063/diff/1/?file=1187234#file1187234line72>
> >
> > I guess this is needed only for dev-testing. Please review and remember to remove before pushing the commit.
After looking further, I understand audit library reference is added to use MiscUtil.authWithConfig(). I think it will be cleaner to move this implementation to plugins-common library and have tagsync (and MiscUtil.authWithConfig()) use the implementation in plugins-common.
- Madhan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42063/#review113482
-----------------------------------------------------------
On Jan. 8, 2016, 8:21 a.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42063/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 8:21 a.m.)
>
>
> Review request for ranger, Abhay Kulkarni and Madhan Neethiraj.
>
>
> Bugs: RANGER-801
> https://issues.apache.org/jira/browse/RANGER-801
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Enable tagsync to run in secure mode. Please ignore prior review request for this change.
> - Since kafka clients only work with jass files, this change does authentication only using jass files. Thanks @Abhay for that feedback during my offline discussion.
> - service command passes the jass file argument during startup.
>
>
> Diffs
> -----
>
> src/main/assembly/tagsync.xml 8adc5cc
> tagsync/conf.dist/ranger-tagsync-env-setup-hadoop-home.sh c171d2a
> tagsync/scripts/install.properties b5ad580
> tagsync/scripts/ranger-tagsync-services.sh ca82ead
> tagsync/scripts/setup.py f7455b8
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java e1b5130
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java 7bae973
>
> Diff: https://reviews.apache.org/r/42063/diff/
>
>
> Testing
> -------
>
> - Modified the tagsync code by hand to write to hdfs in a secure cluster.
> - Current junits all work.
>
>
> Thanks,
>
> Alok Lal
>
>
Re: Review Request 42063: Enable tagsync to run in secure mode
Posted by Alok Lal <al...@hortonworks.com>.
> On Jan. 8, 2016, 8:12 a.m., Madhan Neethiraj wrote:
> > src/main/assembly/tagsync.xml, line 72
> > <https://reviews.apache.org/r/42063/diff/1/?file=1187234#file1187234line72>
> >
> > I guess this is needed only for dev-testing. Please review and remember to remove before pushing the commit.
>
> Madhan Neethiraj wrote:
> After looking further, I understand audit library reference is added to use MiscUtil.authWithConfig(). I think it will be cleaner to move this implementation to plugins-common library and have tagsync (and MiscUtil.authWithConfig()) use the implementation in plugins-common.
Common depends on audit. Upstream projects of audit are: agents-cred and ranger-solrj
> On Jan. 8, 2016, 8:12 a.m., Madhan Neethiraj wrote:
> > tagsync/scripts/install.properties, line 61
> > <https://reviews.apache.org/r/42063/diff/1/?file=1187236#file1187236line61>
> >
> > Looking at the contents of this file, I guess this patch is from tag-policy branch. That branch is no more used for development. Please create the patch from master branch.
I have the patch ready for master. to make it easy to see changes, I am submitting this patch on an old branch. I have verified that it applies on master.
> On Jan. 8, 2016, 8:12 a.m., Madhan Neethiraj wrote:
> > tagsync/scripts/setup.py, line 273
> > <https://reviews.apache.org/r/42063/diff/1/?file=1187238#file1187238line273>
> >
> > Review how jassFilenameFileName file would be updated in Ambari managed cluster. For example, the jass file location configuration is updated in Ambari, how will jaasFilenameFileName be refreshed?
> >
> > One option to consider is to not use jassFilenameFileName file at all. The startup script should read the configuraion directly from ranger-tagsync-site.xml. To help read the config file, a simple Java program can be used - similar to XmlConfigChanger.java used to update config files during install.
Done.
- Alok
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42063/#review113482
-----------------------------------------------------------
On Jan. 11, 2016, 12:54 a.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42063/
> -----------------------------------------------------------
>
> (Updated Jan. 11, 2016, 12:54 a.m.)
>
>
> Review request for ranger, Abhay Kulkarni and Madhan Neethiraj.
>
>
> Bugs: RANGER-801
> https://issues.apache.org/jira/browse/RANGER-801
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Enable tagsync to run in secure mode. Please ignore prior review request for this change.
> - Since kafka clients only work with jass files, this change does authentication only using jass files. Thanks @Abhay for that feedback during my offline discussion.
> - service command passes the jass file argument during startup.
>
>
> Diffs
> -----
>
> agents-installer/pom.xml 633da6d
> agents-installer/src/main/java/org/apache/ranger/utils/install/ConfigPropertyReader.java PRE-CREATION
> pom.xml d3a7a99
> src/main/assembly/tagsync.xml 8adc5cc
> tagsync/conf.dist/ranger-tagsync-env-setup-hadoop-home.sh c171d2a
> tagsync/conf/templates/installprop2xml.properties 101a1ba
> tagsync/conf/templates/ranger-tagsync-template.xml 9a88681
> tagsync/scripts/install.properties b5ad580
> tagsync/scripts/ranger-tagsync-services.sh ca82ead
> tagsync/scripts/setup.py f7455b8
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java e1b5130
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java 7bae973
>
> Diff: https://reviews.apache.org/r/42063/diff/
>
>
> Testing
> -------
>
> - Modified the tagsync code by hand to write to hdfs in a secure cluster.
> - Current junits all work.
>
>
> Thanks,
>
> Alok Lal
>
>
Re: Review Request 42063: Enable tagsync to run in secure mode
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42063/#review113482
-----------------------------------------------------------
src/main/assembly/tagsync.xml (line 72)
<https://reviews.apache.org/r/42063/#comment174222>
I guess this is needed only for dev-testing. Please review and remember to remove before pushing the commit.
tagsync/conf.dist/ranger-tagsync-env-setup-hadoop-home.sh (line 24)
<https://reviews.apache.org/r/42063/#comment174221>
How about the condition [ $CLASSPATH != "" ]? Shouldn't $HADOOP_HOME/conf be appended to CLASSPATH?
tagsync/scripts/install.properties (line 61)
<https://reviews.apache.org/r/42063/#comment174220>
Looking at the contents of this file, I guess this patch is from tag-policy branch. That branch is no more used for development. Please create the patch from master branch.
tagsync/scripts/setup.py (line 273)
<https://reviews.apache.org/r/42063/#comment174223>
Review how jassFilenameFileName file would be updated in Ambari managed cluster. For example, the jass file location configuration is updated in Ambari, how will jaasFilenameFileName be refreshed?
One option to consider is to not use jassFilenameFileName file at all. The startup script should read the configuraion directly from ranger-tagsync-site.xml. To help read the config file, a simple Java program can be used - similar to XmlConfigChanger.java used to update config files during install.
tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java (line 84)
<https://reviews.apache.org/r/42063/#comment174224>
Consider renaming the method as doKerberosLogin().
- Madhan Neethiraj
On Jan. 8, 2016, 8:21 a.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42063/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2016, 8:21 a.m.)
>
>
> Review request for ranger, Abhay Kulkarni and Madhan Neethiraj.
>
>
> Bugs: RANGER-801
> https://issues.apache.org/jira/browse/RANGER-801
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Enable tagsync to run in secure mode. Please ignore prior review request for this change.
> - Since kafka clients only work with jass files, this change does authentication only using jass files. Thanks @Abhay for that feedback during my offline discussion.
> - service command passes the jass file argument during startup.
>
>
> Diffs
> -----
>
> src/main/assembly/tagsync.xml 8adc5cc
> tagsync/conf.dist/ranger-tagsync-env-setup-hadoop-home.sh c171d2a
> tagsync/scripts/install.properties b5ad580
> tagsync/scripts/ranger-tagsync-services.sh ca82ead
> tagsync/scripts/setup.py f7455b8
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java e1b5130
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java 7bae973
>
> Diff: https://reviews.apache.org/r/42063/diff/
>
>
> Testing
> -------
>
> - Modified the tagsync code by hand to write to hdfs in a secure cluster.
> - Current junits all work.
>
>
> Thanks,
>
> Alok Lal
>
>
Re: Review Request 42063: Enable tagsync to run in secure mode
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42063/#review113786
-----------------------------------------------------------
tagsync/conf.dist/ranger-tagsync-env-setup-hadoop-home.sh (line 23)
<https://reviews.apache.org/r/42063/#comment174598>
Existing CLASSPATH value will be lost with this assignment. To be safe, handle both cases - 1) CLASSPATH is empty 2) CLASSPATH is not empty
agents-installer/src/main/java/org/apache/ranger/utils/install/ConfigPropertyReader.java (line 121)
<https://reviews.apache.org/r/42063/#comment174596>
XmlConfigChanger - is this class used/needed?
- Madhan Neethiraj
On Jan. 11, 2016, 8:54 a.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42063/
> -----------------------------------------------------------
>
> (Updated Jan. 11, 2016, 8:54 a.m.)
>
>
> Review request for ranger, Abhay Kulkarni and Madhan Neethiraj.
>
>
> Bugs: RANGER-801
> https://issues.apache.org/jira/browse/RANGER-801
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Enable tagsync to run in secure mode. Please ignore prior review request for this change.
> - Since kafka clients only work with jass files, this change does authentication only using jass files. Thanks @Abhay for that feedback during my offline discussion.
> - service command passes the jass file argument during startup.
>
>
> Diffs
> -----
>
> agents-installer/pom.xml 633da6d
> agents-installer/src/main/java/org/apache/ranger/utils/install/ConfigPropertyReader.java PRE-CREATION
> pom.xml d3a7a99
> src/main/assembly/tagsync.xml 8adc5cc
> tagsync/conf.dist/ranger-tagsync-env-setup-hadoop-home.sh c171d2a
> tagsync/conf/templates/installprop2xml.properties 101a1ba
> tagsync/conf/templates/ranger-tagsync-template.xml 9a88681
> tagsync/scripts/install.properties b5ad580
> tagsync/scripts/ranger-tagsync-services.sh ca82ead
> tagsync/scripts/setup.py f7455b8
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java e1b5130
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java 7bae973
>
> Diff: https://reviews.apache.org/r/42063/diff/
>
>
> Testing
> -------
>
> - Modified the tagsync code by hand to write to hdfs in a secure cluster.
> - Current junits all work.
>
>
> Thanks,
>
> Alok Lal
>
>
Re: Review Request 42063: Enable tagsync to run in secure mode
Posted by Alok Lal <al...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42063/
-----------------------------------------------------------
(Updated Jan. 11, 2016, 12:54 a.m.)
Review request for ranger, Abhay Kulkarni and Madhan Neethiraj.
Changes
-------
Main Changes done in this patch
- Used hadoop configuration library to read values from site.xml for jass file name.
- Installer will put some files in startup/lib (similar to install/lib). This has classes that startup script uses to get the jass filename.
- Added slf4j-log4j12 dependency to tagsync and version info to main pom. This was needed to help diagnoses auth and connection problems in zookeeper and kafka code inside Atlas client code.
- Still on same version so diff is easy to see. I have applied this on master and it applies cleanly, I'll use that to push.
Testing done:
Tagsync is able to connect to standalone Kafka queue, verified that the consumer for group shows up. Haven't actually gotten Atlas messages to be consumed yet. But that is does not seem to be a security related issue.
Bugs: RANGER-801
https://issues.apache.org/jira/browse/RANGER-801
Repository: ranger
Description
-------
Enable tagsync to run in secure mode. Please ignore prior review request for this change.
- Since kafka clients only work with jass files, this change does authentication only using jass files. Thanks @Abhay for that feedback during my offline discussion.
- service command passes the jass file argument during startup.
Diffs (updated)
-----
agents-installer/pom.xml 633da6d
agents-installer/src/main/java/org/apache/ranger/utils/install/ConfigPropertyReader.java PRE-CREATION
pom.xml d3a7a99
src/main/assembly/tagsync.xml 8adc5cc
tagsync/conf.dist/ranger-tagsync-env-setup-hadoop-home.sh c171d2a
tagsync/conf/templates/installprop2xml.properties 101a1ba
tagsync/conf/templates/ranger-tagsync-template.xml 9a88681
tagsync/scripts/install.properties b5ad580
tagsync/scripts/ranger-tagsync-services.sh ca82ead
tagsync/scripts/setup.py f7455b8
tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java e1b5130
tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSynchronizer.java 7bae973
Diff: https://reviews.apache.org/r/42063/diff/
Testing
-------
- Modified the tagsync code by hand to write to hdfs in a secure cluster.
- Current junits all work.
Thanks,
Alok Lal