You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Attila Bukor (Code Review)" <ge...@cloudera.org> on 2020/03/18 21:36:28 UTC
[kudu-CR] KUDU-3079 Add MiniRanger pt 3
Attila Bukor has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15483
Change subject: KUDU-3079 Add MiniRanger pt 3
......................................................................
KUDU-3079 Add MiniRanger pt 3
With the previous patch MiniRanger was unable to completely start up due
to various configuration issues.
One of them was that Ranger's Python scripts weren't compatible with
Python 3. To work around this I added a patch to thirdparty that changes
the syntax from 2 to 3 and manually fixes an instance were I noticed an
incompatibility (bytes and str are not equal anymore).
Other issues had to do with the classpath and (mostly) database config,
these are all fixed now and MiniRanger can properly start up Ranger
admin. Login is the default admin:admin on localhost:6080 with no SSL
configured currently.
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M src/kudu/ranger/mini_postgres.cc
M src/kudu/ranger/mini_ranger.cc
M src/kudu/ranger/mini_ranger.h
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
7 files changed, 893 insertions(+), 90 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/1
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 1
Gerrit-Owner: Attila Bukor <ab...@apache.org>
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#42).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,752 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/42
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 42
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#11).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
17 files changed, 2,276 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/11
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 11
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 44:
(16 comments)
Thanks a lot Attila for working on the MiniRanger patch!
http://gerrit.cloudera.org:8080/#/c/15483/44//COMMIT_MSG
Commit Message:
PS44:
I don't see configurations for initializing/running Ranger Kudu plugin in this commit. If we are going to do that in a follow up patch, we probably want to note that in the commit msg?
http://gerrit.cloudera.org:8080/#/c/15483/44//COMMIT_MSG@20
PS44, Line 20: "2to3"
nit: python "2to3" program.
http://gerrit.cloudera.org:8080/#/c/15483/44//COMMIT_MSG@20
PS44, Line 20: some manual changes
Can you list what are these manual changes?
http://gerrit.cloudera.org:8080/#/c/15483/44//COMMIT_MSG@33
PS44, Line 33: RESY
nit: REST
http://gerrit.cloudera.org:8080/#/c/15483/44/build-support/dist_test.py
File build-support/dist_test.py:
http://gerrit.cloudera.org:8080/#/c/15483/44/build-support/dist_test.py@97
PS44, Line 97: "thirdparty/installed/common/opt/ranger"
nit: add a comment?
http://gerrit.cloudera.org:8080/#/c/15483/44/build-support/run_dist_test.py
File build-support/run_dist_test.py:
http://gerrit.cloudera.org:8080/#/c/15483/44/build-support/run_dist_test.py@178
PS44, Line 178: # Crea
nit: indent?
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.h@50
PS40, Line 50: UserList
Can you elaborate why do we need a list here, why not have user to be separated from AuthorizationPolicy? And then associate user with AuthorizationPolicy?
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.h@91
PS40, Line 91: admin_home
nit: can you elaborate what 'admin_home' is for and why only fresh install need it? Also what does the initialization do other than create admin_home.
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.cc@60
PS40, Line 60: LOG(INFO) << "Stopping Ranger...";
nit: should we be consistent to log 'Starting Ranger' at Start() as well?
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.cc@87
PS40, Line 87: ranger_port_ is the RPC port.
: // - ranger_admin_port is the port for the admin REST API
nit: can you elaborate a bit more on which RPC is for with ranger_port_?
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h
File src/kudu/ranger/mini_ranger_configs.h:
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@64
PS44, Line 64: external ranger port
It is not clear which port 'external ranger port' maps to in the doc you linked above.
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@92
PS44, Line 92: Serice
nit: service
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@101
PS44, Line 101: true
Also will this be configurable on a secure env?
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@105
PS44, Line 105: ranger.authentication.method
Should this be configurable when Kerberos is enabled?
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@247
PS44, Line 247: xa_logger
nit: what is xa_logger for?
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@313
PS44, Line 313: std::string GetRangerInstallProperties(std::string bin_dir, uint16_t pg_port) {
: return strings::Substitute(kInstallProperties, bin_dir, pg_port);
: }
:
: std::string GetRangerAdminSiteXml(uint16_t port, uint16_t pg_port,
: uint16_t admin_port) {
: return strings::Substitute(kRangerAdminSiteTemplate, pg_port,
: port, admin_port);
: }
:
: std::string GetRangerAdminDefaultSiteXml(std::string pg_driver,
: uint16_t shutdown_port) {
: return strings::Substitute(kRangerAdminDefaultSiteTemplate, pg_driver,
: shutdown_port);
: }
:
: std::string GetRangerLog4jProperties(std::string log_level) {
: return strings::Substitute(kLog4jPropertiesTemplate, log_level);
: }
:
: std::string GetRangerCoreSiteXml(bool secure) {
: return strings::Substitute(kCoreSiteTemplate, secure ? "kerberos" : "simple");
: }
:
nit: would you mind briefly documenting what is each method for? Especially it is not clear from the name what is the difference between GetRangerAdminSiteXml and GetRangerAdminDefaultSiteXml.
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 44
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Mon, 23 Mar 2020 22:59:05 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#47).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by Python's "2to3" script but it required
some manual changes as well to run on Python 3 (subprocesses return
bytes and they were compared with strings which wasn't a problem in
Python 2 but fails in Python 3 so the subprocess result had to be
decoded).
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin REST API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env.
Starting Ranger subprocess is not handled in this commit, it is going to
be done in a follow-up patch.
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,773 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/47
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 47
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by Python's "2to3" script but it required
some manual changes as well to run on Python 3 (subprocesses return
bytes and they were compared with strings which wasn't a problem in
Python 2 but fails in Python 3 so the subprocess result had to be
decoded).
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin REST API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env.
Starting MiniRanger (fresh install) takes ~20s on my machine (2019
MacBook Pro) and ~30-40s on dist-test server.
Starting Ranger subprocess is not handled in this commit, it is going to
be done in a follow-up patch.
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Reviewed-on: http://gerrit.cloudera.org:8080/15483
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Attila Bukor <ab...@apache.org>
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,779 insertions(+), 31 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Attila Bukor: Verified
Andrew Wong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 52
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 41: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 41
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Mon, 23 Mar 2020 21:27:07 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 47: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15483/47//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15483/47//COMMIT_MSG@1
PS47, Line 1: Parent: ee86f568 ([util] GetBindIpForDaemon() consumes PID wider than 18 bit)
Can you comment on how long it is expected to start up MiniRanger?
http://gerrit.cloudera.org:8080/#/c/15483/47/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/47/src/kudu/ranger/mini_ranger.cc@61
PS47, Line 61: ranger_process_
Do we need to reset ranger_process_?
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 47
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Tue, 24 Mar 2020 23:16:00 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 50: Code-Review+2
Looks good to me, not sure if Andrew or Adar want to take another pass. Thanks!
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 50
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 25 Mar 2020 18:37:38 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#41).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,752 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/41
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 41
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#32).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,753 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/32
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 32
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 22:
(12 comments)
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/mini-cluster/external_mini_cluster.cc
File src/kudu/mini-cluster/external_mini_cluster.cc:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/mini-cluster/external_mini_cluster.cc@337
PS16, Line 337: if (opts_.enable_ranger) {
> I was planning to do that in a follow-up, we need to configure the subproce
No, follow-up is fine. Just wanted clarification.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger-test.cc
File src/kudu/ranger/mini_ranger-test.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger-test.cc@75
PS22, Line 75: ASSERT_TRUE(s.IsRemoteError());
Anything we can check in the error message to prove to ourselves that "policy already found" was the result?
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.h@72
PS22, Line 72: private:
Please doc all of the non-accessor methods.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@40
PS16, Line 40: typedef std::pair<std::vector<std::string>, std::vector<ActionPB>> PolicyItem;
> Done
No doc?
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@91
PS22, Line 91: We're not using this shutdown port as we simply send a SIGTERM,
: // but it's necessary to set it to a random value to avoid collisions in
: // parallel testing.
Not possible to tell Ranger not to bind to it at all?
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@131
PS22, Line 131: Env* env = Env::Default();
This seems to be repeated a bunch; how about create a single env_ at MiniRanger constructor time?
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@150
PS22, Line 150: // TODO(abukor): load a db dump instead as this is very slow
How slow? Anything we can do on the postgres side (some reconfiguration perhaps) to speed it up?
For example, we don't need durability in these tests at all, so if we can tell postgres not to fsync or whatever, we may be able to speed things up.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@316
PS22, Line 316: EasyCurl curl;
: faststring result;
: curl.set_auth(CurlAuthType::BASIC, "admin", "admin");
: RETURN_NOT_OK(curl.PostToURL(JoinPathSegments(ranger_admin_url_,
: "service/plugins/policies"),
: policy_json.ToString(), &result,
: {"Content-Type: application/json"}));
CreateKuduService shares a lot of this. Decompose into a (parameterized) helper?
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.h
File src/kudu/util/curl_util.h:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.h@61
PS22, Line 61: Status PostToURL(const std::string& url,
Doc 'headers'.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.h@143
PS22, Line 143: std::string username_ = "";
:
: std::string password_ = "";
Don't need the initializers.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.cc
File src/kudu/util/curl_util.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.cc@125
PS22, Line 125: CURL_RETURN_NOT_OK(curl_easy_setopt(curl_, CURLOPT_HTTPAUTH, CURLAUTH_ANY));
Set this in the default case of the switch below.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.cc@126
PS22, Line 126: switch (auth_type_) {
Should add a default so that the compiler doesn't warn.
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 22
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Sun, 22 Mar 2020 22:19:47 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 45:
Something we'll want to address soon-ish: I ran mini_ranger-test 50 times and saw two failures with the following error to start Postgres:
../../src/kudu/ranger/mini_ranger-test.cc:74: Failure
Failed
Bad status: Runtime error: failed to start Postgres: /usr/bin/lsof: process exited with non-zero status 1
I0324 03:44:25.806879 348 mini_ranger.cc:60] Stopping Ranger...
F0324 03:44:25.806896 348 subprocess.cc:646] Sub-process is not running
*** Check failure stack trace: ***
*** Aborted at 1585021465 (unix time) try "date -d @1585021465" if you are using GNU date ***
PC: @ 0x7f40050adc37 gsignal
*** SIGABRT (@0x3e80000015c) received by PID 348 (TID 0x7f400d47d880) from PID 348; stack trace: ***
@ 0x7f4005450330 (unknown) at ??:0
@ 0x7f40050adc37 gsignal at ??:0
@ 0x7f40050b1028 abort at ??:0
@ 0x7f4006bd1a19 google::logging_fail() at ??:0
@ 0x7f4006bd330d google::LogMessage::Fail() at ??:0
@ 0x7f4006bd51cd google::LogMessage::SendToLog() at ??:0
@ 0x7f4006bd2e49 google::LogMessage::Flush() at ??:0
@ 0x7f4006bd30a1 google::LogMessage::~LogMessage() at ??:0
@ 0x7f4009f5d38f kudu::Subprocess::KillAndWait() at ??:0
@ 0x7f400ce6605d kudu::ranger::MiniRanger::Stop() at ??:0
@ 0x7f400ce65eaf kudu::ranger::MiniRanger::~MiniRanger() at ??:0
@ 0x41538a kudu::ranger::MiniRangerTest::~MiniRangerTest() at /data/8/awong/Repositories/kudu/build/debug/../../src/kudu/ranger/mini_ranger-test.cc:36
@ 0x41a3ae kudu::ranger::MiniRangerTest_TestGrantSamePrivilegeAfterRestart_Test::~MiniRangerTest_TestGrantSamePrivilegeAfterRestart_Test() at /data/8/awong/Repositories/kudu/build/debug/../../src/kudu/ranger/mini_ranger-test.cc:60
@ 0x41a3de kudu::ranger::MiniRangerTest_TestGrantSamePrivilegeAfterRestart_Test::~MiniRangerTest_TestGrantSamePrivilegeAfterRestart_Test() at /data/8/awong/Repositories/kudu/build/debug/../../src/kudu/ranger/mini_ranger-test.cc:60
@ 0x7f400a7e6b98 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f400a7d42b5 testing::TestInfo::Run() at ??:0
@ 0x7f400a7d43d5 testing::TestCase::Run() at ??:0
@ 0x7f400a7daed8 testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f400a7e70a8 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f400a7d44ad testing::UnitTest::Run() at ??:0
@ 0x7f400d07bf5f RUN_ALL_TESTS() at ??:0
@ 0x7f400d079d70 main at ??:0
@ 0x7f4005098f45 __libc_start_main at ??:0
@ 0x412d59 (unknown) at ??:?
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 45
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Tue, 24 Mar 2020 03:48:48 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#10).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action_test.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
15 files changed, 2,258 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/10
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 10
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#2).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
13 files changed, 2,243 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/2
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 2
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 46:
(16 comments)
http://gerrit.cloudera.org:8080/#/c/15483/44//COMMIT_MSG
Commit Message:
PS44:
> I don't see configurations for initializing/running Ranger Kudu plugin in t
Done
http://gerrit.cloudera.org:8080/#/c/15483/44//COMMIT_MSG@20
PS44, Line 20: Python
> nit: python "2to3" program.
Done
http://gerrit.cloudera.org:8080/#/c/15483/44//COMMIT_MSG@20
PS44, Line 20: but it required
> Can you list what are these manual changes?
Done
http://gerrit.cloudera.org:8080/#/c/15483/44//COMMIT_MSG@33
PS44, Line 33:
> nit: REST
Done
http://gerrit.cloudera.org:8080/#/c/15483/44/build-support/dist_test.py
File build-support/dist_test.py:
http://gerrit.cloudera.org:8080/#/c/15483/44/build-support/dist_test.py@97
PS44, Line 97: # Tests that use the external minicluste
> nit: add a comment?
Done
http://gerrit.cloudera.org:8080/#/c/15483/44/build-support/run_dist_test.py
File build-support/run_dist_test.py:
http://gerrit.cloudera.org:8080/#/c/15483/44/build-support/run_dist_test.py@178
PS44, Line 178: # co
> nit: indent?
Done
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.h@50
PS40, Line 50: UserList
> Can you elaborate why do we need a list here, why not have user to be separ
You can't create multiple policies with identical sets of resources, so if you want to give SELECT and INSERT to johndoe and janedoe on foo.bar and DELETE and ALTER to johnsmith and janesmith to foo.bar, you need to create one policy with two policy items.
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.h@91
PS40, Line 91: admin_home
> nit: can you elaborate what 'admin_home' is for and why only fresh install
it isn't only needed by fresh install, fresh install simply indicates if admin home existed before calling InitRanger. admin_home is the home directory for Ranger admin.
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.cc@60
PS40, Line 60: LOG(INFO) << "Stopping Ranger...";
> nit: should we be consistent to log 'Starting Ranger' at Start() as well?
Done
http://gerrit.cloudera.org:8080/#/c/15483/40/src/kudu/ranger/mini_ranger.cc@87
PS40, Line 87: ranger_port_ is the RPC port (REST API) that the Ranger subprocess and
: // EasyCurl can talk to
> nit: can you elaborate a bit more on which RPC is for with ranger_port_?
Done
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h
File src/kudu/ranger/mini_ranger_configs.h:
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@64
PS44, Line 64: admin port/RPC (REST
> It is not clear which port 'external ranger port' maps to in the doc you li
Done
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@92
PS44, Line 92:
> nit: service
Done
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@101
PS44, Line 101: ptio
> Also will this be configurable on a secure env?
I'm not sure what exactly will be needed for a secure env so din't want to make too many assumptions, we can always change this.
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@105
PS44, Line 105: >NONE</value>
> Should this be configurable when Kerberos is enabled?
probably, but didn't want to make too many assumptions as mentioned above.
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@247
PS44, Line 247: g4j.rootL
> nit: what is xa_logger for?
this is the main logger I think, this is unmodified, except for the log level above. I added a comment
http://gerrit.cloudera.org:8080/#/c/15483/44/src/kudu/ranger/mini_ranger_configs.h@313
PS44, Line 313: </configuration>
: )";
:
: // Gets the contents of the install.properties file used by the db_setup.py
: // script.
: std::string GetRangerInstallProperties(std::string bin_dir, uint16_t pg_port) {
: return strings::Substitute(kInstallProperties, bin_dir, pg_port);
: }
:
: // Gets the contents of the ranger-admin-site.xml config that has most of the
: // configuration needed to start Ranger.
: std::string GetRangerAdminSiteXml(uint16_t port, uint16_t pg_port) {
: return strings::Substitute(kRangerAdminSiteTemplate, pg_port, port);
: }
:
: // Gets the ranger-admin-default-site.xml that has some additional configuration
: // needed to start Ranger. It's unclear why this has to be a separate file.
: std::string GetRangerAdminDefaultSiteXml(std::string pg_driver,
: uint16_t shutdown_port) {
: return strings::Substitute(kRangerAdminDefaultSiteTemplate, pg_driver,
: shutdown_port);
: }
:
:
> nit: would you mind briefly documenting what is each method for? Especially
Done
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 46
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Tue, 24 Mar 2020 09:36:34 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 16:
(43 comments)
http://gerrit.cloudera.org:8080/#/c/15483/16//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15483/16//COMMIT_MSG@33
PS16, Line 33: RESY
REST
http://gerrit.cloudera.org:8080/#/c/15483/16/build-support/dist_test.py
File build-support/dist_test.py:
http://gerrit.cloudera.org:8080/#/c/15483/16/build-support/dist_test.py@116
PS16, Line 116: # Add Postgres and Ranger
Nit: terminate with a period.
May also want to note that these are symlinks.
http://gerrit.cloudera.org:8080/#/c/15483/16/build-support/run_dist_test.py
File build-support/run_dist_test.py:
http://gerrit.cloudera.org:8080/#/c/15483/16/build-support/run_dist_test.py@165
PS16, Line 165: for bin_path in glob.glob(os.path.join(ROOT, "build/*/bin")):
Can you just merge this loop with the chrony one, and update the comment?
http://gerrit.cloudera.org:8080/#/c/15483/16/build-support/run_dist_test.py@176
PS16, Line 176: glob.glob("/usr/lib/jvm/java-1.8.0-*")[0]
This was already done above (L154); reuse.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/mini-cluster/external_mini_cluster.cc
File src/kudu/mini-cluster/external_mini_cluster.cc:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/mini-cluster/external_mini_cluster.cc@337
PS16, Line 337: if (opts_.enable_ranger) {
Where do we configure the Kudu Masters to talk to MiniRanger? Or is that going to be in a follow-up?
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/CMakeLists.txt
File src/kudu/ranger/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/CMakeLists.txt@69
PS16, Line 69: # need to chmod +x thirdparty/src/postgresql-42.2.10/postgresql-42.2.10.jar
What is this comment telling us? Is this a TODO?
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/CMakeLists.txt@100
PS16, Line 100: mini_ranger
Alphabetical sorting.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger-test.cc
File src/kudu/ranger/mini_ranger-test.cc:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger-test.cc@40
PS16, Line 40: MiniRanger ranger;
Should be ranger_
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger-test.cc@41
PS16, Line 41:
Nit: empty line here.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger-test.cc@44
PS16, Line 44: TEST_F(MiniRangerTest, TestGrantPrivilege) {
We should at least a full lifecycle (start, restart, stop, etc)
Maybe also test persistence (i.e. you started Ranger and did something. If you restart it, are the effects of that change still there?)
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger-test.cc@52
PS16, Line 52: policy.items.emplace_back(item);
std::move
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@40
PS16, Line 40: typedef std::pair<std::vector<std::string>, std::vector<ActionPB>> policy_item;
This needs documentation, and should be called PolicyItem.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@42
PS16, Line 42: struct AuthorizationPolicy {
Doc.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@50
PS16, Line 50: class MiniRanger {
Doc.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@57
PS16, Line 57: move
How did you get away with this? We shouldn't be 'using' anything in this header. Should be std::move.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@64
PS16, Line 64: Status AddPolicy(const AuthorizationPolicy& policy) WARN_UNUSED_RESULT;
Doc.
Also, can we pass by value and move into it?
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@69
PS16, Line 69: static Status CreateRangerConfigs(const std::string& config_string,
Doc.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@94
PS16, Line 94: std::string ranger_admin_url_;
Doc.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@53
PS16, Line 53:
What a mess. In future work, could you:
1. Take a stab at removing every property/parameter we don't need.
2. Documenting the effect of what remains?
I know it's a big ask, but these configs are just completely inscrutable. I have no idea how anyone in the future could modify them safely.
Might also consider storing these templates in a separate file, with a header to retrieve the de-templatized versions of them.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1132
PS16, Line 1132: <value>simple</value>
Does this warrant a TODO or something? I presume we want to support Kerberos at some point.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1141
PS16, Line 1141: // TODO(awong): codify that we're waiting for Postgres to become usable.
Yeah, this should be the next priority, otherwise the massive number of Sentry tests that we should (hopefully) bring to bear with Ranger will take a very long time to run.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1147
PS16, Line 1147: if (ranger_process_ && ranger_process_->IsStarted()) {
What if Start() failed mid-way such that mini_pg_ has started but Ranger hasn't? Don't we want to explicitly stop mini_pg_ (so we can warn if it failed)?
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1148
PS16, Line 1148: CHECK_OK(Stop());
Should WARN_NOT_OK.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1157
PS16, Line 1157: unique_ptr<Subprocess> ranger_shutdown(new Subprocess({
Holy moly you can't just send the thing a TERM/KILL? What happens if you don't do this fancy shutdown?
Also, doesn't need to be heap allocated.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1179
PS16, Line 1179: if (data_root_.empty()) {
: data_root_ = GetTestDataDirectory();
: }
Do in the constructor and make data_root_ const?
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1213
PS16, Line 1213: if (!env->FileExists(admin_home)) {
Decompose this block into a helper.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1215
PS16, Line 1215: env->CreateDir(admin_home);
RETURN_NOT_OK.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1223
PS16, Line 1223: LOG(INFO) << "Starting Ranger out of " << admin_home;
Shouldn't this be outside the "fresh install" case?
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1232
PS16, Line 1232: RETURN_NOT_OK(GetFQDN(&fqdn));
What about setting up Ranger on a UNIQUE_LOOPBACK address?
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1237
PS16, Line 1237: uint16_t ranger_shutdown_port = GetRandomPort();
There should be a comment somewhere explaining all of the ports that we need to set up, and what purpose they serve.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1238
PS16, Line 1238: RETURN_NOT_OK(CreateRangerConfigs(
Decompose all the config stuff into a helper.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1258
PS16, Line 1258: if (fresh_install) {
Decompose this block into a helper.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1261
PS16, Line 1261: symlink(JoinPathSegments(ranger_home_, "ews/webapp").c_str(),
: kWebAppDir.c_str());
Add a symlink function to env.h and env_posix.h (and a test to env-test.cc).
Can be done in a follow-up if you prefer, but soon (as this doesn't even check return value).
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1264
PS16, Line 1264: Much of this encapsulates setup.sh from apache/ranger
Link?
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1275
PS16, Line 1275: unique_ptr<Subprocess> db_setup(new Subprocess(
Doesn't need to be heap allocated.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1343
PS16, Line 1343: EasyCurl curl;
: EasyJson service;
: service.Set("name", "kudu");
: service.Set("type", "kudu");
: faststring result;
: Status curl_result = curl.PostToURL(JoinPathSegments(ranger_admin_url_,
: "service/plugins/services"),
: service.ToString(), &result, {"Content-Type: application/json"},
: "admin", "admin");
Decompose into a helper.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1357
PS16, Line 1357: Status MiniRanger::CreateRangerConfigs(const string& config_string, const string& file_path) {
Pass by value and std::move both of these.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1366
PS16, Line 1366: int16_t MiniRanger::GetRandomPort() {
Duplicated from MiniPostgres; decompose into net_util somewhere.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1434
PS16, Line 1434: LOG(INFO) << result.ToString();
If there's an error, will it be reflected in the HTTP status code (i.e. will curl fail?) Or do we need to process 'result' to find it?
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/util/curl_util.h
File src/kudu/util/curl_util.h:
PS16:
Can you unit test this change? See PasswdWebserverTest in webserver-test (which you may want to refactor to take advantage of the new functionality anyway).
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/util/curl_util.h@51
PS16, Line 51: const std::string& username = "",
: const std::string& password = "");
These are likely to be the same with each call to FetchURL or PostToURL; could you plumb them as a setter instead?
Also, please make that setter mutually exclusive with set_use_spnego() (i.e. allow just one to be used).
http://gerrit.cloudera.org:8080/#/c/15483/16/thirdparty/build-thirdparty.sh
File thirdparty/build-thirdparty.sh:
http://gerrit.cloudera.org:8080/#/c/15483/16/thirdparty/build-thirdparty.sh@298
PS16, Line 298: cp -R \
This isn't a directory, so why -R?
Also don't you want -f so you can overwrite the destination file if it exists (i.e. multiple invocations of this script).
Alternatively, maybe a symlink would be good enough?
http://gerrit.cloudera.org:8080/#/c/15483/16/thirdparty/patches/ranger-fixscripts.patch
File thirdparty/patches/ranger-fixscripts.patch:
PS16:
The patch is obviously just a limited view into the script itself, but is it OK to just outright remove these three variables? Looks like XAPOLICYMGR_DIR is needed to set XAPOLICYMGR_EWS_DIR just below; how does this work?
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 16
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Sat, 21 Mar 2020 00:33:23 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 49:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15483/47//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15483/47//COMMIT_MSG@1
PS47, Line 1: Parent: ee86f568 ([util] GetBindIpForDaemon() consumes PID wider than 18 bit)
> Can you comment on how long it is expected to start up MiniRanger?
Done
http://gerrit.cloudera.org:8080/#/c/15483/47/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/47/src/kudu/ranger/mini_ranger.cc@61
PS47, Line 61: process_->KillA
> Do we need to reset ranger_process_?
Done
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 49
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 25 Mar 2020 10:54:36 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#40).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,752 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/40
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 40
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#19).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,744 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/19
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 19
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#3).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
13 files changed, 2,249 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/3
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 3
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 28:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.h@53
PS26, Line 53: on a resource
> Is this the policy for a single resource? The vectors of databases, tables,
I think so, but couldn't really find any docs that were clear about how this works. We'll likely need to test it.
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.h@54
PS26, Line 54: user
> nit: users
Done
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.h@76
PS26, Line 76: curl_.set_auth(CurlAuthType::BASIC, "admin", "admin");
: }
> nit: remove a few spaces
how should this be formatted?
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.h@126
PS26, Line 126:
> Will this be evaluated too?
nope, removed
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.cc@238
PS26, Line 238:
> nit: Failed
Done
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.cc@235
PS26, Line 235: "Failed to create Kudu service");
: return Status::OK();
: }
:
: Status MiniRanger::AddPolicy(AuthorizationPolicy policy) {
: EasyJson policy_json;
: policy_js
> Could probably just
Done
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.cc@303
PS26, Line 303: er
> nit: keep the casing consistent? PostToRanger?
Done
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.cc@303
PS26, Line 303:
> nit: const string&
shouldn't we use pass-by-value and move instead of const-ref?
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger_configs.h
File src/kudu/ranger/mini_ranger_configs.h:
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger_configs.h@64
PS26, Line 64: RPC
> nit: RPCs
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/tools/tool.proto
File src/kudu/tools/tool.proto:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/tools/tool.proto@45
PS22, Line 45: // Whether or not Ranger should be enabled
: optional bool enable_ranger = 9;
:
> Ah I see. I think virtually all of our usage of Sentry happened through uni
Done
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 28
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Mon, 23 Mar 2020 10:10:38 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has removed a vote on this change.
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 51
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 39:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/15483/39/build-support/run_dist_test.py
File build-support/run_dist_test.py:
http://gerrit.cloudera.org:8080/#/c/15483/39/build-support/run_dist_test.py@172
PS39, Line 172: os.path.join(bin_path, "postgresql.jar"))
Got some trailing whitespace here.
http://gerrit.cloudera.org:8080/#/c/15483/39/build-support/run_dist_test.py@178
PS39, Line 178: # Create symlink for Ranger default config directory.
Hmm, when dist-test copied the local bundle and sent it to the remote slave, didn't it convert the symlink into a real file?
Or did it not copy this at all?
http://gerrit.cloudera.org:8080/#/c/15483/39/src/kudu/ranger/mini_ranger-test.cc
File src/kudu/ranger/mini_ranger-test.cc:
http://gerrit.cloudera.org:8080/#/c/15483/39/src/kudu/ranger/mini_ranger-test.cc@74
PS39, Line 74: const std::string kExpectedError = "Another policy already exists for matching resource";
Nit: add a using for this.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@150
PS22, Line 150: return db_setup.WaitAndCheckExitCode();
> Around 8 seconds. I don't think Postgres is the bottleneck here though. Als
OK, future work then.
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 39
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Mon, 23 Mar 2020 20:40:38 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#25).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,761 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/25
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 25
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/15483/3/src/kudu/ranger/CMakeLists.txt
File src/kudu/ranger/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/15483/3/src/kudu/ranger/CMakeLists.txt@53
PS3, Line 53: COMMAND ./gradlew :kudu-ranger:jar ${EXTRA_GRADLE_FLAGS}
Does this still exist? I think the subprocess provides what you need.
http://gerrit.cloudera.org:8080/#/c/15483/3/thirdparty/patches/ranger-fixscripts.patch
File thirdparty/patches/ranger-fixscripts.patch:
http://gerrit.cloudera.org:8080/#/c/15483/3/thirdparty/patches/ranger-fixscripts.patch@1
PS3, Line 1: *** ews/ranger-admin-services.sh.orig 2020-03-18 00:27:04.000000000 +0100
Is there a Raner PR for this fix?
http://gerrit.cloudera.org:8080/#/c/15483/3/thirdparty/patches/ranger-python3.patch
File thirdparty/patches/ranger-python3.patch:
http://gerrit.cloudera.org:8080/#/c/15483/3/thirdparty/patches/ranger-python3.patch@1
PS3, Line 1: --- ./update_property.py (original)
Is there a Raner PR for this fix?
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 3
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 18 Mar 2020 23:51:23 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#28).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,754 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/28
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 28
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 24:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@195
PS22, Line 195: "-Dservername=miniranger",
: Substitute("-Dcatalina.base=$0", kEwsDir),
: Substitute("-Dlogdir=$0", JoinPathSegments(kAdminHome, "logs")),
: "-Dranger.audit.solr.bootstrap.enabled=false",
: "-cp", classpath, "org.apache.ranger.server.tomcat.EmbeddedServer"
: }));
: ranger_process_->SetEnvVars({
: { "XAPOLICYMGR_DIR", kAdminHome },
: { "XAPOLICYMGR_EWS_DIR", kEwsDir },
: { "RANGER_JAAS_LIB_DIR", JoinPathSegments(kWebAppDir, "ranger_jaas") },
: { "RANGER_JAAS_CONF_DIR", JoinPathSegments(kConfDir, "ranger_jaas") },
: { "JAVA_HOME", java_home_ },
: { "RANGER_PID_DI
> interestingly I can't find ranger-admin-services.sh in the repo, not sure w
https://github.com/apache/ranger/blob/master/embeddedwebserver/scripts/ranger-admin-services.sh, but you may want to use one from a specific commit in case the link to master becomes stale.
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 24
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Mon, 23 Mar 2020 01:25:00 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 40:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/15483/39/build-support/run_dist_test.py
File build-support/run_dist_test.py:
http://gerrit.cloudera.org:8080/#/c/15483/39/build-support/run_dist_test.py@172
PS39, Line 172: os.path.join(bin_path, "postgresql.jar"))
> Got some trailing whitespace here.
Done
http://gerrit.cloudera.org:8080/#/c/15483/39/build-support/run_dist_test.py@178
PS39, Line 178: env['LD_LIBRARY_PATH'] = ":".join(
> Hmm, when dist-test copied the local bundle and sent it to the remote slave
TBH that's kinda what I'm trying to figure out. I tried running dist-test manually for this test which worked fine, the issue is definitely with the thirdparty libs not being in place where the MiniRanger expects them to be. Unfortunately I can't think of a way to test/fix this other than trying out different ways by changing these and pushing to gerrit.
http://gerrit.cloudera.org:8080/#/c/15483/39/src/kudu/ranger/mini_ranger-test.cc
File src/kudu/ranger/mini_ranger-test.cc:
http://gerrit.cloudera.org:8080/#/c/15483/39/src/kudu/ranger/mini_ranger-test.cc@74
PS39, Line 74: ASSERT_OK(ranger_.Start());
> Nit: add a using for this.
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@150
PS22, Line 150: return db_setup.WaitAndCheckExitCode();
> OK, future work then.
Done
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 40
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Mon, 23 Mar 2020 20:54:47 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#4).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
13 files changed, 2,248 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/4
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 4
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 25:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@195
PS22, Line 195: "-Duser=miniranger",
: Substitute("-Dranger.service.host=$0", fqdn),
: "-Dservername=miniranger",
: Substitute("-Dcatalina.base=$0", kEwsDir),
: Substitute("-Dlogdir=$0", JoinPathSegments(kAdminHome, "logs")),
: "-Dranger.audit.solr.bootstrap.enabled=false",
: "-cp", classpath, "org.apache.ranger.server.tomcat.EmbeddedServer"
: }));
: ranger_process_->SetEnvVars({
: { "XAPOLICYMGR_DIR", kAdminHome },
: { "XAPOLICYMGR_EWS_DIR", kEwsDir },
: { "RANGER_JAAS_LIB_DIR", JoinPathSegments(kWebAppDir, "ranger_jaas") },
: { "RANGER_JAAS_C
> https://github.com/apache/ranger/blob/master/embeddedwebserver/scripts/rang
Done
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 25
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Mon, 23 Mar 2020 06:35:21 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#51).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by Python's "2to3" script but it required
some manual changes as well to run on Python 3 (subprocesses return
bytes and they were compared with strings which wasn't a problem in
Python 2 but fails in Python 3 so the subprocess result had to be
decoded).
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin REST API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env.
Starting MiniRanger (fresh install) takes ~20s on my machine (2019
MacBook Pro) and ~30-40s on dist-test server.
Starting Ranger subprocess is not handled in this commit, it is going to
be done in a follow-up patch.
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,779 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/51
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 51
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#31).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,753 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/31
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 31
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 50:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/15483/50/src/kudu/postgres/mini_postgres.cc
File src/kudu/postgres/mini_postgres.cc:
http://gerrit.cloudera.org:8080/#/c/15483/50/src/kudu/postgres/mini_postgres.cc@99
PS50, Line 99: Status MiniPostgres::Stop() {
Stop() called twice in a row will SIGSEGV.
http://gerrit.cloudera.org:8080/#/c/15483/50/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/50/src/kudu/ranger/mini_ranger.cc@59
PS50, Line 59: Status MiniRanger::Stop() {
If you call this twice in a row, it'll SIGSEGV.
http://gerrit.cloudera.org:8080/#/c/15483/50/src/kudu/ranger/mini_ranger.cc@220
PS50, Line 220: LOG(INFO) << Substitute("Ranger started in $0ms",
: (MonoTime::Now() - start).ToMilliseconds());
If you want to measure the amount of time a block of code takes to run, use LOG_TIMING (from stopwatch.h).
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 50
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 25 Mar 2020 20:21:31 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#27).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/CMakeLists.txt
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,761 insertions(+), 32 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/27
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 27
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#16).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
20 files changed, 2,441 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/16
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 16
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#49).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by Python's "2to3" script but it required
some manual changes as well to run on Python 3 (subprocesses return
bytes and they were compared with strings which wasn't a problem in
Python 2 but fails in Python 3 so the subprocess result had to be
decoded).
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin REST API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env.
Starting MiniRanger (fresh install) takes ~20s on my machine (2019
MacBook Pro) and ~30-40s on dist-test server.
Starting Ranger subprocess is not handled in this commit, it is going to
be done in a follow-up patch.
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,776 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/49
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 49
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 10:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/15483/9/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/9/src/kudu/ranger/mini_ranger.h@38
PS9, Line 38: explicit MiniRanger(std::string data_root) :
> warning: single-argument constructors must be marked explicit to avoid unin
Done
http://gerrit.cloudera.org:8080/#/c/15483/9/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/9/src/kudu/ranger/mini_ranger.cc@55
PS9, Line 55: // TODO(abukor): Has some hardcoded values.
> warning: missing username/bug in TODO [google-readability-todo]
Done
http://gerrit.cloudera.org:8080/#/c/15483/9/src/kudu/ranger/mini_ranger.cc@56
PS9, Line 56: const string kInstallProperties = R"(
> warning: 'kInstallProperties' is a static definition in anonymous namespace
Done
http://gerrit.cloudera.org:8080/#/c/15483/9/src/kudu/ranger/mini_ranger.cc@240
PS9, Line 240: const string kRangerAdminSiteTemplate = R"(
> warning: 'kRangerAdminSiteTemplate' is a static definition in anonymous nam
Done
http://gerrit.cloudera.org:8080/#/c/15483/9/src/kudu/ranger/mini_ranger.cc@534
PS9, Line 534: const string kRangerAdminDefaultSiteTemplate = R"(
> warning: 'kRangerAdminDefaultSiteTemplate' is a static definition in anonym
Done
http://gerrit.cloudera.org:8080/#/c/15483/9/src/kudu/ranger/mini_ranger.cc@1119
PS9, Line 1119: const string kCoreSiteTemplate = R"(
> warning: 'kCoreSiteTemplate' is a static definition in anonymous namespace;
Done
http://gerrit.cloudera.org:8080/#/c/15483/9/src/kudu/ranger/mini_ranger.cc@1304
PS9, Line 1304: Status MiniRanger::CreateRangerConfigs(const string& config_string, const string& file_path) {
> warning: method 'CreateRangerConfigs' can be made static [readability-conve
Done
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 10
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Thu, 19 Mar 2020 18:13:12 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#20).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,744 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/20
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 20
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 51: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 51
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 25 Mar 2020 20:51:17 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#9).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action_test.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
15 files changed, 2,258 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/9
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 9
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#34).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,752 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/34
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 34
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#44).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,758 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/44
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 44
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#17).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,756 insertions(+), 29 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/17
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 17
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#15).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
20 files changed, 2,435 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/15
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 15
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#22).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,744 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/22
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 22
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/15483/3/src/kudu/ranger/CMakeLists.txt
File src/kudu/ranger/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/15483/3/src/kudu/ranger/CMakeLists.txt@53
PS3, Line 53:
> Does this still exist? I think the subprocess provides what you need.
Done
http://gerrit.cloudera.org:8080/#/c/15483/3/thirdparty/patches/ranger-fixscripts.patch
File thirdparty/patches/ranger-fixscripts.patch:
http://gerrit.cloudera.org:8080/#/c/15483/3/thirdparty/patches/ranger-fixscripts.patch@1
PS3, Line 1: *** ews/ranger-admin-services.sh.orig 2020-03-18 00:27:04.000000000 +0100
> Is there a Raner PR for this fix?
not for this one, this is a custom one, see commit message for details. do you think I should find a more generic approach and submit a PR?
http://gerrit.cloudera.org:8080/#/c/15483/3/thirdparty/patches/ranger-python3.patch
File thirdparty/patches/ranger-python3.patch:
http://gerrit.cloudera.org:8080/#/c/15483/3/thirdparty/patches/ranger-python3.patch@1
PS3, Line 1: --- ./update_property.py (original)
> Is there a Raner PR for this fix?
yep, https://reviews.apache.org/r/72244/ RANGER-2759
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 5
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 19 Mar 2020 00:15:31 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#37).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,752 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/37
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 37
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 16:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15483/15/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/15/src/kudu/ranger/mini_ranger.cc@1376
PS15, Line 1376:
> warning: the parameter 'policy' is copied for each invocation but only used
Done
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 16
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Fri, 20 Mar 2020 23:10:32 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#35).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,751 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/35
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 35
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#46).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by Python's "2to3" script but it required
some manual changes as well to run on Python 3 (subprocesses return
bytes and they were compared with strings which wasn't a problem in
Python 2 but fails in Python 3 so the subprocess result had to be
decoded).
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin REST API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Starting Ranger subprocess is not handled in this commit, it is going to
be done in a follow-up patch.
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,775 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/46
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 46
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#33).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,753 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/33
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 33
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 49:
Looks like this will need a rebase.
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 49
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 25 Mar 2020 17:30:00 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#36).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,752 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/36
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 36
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#24).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,757 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/24
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 24
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#12).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
18 files changed, 2,291 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/12
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 12
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#5).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
13 files changed, 2,235 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/5
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 5
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has uploaded a new patch set (#45) to the change originally created by Attila Bukor. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,757 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/45
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 45
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 51: Verified+1
failure was an unrelated flaky test
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 51
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 25 Mar 2020 21:32:32 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 51:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/15483/50/src/kudu/postgres/mini_postgres.cc
File src/kudu/postgres/mini_postgres.cc:
http://gerrit.cloudera.org:8080/#/c/15483/50/src/kudu/postgres/mini_postgres.cc@99
PS50, Line 99: Status MiniPostgres::Stop() {
> Stop() called twice in a row will SIGSEGV.
Done
http://gerrit.cloudera.org:8080/#/c/15483/50/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/50/src/kudu/ranger/mini_ranger.cc@59
PS50, Line 59:
> If you call this twice in a row, it'll SIGSEGV.
Done
http://gerrit.cloudera.org:8080/#/c/15483/50/src/kudu/ranger/mini_ranger.cc@220
PS50, Line 220: MonoDelta::FromMilliseconds(kRangerStartTimeoutMs)));
: LOG(INFO) << "Ranger bound to " << port;
> If you want to measure the amount of time a block of code takes to run, use
Done
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 51
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 25 Mar 2020 20:55:18 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#30).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,753 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/30
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 30
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#21).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,744 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/21
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 21
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 17:
(42 comments)
http://gerrit.cloudera.org:8080/#/c/15483/16/build-support/dist_test.py
File build-support/dist_test.py:
http://gerrit.cloudera.org:8080/#/c/15483/16/build-support/dist_test.py@116
PS16, Line 116: # Add Postgres and Ranger. These are symlinks to directories in thirdparty.
> Nit: terminate with a period.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/build-support/run_dist_test.py
File build-support/run_dist_test.py:
http://gerrit.cloudera.org:8080/#/c/15483/16/build-support/run_dist_test.py@165
PS16, Line 165: os.path.join(bin_path, "postgres"))
> Can you just merge this loop with the chrony one, and update the comment?
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/build-support/run_dist_test.py@176
PS16, Line 176: in(ROOT, "build/dist-test-system-libs/")]
> This was already done above (L154); reuse.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/mini-cluster/external_mini_cluster.cc
File src/kudu/mini-cluster/external_mini_cluster.cc:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/mini-cluster/external_mini_cluster.cc@337
PS16, Line 337: if (opts_.enable_ranger) {
> Where do we configure the Kudu Masters to talk to MiniRanger? Or is that go
I was planning to do that in a follow-up, we need to configure the subprocess for that. Should I include that in this patch?
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/CMakeLists.txt
File src/kudu/ranger/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/CMakeLists.txt@69
PS16, Line 69: "${EXECUTABLE_OUTPUT_PATH}/hadoop-home")
> What is this comment telling us? Is this a TODO?
not sure, this was copied from Andrew's abandoned patch. Removed it, it works without execute.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/CMakeLists.txt@100
PS16, Line 100:
> Alphabetical sorting.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger-test.cc
File src/kudu/ranger/mini_ranger-test.cc:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger-test.cc@40
PS16, Line 40: MiniRanger ranger_;
> Should be ranger_
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger-test.cc@41
PS16, Line 41: };
> Nit: empty line here.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger-test.cc@44
PS16, Line 44: PolicyItem item;
> We should at least a full lifecycle (start, restart, stop, etc)
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger-test.cc@52
PS16, Line 52: policy.name = "test1";
> std::move
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@40
PS16, Line 40: typedef std::pair<std::vector<std::string>, std::vector<ActionPB>> PolicyItem;
> This needs documentation, and should be called PolicyItem.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@42
PS16, Line 42: // The AuthorizationPolicy contains a set of privileges on a resource to one or
> Doc.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@50
PS16, Line 50: std::vector<PolicyItem> items;
> Doc.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@57
PS16, Line 57: GetT
> How did you get away with this? We shouldn't be 'using' anything in this he
No idea why the compiler didn't complain, but fixed it.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@64
PS16, Line 64:
> Doc.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@69
PS16, Line 69: // Adds a new policy to Ranger.
> Doc.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@94
PS16, Line 94: JoinPathSegments(ranger_home_, "ews"), java_home_,
> Doc.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@53
PS16, Line 53:
> What a mess. In future work, could you:
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1132
PS16, Line 1132:
> Does this warrant a TODO or something? I presume we want to support Kerbero
changed it to configurable
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1141
PS16, Line 1141:
> Yeah, this should be the next priority, otherwise the massive number of Sen
Apparently Postgres seems to be ready to use when the port is up so it works fine without this, so removed the todo and the wait.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1147
PS16, Line 1147:
> What if Start() failed mid-way such that mini_pg_ has started but Ranger ha
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1148
PS16, Line 1148:
> Should WARN_NOT_OK.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1157
PS16, Line 1157:
> Holy moly you can't just send the thing a TERM/KILL? What happens if you do
that seems to work too, changed it to send SIGTERM instead. No idea why Ranger service script chose to use this shutdown method.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1179
PS16, Line 1179:
:
:
> Do in the constructor and make data_root_ const?
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1213
PS16, Line 1213:
> Decompose this block into a helper.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1215
PS16, Line 1215:
> RETURN_NOT_OK.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1223
PS16, Line 1223:
> Shouldn't this be outside the "fresh install" case?
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1232
PS16, Line 1232:
> What about setting up Ranger on a UNIQUE_LOOPBACK address?
wouldn't work on MacOS
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1237
PS16, Line 1237:
> There should be a comment somewhere explaining all of the ports that we nee
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1238
PS16, Line 1238:
> Decompose all the config stuff into a helper.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1258
PS16, Line 1258:
> Decompose this block into a helper.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1261
PS16, Line 1261:
:
> Add a symlink function to env.h and env_posix.h (and a test to env-test.cc)
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1264
PS16, Line 1264:
> Link?
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1275
PS16, Line 1275:
> Doesn't need to be heap allocated.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1343
PS16, Line 1343:
:
:
:
:
:
:
:
:
> Decompose into a helper.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1357
PS16, Line 1357:
> Pass by value and std::move both of these.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1366
PS16, Line 1366:
> Duplicated from MiniPostgres; decompose into net_util somewhere.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.cc@1434
PS16, Line 1434:
> If there's an error, will it be reflected in the HTTP status code (i.e. wil
EasyCurl returns a RemoteError if the HTTP status code is not 2xx (fixed it now, it was 200 while all 2xx codes indicate success.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/util/curl_util.h
File src/kudu/util/curl_util.h:
PS16:
> Can you unit test this change? See PasswdWebserverTest in webserver-test (w
Did some refactoring of the tests, but it seems the webserver is using DIGEST authn (couldn't get BASIC to work properly at least) and Ranger is using BASIC. So I ended up refactoring EasyCurl a bit more, DIGEST and SPNEGO are tested by webserver-test, BASIC is only tested in MiniRanger.
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/util/curl_util.h@51
PS16, Line 51: // Fetch the given URL into the provided buffer.
: // Any existing data in the buffer is replaced.
> These are likely to be the same with each call to FetchURL or PostToURL; co
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/thirdparty/build-thirdparty.sh
File thirdparty/build-thirdparty.sh:
http://gerrit.cloudera.org:8080/#/c/15483/16/thirdparty/build-thirdparty.sh@298
PS16, Line 298: ln -nsf \
> This isn't a directory, so why -R?
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/thirdparty/patches/ranger-fixscripts.patch
File thirdparty/patches/ranger-fixscripts.patch:
PS16:
> The patch is obviously just a limited view into the script itself, but is i
Basically the script didn't allow setting XAPOLICYMGR_EWS_DIR directly and readlink -f doesn't work on Mac, so I opted to remove this part and set XAPOLICYMGR_EWS_DIR from Subprocess.
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 17
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Sun, 22 Mar 2020 17:33:31 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#23).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,781 insertions(+), 29 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/23
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 23
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 26:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.h@53
PS26, Line 53: on a resource
Is this the policy for a single resource? The vectors of databases, tables, and columns still isn't super clear. Are the privileges granted to the cross-product? E.g. If I have {db1, db2}->{table1, table2}->{column}, does that mean
- db1->table1->column
- db1->table2->column
- db2->table1->column
- db2->table2->column
?
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.h@54
PS26, Line 54: user
nit: users
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.h@76
PS26, Line 76: curl_.set_auth(CurlAuthType::BASIC, "admin", "admin");
: }
nit: remove a few spaces
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.h@126
PS26, Line 126: $$CLASSPATH
Will this be evaluated too?
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.cc@238
PS26, Line 238: Faled
nit: Failed
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.cc@235
PS26, Line 235: Status s = POSTToRanger("service/plugins/services", std::move(service), &result);
: if (s.IsRemoteError()) {
: return Status::RemoteError(
: Substitute("Faled to create Kudu service. Status: $0; message: $1",
: s.ToString(), result.ToString()));
: }
: return s;
Could probably just
RETURN_NOT_OK_PREPEND(POSTToRanger(...), Substitute("failed to create Kudu service: {result}));
return Status::OK();
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.cc@303
PS26, Line 303: POSTToRanger
nit: keep the casing consistent? PostToRanger?
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger.cc@303
PS26, Line 303: string
nit: const string&
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger_configs.h
File src/kudu/ranger/mini_ranger_configs.h:
http://gerrit.cloudera.org:8080/#/c/15483/26/src/kudu/ranger/mini_ranger_configs.h@64
PS26, Line 64: RPC
nit: RPCs
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/tools/tool.proto
File src/kudu/tools/tool.proto:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/tools/tool.proto@45
PS22, Line 45: // Whether or not Ranger should be enabled
: optional bool enable_ranger = 9;
:
> not sure why Sentry doesn't have one. I did this so I can test the REST API
Ah I see. I think virtually all of our usage of Sentry happened through unit tests. I don't mind keeping it in.
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 26
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Mon, 23 Mar 2020 08:29:21 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#38).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,752 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/38
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 38
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 23:
(13 comments)
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/CMakeLists.txt
File src/kudu/ranger/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/CMakeLists.txt@53
PS22, Line 53:
: execute_process(COMMAND ln -nsf
: "${CMAKE_SOURCE_DIR}/thirdparty/installed/common/bin"
: "${EXECUTABLE_OUTPUT_PATH}/postgres")
: execute_process(COMMAND ln -nsf
: "${CMAKE_SOURCE_DIR}/thirdparty/installed/common/lib"
: "${EXECUTABLE_OUTPUT_PATH}/postgres-lib")
: execute_process(COMMAND ln -nsf
: "${CMAKE_SOURCE_DIR}/thirdparty/installed/common/opt/jdbc/postgresql.jar"
: "${EXECUTABLE_OUTPUT_PATH}/postgres/")
Can this be removed now that Postgres is in its own module?
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.h@40
PS22, Line 40:
nit: maybe typedef this too as UserList so this is more self-documenting.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.h@47
PS22, Line 47:
: // The AuthorizationPolicy contains a set of privileges on a resource to one or
: // more user. 'items' is a vector of user-list of actions pair. This struct can
: // be used to create new Ranger
nit: it isn't clear upon reading the comment whether the vectors here should be all the same length (e.g. each size N to specify N database->table->column resources), or should have their cross-product taken. Could you clarify?
Alternatively, it seems like this was done to match the Ranger REST endpoint. Would something like this be closer to what Ranger exposes to users? It'd be nice to hide internal details of Ranger's more developer-facing endpoints, especially if it improves composability in tests.
// Strings may be real values to target specific resources, "*" to assign wildcard privileges, or empty.
struct RangerPrivilege {
string database;
string table;
string column;
vector<PolicyItem> actions; // btw maybe we should rename PolicyItem to UsersAndActions or something?
};
struct AuthorizationPolicy {
string policy_name;
vector<RangerPrivilege> privileges;
};
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@61
PS22, Line 61: ranger_process_
RETURN_NOT_OK
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@195
PS22, Line 195: // Set JAVA_HOME
: // Set ranger-admin-env* variables
: // - RANGER_PID_DIR_PATH = /tmp (probably don't need)
: // - RANGER_ADMIN_PID_NAME=rangeradmin.pid
: // - RANGER_USER
: // Set JAVA_OPTS and JAVA_HOME
: // RANGER_ADMIN_LOG_DIR=${XAPOLICYMGR_EWS_DIR}/logs
: // Set up pid file (probably don't need)
: // Run tomcat server
:
: string classpath = ranger_classpath();
:
: LOG(INFO) << "Using
Probably don't need these comments anymore, but it's probably worth linking to a specific hash of Ranger's ranger-admin-services.sh
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@234
PS22, Line 234: });
You can remove these :)
XXX(awong) comments are just notes to myself that I intend on addressing before pushing/de-wipping.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@256
PS22, Line 256: ststring result;
: Status s = POSTToRanger("service/plugins/services", std::move(service), &result);
: if (s.IsRemoteError()) {
: return Status::RemoteError(Substitute("Faled to create Kudu service. Status: $0; message: $1",
: s.ToString(), result.ToString()));
: }
: return s;
: }
:
nit: shift back a couple spaces
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger_configs.h
File src/kudu/ranger/mini_ranger_configs.h:
PS22:
Thanks for grokking and organizing these configs! Makes it significantly more understandable/maintainable :)
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger_configs.h@63
PS22, Line 63: external ranger port (this is the one that actually matters)
nit: could you reword this? Matters for what?
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger_configs.h@78
PS22, Line 78: 1
nit: reorder with $0?
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger_configs.h@139
PS22, Line 139: <property>
: <name>ranger.service.shutdown.port</name>
: <value>$1</value>
: </property>
Now that we're just using SIGTERM, do we still need this?
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger_configs.h@224
PS22, Line 224: jdbc:log4jdbc:mysql
postgresql? Does this actually matter?
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/tools/tool.proto
File src/kudu/tools/tool.proto:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/tools/tool.proto@45
PS22, Line 45: // Whether or not Ranger should be enabled
: optional bool enable_ranger = 9;
:
Hrm, why don't we have an enable_sentry? Will we actually use this?
FWIW it would've been nice to have this patch focus solely on getting MiniRanger up and running, before plumbing it into the mini-cluster, especially if there aren't mini-cluster tests that exercise this.
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 23
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Sun, 22 Mar 2020 23:43:27 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 47:
(3 comments)
> Patch Set 45:
>
> Something we'll want to address soon-ish: I ran mini_ranger-test 50 times and saw two failures with the following error to start Postgres:
>
> ../../src/kudu/ranger/mini_ranger-test.cc:74: Failure
> Failed
> Bad status: Runtime error: failed to start Postgres: /usr/bin/lsof: process exited with non-zero status 1
> I0324 03:44:25.806879 348 mini_ranger.cc:60] Stopping Ranger...
> F0324 03:44:25.806896 348 subprocess.cc:646] Sub-process is not running
> *** Check failure stack trace: ***
> *** Aborted at 1585021465 (unix time) try "date -d @1585021465" if you are using GNU date ***
> PC: @ 0x7f40050adc37 gsignal
> *** SIGABRT (@0x3e80000015c) received by PID 348 (TID 0x7f400d47d880) from PID 348; stack trace: ***
> @ 0x7f4005450330 (unknown) at ??:0
> @ 0x7f40050adc37 gsignal at ??:0
> @ 0x7f40050b1028 abort at ??:0
> @ 0x7f4006bd1a19 google::logging_fail() at ??:0
> @ 0x7f4006bd330d google::LogMessage::Fail() at ??:0
> @ 0x7f4006bd51cd google::LogMessage::SendToLog() at ??:0
> @ 0x7f4006bd2e49 google::LogMessage::Flush() at ??:0
> @ 0x7f4006bd30a1 google::LogMessage::~LogMessage() at ??:0
> @ 0x7f4009f5d38f kudu::Subprocess::KillAndWait() at ??:0
> @ 0x7f400ce6605d kudu::ranger::MiniRanger::Stop() at ??:0
> @ 0x7f400ce65eaf kudu::ranger::MiniRanger::~MiniRanger() at ??:0
> @ 0x41538a kudu::ranger::MiniRangerTest::~MiniRangerTest() at /data/8/awong/Repositories/kudu/build/debug/../../src/kudu/ranger/mini_ranger-test.cc:36
> @ 0x41a3ae kudu::ranger::MiniRangerTest_TestGrantSamePrivilegeAfterRestart_Test::~MiniRangerTest_TestGrantSamePrivilegeAfterRestart_Test() at /data/8/awong/Repositories/kudu/build/debug/../../src/kudu/ranger/mini_ranger-test.cc:60
> @ 0x41a3de kudu::ranger::MiniRangerTest_TestGrantSamePrivilegeAfterRestart_Test::~MiniRangerTest_TestGrantSamePrivilegeAfterRestart_Test() at /data/8/awong/Repositories/kudu/build/debug/../../src/kudu/ranger/mini_ranger-test.cc:60
> @ 0x7f400a7e6b98 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
> @ 0x7f400a7d42b5 testing::TestInfo::Run() at ??:0
> @ 0x7f400a7d43d5 testing::TestCase::Run() at ??:0
> @ 0x7f400a7daed8 testing::internal::UnitTestImpl::RunAllTests() at ??:0
> @ 0x7f400a7e70a8 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
> @ 0x7f400a7d44ad testing::UnitTest::Run() at ??:0
> @ 0x7f400d07bf5f RUN_ALL_TESTS() at ??:0
> @ 0x7f400d079d70 main at ??:0
> @ 0x7f4005098f45 __libc_start_main at ??:0
> @ 0x412d59 (unknown) at ??:?
thanks for the heads up, fixed it, ran this test suite with 100 iterations with no failures.
http://gerrit.cloudera.org:8080/#/c/15483/46//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15483/46//COMMIT_MSG@39
PS46, Line 39: Env
> nit: add a period
Done
http://gerrit.cloudera.org:8080/#/c/15483/46/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/46/src/kudu/ranger/mini_ranger.cc@151
PS46, Line 151: Startin
> nit: Starting
Done
http://gerrit.cloudera.org:8080/#/c/15483/46/src/kudu/ranger/mini_ranger.cc@288
PS46, Line 288: "Failed to add policy");
: return Status::OK();
> Seems this should be put into CreateKuduService() instead
Done
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 47
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Tue, 24 Mar 2020 21:49:17 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#43).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,756 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/43
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 43
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#39).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,755 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/39
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 39
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#26).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,760 insertions(+), 29 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/26
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 26
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#48).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by Python's "2to3" script but it required
some manual changes as well to run on Python 3 (subprocesses return
bytes and they were compared with strings which wasn't a problem in
Python 2 but fails in Python 3 so the subprocess result had to be
decoded).
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin REST API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env.
Starting MiniRanger (fresh install) takes ~20s on my machine (2019
MacBook Pro) and ~40s on dist-test server.
Starting Ranger subprocess is not handled in this commit, it is going to
be done in a follow-up patch.
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,776 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/48
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 48
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#50).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by Python's "2to3" script but it required
some manual changes as well to run on Python 3 (subprocesses return
bytes and they were compared with strings which wasn't a problem in
Python 2 but fails in Python 3 so the subprocess result had to be
decoded).
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin REST API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env.
Starting MiniRanger (fresh install) takes ~20s on my machine (2019
MacBook Pro) and ~30-40s on dist-test server.
Starting Ranger subprocess is not handled in this commit, it is going to
be done in a follow-up patch.
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,776 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/50
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 50
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#14).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
20 files changed, 2,437 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/14
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 14
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#29).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
26 files changed, 1,753 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/29
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 29
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15483
to look at the new patch set (#18).
Change subject: KUDU-3079 Add MiniRanger
......................................................................
KUDU-3079 Add MiniRanger
Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].
As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).
Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by "2to3" but it required some manual changes
as well to run on Python 3.
I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].
The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin RESY API when creating the Kudu service after
startup and adding policies.
Also adds a CreateSymLink() method to Env
Thanks to Andrew Wong for his help with the Ranger setup[5].
[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5
Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
---
M build-support/dist_test.py
M build-support/run_dist_test.py
M src/kudu/mini-cluster/CMakeLists.txt
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/postgres/mini_postgres.cc
M src/kudu/ranger/CMakeLists.txt
A src/kudu/ranger/mini_ranger-test.cc
A src/kudu/ranger/mini_ranger.cc
A src/kudu/ranger/mini_ranger.h
A src/kudu/ranger/mini_ranger_configs.h
M src/kudu/security/test/test_pass.h
M src/kudu/server/webserver-test.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action.h
M src/kudu/tools/tool_action_test.cc
M src/kudu/util/curl_util.cc
M src/kudu/util/curl_util.h
M src/kudu/util/env-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M thirdparty/LICENSE.txt
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/patches/ranger-fixscripts.patch
A thirdparty/patches/ranger-python3.patch
M thirdparty/vars.sh
27 files changed, 1,755 insertions(+), 29 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/15483/18
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 18
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 51: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 51
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Wed, 25 Mar 2020 21:36:35 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 23:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger-test.cc
File src/kudu/ranger/mini_ranger-test.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger-test.cc@75
PS22, Line 75: const std::string kExpectedError = "Another policy already exists for matching resource";
> Anything we can check in the error message to prove to ourselves that "poli
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.h@72
PS22, Line 72: }
> Please doc all of the non-accessor methods.
Done
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/16/src/kudu/ranger/mini_ranger.h@40
PS16, Line 40:
> No doc?
sorry, missed the part about the doc
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@91
PS22, Line 91: necessary to set it to a random value to avoid collisions in
: // parallel testing.
: RETURN_NOT_OK(GetRando
> Not possible to tell Ranger not to bind to it at all?
Doesn't seem so based on the code. If we just remove that part it will bind to the default port (6185).
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@131
PS22, Line 131: web_app_dir.c_str()));
> This seems to be repeated a bunch; how about create a single env_ at MiniRa
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@150
PS22, Line 150: db_setup.SetEnvVars({
> How slow? Anything we can do on the postgres side (some reconfiguration per
Around 8 seconds. I don't think Postgres is the bottleneck here though. Also, the Ranger startup itself (after db is set up) is another ~9 seconds and I have no idea if there's anything we can do about it.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@316
PS22, Line 316: if (s.IsRemoteError()) {
: return Status::RemoteError(Substitute("Failed to add policy. Status: $0; message: $1",
: s.ToString(), result.ToString()));
: }
:
: return s;
: }
> CreateKuduService shares a lot of this. Decompose into a (parameterized) he
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.h
File src/kudu/util/curl_util.h:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.h@61
PS22, Line 61: // The optional param 'headers' holds additional headers.
> Doc 'headers'.
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.h@143
PS22, Line 143: char errbuf_[kErrBufSize];
:
: std::string username_;
> Don't need the initializers.
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.cc
File src/kudu/util/curl_util.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.cc@125
PS22, Line 125: switch (auth_type_) {
> Set this in the default case of the switch below.
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/util/curl_util.cc@126
PS22, Line 126: case CurlAuthType::SPNEGO:
> Should add a default so that the compiler doesn't warn.
Done
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 23
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Sun, 22 Mar 2020 23:39:43 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 24:
(13 comments)
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/CMakeLists.txt
File src/kudu/ranger/CMakeLists.txt:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/CMakeLists.txt@53
PS22, Line 53:
: execute_process(COMMAND ln -nsf
: "${CMAKE_SOURCE_DIR}/thirdparty/installed/common/opt/ranger"
: "${EXECUTABLE_OUTPUT_PATH}/ranger-home")
: execute_process(COMMAND ln -nsf
: "${CMAKE_SOURCE_DIR}/thirdparty/installed/common/opt/hadoop"
: "${EXECUTABLE_OUTPUT_PATH}/hadoop-home")
: execute_process(COMMAND ln -nsf
: "${JAVA_HOME}"
: "${EXECUTABLE_OUTPUT_PATH}/java-home")
> Can this be removed now that Postgres is in its own module?
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.h
File src/kudu/ranger/mini_ranger.h:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.h@40
PS22, Line 40:
> nit: maybe typedef this too as UserList so this is more self-documenting.
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.h@47
PS22, Line 47: // AuthorizationPolicy. Number of users and actions doesn't have to match, their
: // cross-product is taken.
: typedef std::pair<UserList, std::vector<ActionPB>> PolicyItem;
:
> nit: it isn't clear upon reading the comment whether the vectors here shoul
Added a clarification. Your model is a bit off, each policy can have multiple databases/tables/columns and the users/actions product is not tied to a single one of them, but all of them in the same policy.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@61
PS22, Line 61: RETURN_NOT_OK(r
> RETURN_NOT_OK
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@195
PS22, Line 195: "-Dservername=miniranger",
: Substitute("-Dcatalina.base=$0", kEwsDir),
: Substitute("-Dlogdir=$0", JoinPathSegments(kAdminHome, "logs")),
: "-Dranger.audit.solr.bootstrap.enabled=false",
: "-cp", classpath, "org.apache.ranger.server.tomcat.EmbeddedServer"
: }));
: ranger_process_->SetEnvVars({
: { "XAPOLICYMGR_DIR", kAdminHome },
: { "XAPOLICYMGR_EWS_DIR", kEwsDir },
: { "RANGER_JAAS_LIB_DIR", JoinPathSegments(kWebAppDir, "ranger_jaas") },
: { "RANGER_JAAS_CONF_DIR", JoinPathSegments(kConfDir, "ranger_jaas") },
: { "JAVA_HOME", java_home_ },
: { "RANGER_PID_DI
> Probably don't need these comments anymore, but it's probably worth linking
interestingly I can't find ranger-admin-services.sh in the repo, not sure where it's coming from.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@234
PS22, Line 234: if (s.IsRemoteError()) {
> You can remove these :)
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger.cc@256
PS22, Line 256:
:
: if (!policy.tables.empty()) {
: EasyJson tables = resources.Set("table", EasyJson::kObject);
: EasyJson table_values = tables.Set("values", EasyJson::kArray);
: for (const string& table : policy.tables) {
: table_values.PushBack(table);
: }
: }
> nit: shift back a couple spaces
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger_configs.h
File src/kudu/ranger/mini_ranger_configs.h:
PS22:
> Thanks for grokking and organizing these configs! Makes it significantly mo
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger_configs.h@63
PS22, Line 63: postgres port
> nit: could you reword this? Matters for what?
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger_configs.h@78
PS22, Line 78: 0
> nit: reorder with $0?
Done
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger_configs.h@139
PS22, Line 139: <property>
: <name>ranger.service.shutdown.port</name>
: <value>$1</value>
: </property>
> Now that we're just using SIGTERM, do we still need this?
unfortunately yes, otherwise it would just bind to 6185 and cause collisions in tests
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/ranger/mini_ranger_configs.h@224
PS22, Line 224: jdbc:log4jdbc:mysql
> postgresql? Does this actually matter?
it doesn't matter, but it fails to start without it as it still tries to resolve ranger.jpa.audit.jdbc.url, so I just left it on default.
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/tools/tool.proto
File src/kudu/tools/tool.proto:
http://gerrit.cloudera.org:8080/#/c/15483/22/src/kudu/tools/tool.proto@45
PS22, Line 45: // Whether or not Ranger should be enabled
: optional bool enable_ranger = 9;
:
> Hrm, why don't we have an enable_sentry? Will we actually use this?
not sure why Sentry doesn't have one. I did this so I can test the REST API using mini cluster instead of waiting in a test. Should I break it out?
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 24
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Mon, 23 Mar 2020 00:57:29 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3079 Add MiniRanger
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15483 )
Change subject: KUDU-3079 Add MiniRanger
......................................................................
Patch Set 46: Code-Review+1
(3 comments)
http://gerrit.cloudera.org:8080/#/c/15483/46//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15483/46//COMMIT_MSG@39
PS46, Line 39: Env
nit: add a period
http://gerrit.cloudera.org:8080/#/c/15483/46/src/kudu/ranger/mini_ranger.cc
File src/kudu/ranger/mini_ranger.cc:
http://gerrit.cloudera.org:8080/#/c/15483/46/src/kudu/ranger/mini_ranger.cc@151
PS46, Line 151: Startin
nit: Starting
http://gerrit.cloudera.org:8080/#/c/15483/46/src/kudu/ranger/mini_ranger.cc@288
PS46, Line 288:
: LOG(INFO) << "Created Kudu service";
Seems this should be put into CreateKuduService() instead
--
To view, visit http://gerrit.cloudera.org:8080/15483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Gerrit-Change-Number: 15483
Gerrit-PatchSet: 46
Gerrit-Owner: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Tue, 24 Mar 2020 21:08:13 +0000
Gerrit-HasComments: Yes