You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2017/09/14 19:50:05 UTC
[GitHub] metron pull request #759: METRON-1187 Indexing/Profiler Kafka ACL Groups Not...
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/759
METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly
The Profiler MPack mistakenly uses the wrong flag/guard file to indicate that the Kafka ACL group has been setup. Whichever component (either Profiler or Indexing) that is executed first will complete successfully. The component to run next will not perform this setup task because the duplicated flag/guard file indicates that the setup was already completed successfully.
Most of the changes in this PR enhance the existing logging to help in the future when tracking down an issue like this. The actual fix is a one line.
### Testing
To test this change, you need to kerberize and environment and then ensure that both the Profiler and Indexing topologies are successfully running and consuming data after kerberization.
- [ ] Tested in Full Dev
- [ ] Tested in multi-node cluster
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1187
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/759.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #759
----
commit 5d9dba4ef57f3052697df1f156fc5a48d5ee7577
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-09-14T19:44:36Z
METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly
----
---
[GitHub] metron pull request #759: METRON-1187 Indexing/Profiler Kafka ACL Groups Not...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/759
---
[GitHub] metron issue #759: METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/759
Thanks to @anandsubbu and @justinleet for the assistance on this one. This was tough to track down.
---
[GitHub] metron issue #759: METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup ...
Posted by anandsubbu <gi...@git.apache.org>.
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/759
+1 (non-binding)
Indeed, this was a tricky one. Thank you @nickwallen for your root cause and fix!! I spun up a 12 node cluster with this patch, kerberized the cluster and confirmed that all of the topologies come up fine (esp. indexing and profiler).
---
[GitHub] metron pull request #759: METRON-1187 Indexing/Profiler Kafka ACL Groups Not...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/759#discussion_r138994125
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py ---
@@ -273,7 +272,7 @@
profiler_hbase_table = config['configurations']['metron-profiler-env']['profiler_hbase_table']
profiler_hbase_cf = config['configurations']['metron-profiler-env']['profiler_hbase_cf']
profiler_configured_flag_file = status_params.profiler_configured_flag_file
-profiler_acl_configured_flag_file = status_params.indexing_acl_configured_flag_file
+profiler_acl_configured_flag_file = status_params.profiler_acl_configured_flag_file
--- End diff --
This is the crux of the problem. A simple copy-paste error.
---
[GitHub] metron issue #759: METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup ...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/759
+1 by inspection. This is a good catch to have made, thanks!
---
[GitHub] metron pull request #759: METRON-1187 Indexing/Profiler Kafka ACL Groups Not...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/759#discussion_r139013691
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py ---
@@ -46,6 +46,7 @@ def __init__(self, params):
self.__acl_configured = os.path.isfile(self.__params.indexing_acl_configured_flag_file)
self.__hbase_configured = os.path.isfile(self.__params.indexing_hbase_configured_flag_file)
self.__hbase_acl_configured = os.path.isfile(self.__params.indexing_hbase_acl_configured_flag_file)
+ self.__hdfs_perm_configured = os.path.isfile(self.__params.indexing_hdfs_perm_configured_flag_file)
--- End diff --
This is another issue that I found. I am not sure what the impact is, but seems like we would just always set the HDFS perms.
---