You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by mmiklavc <gi...@git.apache.org> on 2016/11/21 22:17:45 UTC
[GitHub] incubator-metron pull request #365: METRON-565: apps/metron/enrichment/index...
GitHub user mmiklavc opened a pull request:
https://github.com/apache/incubator-metron/pull/365
METRON-565: apps/metron/enrichment/indexed directory path does not get created for metron cluster deployed via Ambari
This PR addresses https://issues.apache.org/jira/browse/METRON-565
Modified the parameters file to point to the correct path for indexing.
**Testing**
https://github.com/apache/incubator-metron/pull/364#issue-190758723
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mmiklavc/incubator-metron METRON-565
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/365.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #365
----
commit d8d4505c1d361a86227662d099f20dc68d6061fb
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-11-21T22:12:54Z
METRON-565: apps/metron/enrichment/indexed directory path does not get created for metron cluster deployed via Ambari
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #365: METRON-565: apps/metron/enrichment/indexed dire...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/365
Ended up making a few additional changes here. First, got rid of the /apps/metron/enrichment directory entirely. With indexing being split off into its own topology, this no longer made sense to live there. Furthermore, "indexed" was the only thing that lived in the enrichment directory in hdfs. The indexed doc path will now live in /apps/metron/indexing/indexed to keep consistent with the dir structure we used in the past. In addition, I've now exposed the directory as a property the user can both see and modify in the advanced properties section in Ambari.
Lastly, I updated the quickdev environment to reflect this change as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #365: METRON-565: apps/metron/enrichment/index...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/365#discussion_r89035528
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.3.0/configuration/metron-env.xml ---
@@ -246,7 +246,7 @@ bolt.hdfs.file.system.url={{ default_fs }}
bolt.hdfs.wip.file.path=/paloalto/wip
bolt.hdfs.finished.file.path=/paloalto/rotated
bolt.hdfs.compression.codec.class=org.apache.hadoop.io.compress.SnappyCodec
-index.hdfs.output={{ metron_apps_enrichment_dir }}
--- End diff --
That is a fair point, since we do expose the metron-env in Ambari. I'm willing to leave it separately, although it does make me wonder if there's a better way to handle configs like that in general. Having said that, I don't see a good way around it based on how Ambari needs things defined.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #365: METRON-565: apps/metron/enrichment/index...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/365#discussion_r89033125
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.3.0/configuration/metron-env.xml ---
@@ -246,7 +246,7 @@ bolt.hdfs.file.system.url={{ default_fs }}
bolt.hdfs.wip.file.path=/paloalto/wip
bolt.hdfs.finished.file.path=/paloalto/rotated
bolt.hdfs.compression.codec.class=org.apache.hadoop.io.compress.SnappyCodec
-index.hdfs.output={{ metron_apps_enrichment_dir }}
--- End diff --
But that param/value is hidden from the user right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #365: METRON-565: apps/metron/enrichment/indexed dire...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/365
Yes, appears to be an intermittent timing problem. If you re-run, I assume it would pass.
In METRON-575 I changed how these tests are validated. I assume that would help prevent such a thing from happening in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #365: METRON-565: apps/metron/enrichment/index...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/365#discussion_r89032519
--- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.3.0/configuration/metron-env.xml ---
@@ -246,7 +246,7 @@ bolt.hdfs.file.system.url={{ default_fs }}
bolt.hdfs.wip.file.path=/paloalto/wip
bolt.hdfs.finished.file.path=/paloalto/rotated
bolt.hdfs.compression.codec.class=org.apache.hadoop.io.compress.SnappyCodec
-index.hdfs.output={{ metron_apps_enrichment_dir }}
--- End diff --
I agree that this works, but could we do
`index.hdfs.output={{ metron_apps_enrichment_dir }}/indexing`
or even defined in the config, e.g.
`index.hdfs.output={{ metron_indexing_hdfs_dir }}/indexing`
I don't have a particularly strong opinion on constructing it in the params vs metron-env, but we should be using metron_apps_enrichment_dir since we already calculate it in the params anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #365: METRON-565: apps/metron/enrichment/indexed dire...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/365
I'm +1 on this, good stuff. @justinleet - you still gtg with the additional changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #365: METRON-565: apps/metron/enrichment/indexed dire...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/365
Yep, I'm +1, too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #365: METRON-565: apps/metron/enrichment/indexed dire...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/365
The Travis failure appears to be an unrelated profiler integration test error, although I'd like @mmiklavc to double-check. Assuming it is unrelated, I'm +1, by inspection. Thanks for picking this up!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #365: METRON-565: apps/metron/enrichment/index...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/365
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #365: METRON-565: apps/metron/enrichment/indexed dire...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/365
@mmiklavc Looks like https://github.com/apache/incubator-metron/pull/362 for METRON-575 got pulled in yesterday. Could you either kick this PR to get it to rerun and hopefully run correctly, or just merge in master so that Travis runs as expected?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #365: METRON-565: apps/metron/enrichment/indexed dire...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/365
@nickwallen can you take a look and verify?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---