You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2017/02/01 16:35:25 UTC
[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/432
METRON-682: Unify and Improve the Flat File Loader
Currently the flat file loader is deficient in a couple ways:
* It only supports importing local data despite there being a separate, poorly named, application which supports importing enrichment via MapReduce called threat_intel_loader.sh
* It does not support local imports from HDFS
* It does not support local imports from URLs
* It does not support importing zipped archives locally
* You cannot import more than one file at once
This JIRA will:
* Unify the MapReduce and local imports into one program and allow the user to specify the import mode with a CLI flag
* Support local imports from HDFS and URLs
* Support local imports from zipped files
* Support importing more than one file at once
Testing plan in comments
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron unified_loader
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/432.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #432
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/432#discussion_r99423987
--- Diff: metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/flatfile/importer/MapReduceImporter.java ---
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.dataloads.nonbulk.flatfile.importer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.mapreduce.TableOutputFormat;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.metron.dataloads.extractor.ExtractorHandler;
+import org.apache.metron.dataloads.hbase.mr.BulkLoadMapper;
+import org.apache.metron.dataloads.nonbulk.flatfile.LoadOptions;
+import org.apache.metron.enrichment.converter.EnrichmentConverter;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+
+public enum MapReduceImporter implements Importer{
+ INSTANCE
+ ;
+
+ @Override
+ public void importData(EnumMap<LoadOptions, Optional<Object>> config
+ , ExtractorHandler handler
+ , Configuration hadoopConfig
+ ) throws IOException {
+ String table = (String) config.get(LoadOptions.HBASE_TABLE).get();
+ String cf = (String) config.get(LoadOptions.HBASE_CF).get();
+ String extractorConfigContents = (String) config.get(LoadOptions.EXTRACTOR_CONFIG).get();
+ Job job = Job.getInstance(hadoopConfig);
+ List<String> inputs = (List<String>) config.get(LoadOptions.INPUT).get();
+ job.setJobName("MapReduceImporter: " + inputs.stream().collect(Collectors.joining(",")) + " => " + table + ":" + cf);
+ System.out.println("Configuring " + job.getJobName());
--- End diff --
good catch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/432
Works like a champ. Nice Job +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/432
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/432
I know it seems like a lot of code changed, but a lot of this was reorganizing the flat file loader class and splitting it into separate reusable components, rather than new code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/432
METRON-682: Unify and Improve the Flat File Loader
Currently the flat file loader is deficient in a couple ways:
* It only supports importing local data despite there being a separate, poorly named, application which supports importing enrichment via MapReduce called threat_intel_loader.sh
* It does not support local imports from HDFS
* It does not support local imports from URLs
* It does not support importing zipped archives locally
* You cannot import more than one file at once
This JIRA will:
* Unify the MapReduce and local imports into one program and allow the user to specify the import mode with a CLI flag
* Support local imports from HDFS and URLs
* Support local imports from zipped files
* Support importing more than one file at once
Testing plan in comments
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron unified_loader
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/432.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #432
----
commit 47d814ef95d67738d20ce5dc530ba7b05d418a96
Author: cstella <ce...@gmail.com>
Date: 2017-01-27T23:15:44Z
Multithreading the SimpleEnrichmentFlatFileLoader
commit 918d4ce4aea5d7dfde992f32bf049c70f35dd182
Author: cstella <ce...@gmail.com>
Date: 2017-01-27T23:23:19Z
doc changes.
commit c6ca3a86881eb77bc9598a61e3c0cf8280ccb03f
Author: cstella <ce...@gmail.com>
Date: 2017-01-27T23:39:56Z
Updating docs.
commit 8c9a79cdfa38ea2fbd161095d5e346147558ec5f
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T03:36:31Z
Investigating integration tests.
commit 315bd181aa634290ab987441d81c28addb7952e2
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T04:09:28Z
Update integration test to be a proper integration test.
commit 004c6f41b6c1cc3ecea70513e1a468501bd32e3c
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T04:49:37Z
Adding spliterator unit test for completeness
commit f8dd48ef920c948e1fc5ff736e386f641e551b2b
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T05:01:42Z
Updating test to use a proper file
commit 9b04f9723d442c8f4fb7a8bcaa1d733fc1305dc4
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T05:17:12Z
Updating docs and renaming a few things.
commit eb5b82cc35bd767a169f548ea8144dd9ae165f84
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T05:23:25Z
Update one more test case.
commit 310c98bd946b2fdb320193cce85d368f016bf8c3
Author: cstella <ce...@gmail.com>
Date: 2017-01-30T20:36:23Z
Merge branch 'master' into unified_loader
commit 3f6e3ba4f30e41c94ff25027f1fd7c839ea6c9bf
Author: cstella <ce...@gmail.com>
Date: 2017-01-31T15:39:03Z
Updating simple enrichment flat file loader to be complete.
commit 79cfdb4fba5e82e9e170bfc77c7133e6646f9787
Author: cstella <ce...@gmail.com>
Date: 2017-01-31T22:12:05Z
Removing old threatintel_bulk_load.sh script and integrating into the flatfile load script
commit bf7756b52e66907ca23a576ba9be9ab40b33f77d
Author: cstella <ce...@gmail.com>
Date: 2017-01-31T22:22:17Z
Forgot licenses.
commit a104f464e6b882121c7ab44079a5570d282c8457
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T00:28:46Z
updating script.
commit b121e13d892834865847ddd806cbf10da63fa44e
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T00:34:28Z
Merge branch 'master' into unified_loader
commit b5a9e5a9243576b27d59e959dfab3e99d34eb761
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T00:57:02Z
Added gzip and zip to regular files
commit 323267ddfb52ab1aa7488e02643a8158044797e2
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T15:04:53Z
Fixed stupid zip issue.
commit bc26b5b3992b91097bb4fc4b214d4b6bacaddfbb
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T16:27:58Z
Updating readme and making progress bar optional and better.
commit 6cdf35d94f72be7da524fd5f854876f131ddb9f9
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T17:39:59Z
updating tests to include gzip and zip
commit 799811ccd587d9fcd2135ea3970d5bbf1121a29b
Author: cstella <ce...@gmail.com>
Date: 2017-02-03T22:21:34Z
Reacted to @mmiklavcic
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/432
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/432
# Testing Plan
## Preliminaries
* Download the alexa 1m dataset:
```
wget http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
unzip top-1m.csv.zip
```
* Stage import files
```
head -n 10000 top-1m.csv > top-10k.csv
hadoop fs -put top-10k.csv /tmp
head -n 10000 top-1m.csv | gzip - > top-10k.csv.gz
head -n 10000 top-1m.csv | zip > top-10k.csv.zip
```
* Create an extractor.json for the CSV data by editing `extractor.json` and pasting in these contents:
```
{
"config" : {
"columns" : {
"domain" : 1,
"rank" : 0
}
,"indicator_column" : "domain"
,"type" : "alexa"
,"separator" : ","
},
"extractor" : "CSV"
}
```
## Import from URL
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase from URL. This should take approximately 5 or 6 minutes
/usr/metron/0.3.0/bin/flatfile_loader.sh -i http://s3.amazonaws.com/alexa-static/top-1m.csv.zip -t enrichment -c t -e ./extractor.json -p 5 -b 128
# count data written and verify it's 1M
echo "count 'enrichment'" | hbase shell
```
## Import from local file (non-zipped)
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-10k.csv -t enrichment -c t -e ./extractor.json -p 5 -b 128
# count data written and verify it's 10k
echo "count 'enrichment'" | hbase shell
```
## Import from local file (gzipped)
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-10k.csv.gz -t enrichment -c t -e ./extractor.json -p 5 -b 128
# count data written and verify it's 10k
echo "count 'enrichment'" | hbase shell
```
## Import from local file (zipped)
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase
/usr/metron/0.3.0/bin/flatfile_loader.sh -i ./top-10k.csv.zip -t enrichment -c t -e ./extractor.json -p 5 -b 128
# count data written and verify it's 10k
echo "count 'enrichment'" | hbase shell
```
## Import from HDFS via MR
```
# truncate hbase
echo "truncate 'enrichment'" | hbase shell
# import data into hbase
/usr/metron/0.3.0/bin/flatfile_loader.sh -i /tmp/top-10k.csv -t enrichment -c t -e ./extractor.json -m MR
# count data written and verify it's 10k
echo "count 'enrichment'" | hbase shell
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/432#discussion_r99421627
--- Diff: metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/flatfile/importer/MapReduceImporter.java ---
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.dataloads.nonbulk.flatfile.importer;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.mapreduce.TableOutputFormat;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.metron.dataloads.extractor.ExtractorHandler;
+import org.apache.metron.dataloads.hbase.mr.BulkLoadMapper;
+import org.apache.metron.dataloads.nonbulk.flatfile.LoadOptions;
+import org.apache.metron.enrichment.converter.EnrichmentConverter;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+
+public enum MapReduceImporter implements Importer{
+ INSTANCE
+ ;
+
+ @Override
+ public void importData(EnumMap<LoadOptions, Optional<Object>> config
+ , ExtractorHandler handler
+ , Configuration hadoopConfig
+ ) throws IOException {
+ String table = (String) config.get(LoadOptions.HBASE_TABLE).get();
+ String cf = (String) config.get(LoadOptions.HBASE_CF).get();
+ String extractorConfigContents = (String) config.get(LoadOptions.EXTRACTOR_CONFIG).get();
+ Job job = Job.getInstance(hadoopConfig);
+ List<String> inputs = (List<String>) config.get(LoadOptions.INPUT).get();
+ job.setJobName("MapReduceImporter: " + inputs.stream().collect(Collectors.joining(",")) + " => " + table + ":" + cf);
+ System.out.println("Configuring " + job.getJobName());
--- End diff --
Why system out instead of a logger?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/432#discussion_r99420892
--- Diff: metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/flatfile/OptionHandler.java ---
@@ -0,0 +1,31 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.dataloads.nonbulk.flatfile;
+
+import com.google.common.base.Function;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+
+import java.util.Optional;
+
+public abstract class OptionHandler implements Function<String, Option>
--- End diff --
Would this be useful in metron-common? Doesn't have to be this PR, but maybe "LoadOptions" becomes a parameterized type OPT_TYPE.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/432
METRON-682: Unify and Improve the Flat File Loader
Currently the flat file loader is deficient in a couple ways:
* It only supports importing local data despite there being a separate, poorly named, application which supports importing enrichment via MapReduce called threat_intel_loader.sh
* It does not support local imports from HDFS
* It does not support local imports from URLs
* It does not support importing zipped archives locally
* You cannot import more than one file at once
This JIRA will:
* Unify the MapReduce and local imports into one program and allow the user to specify the import mode with a CLI flag
* Support local imports from HDFS and URLs
* Support local imports from zipped files
* Support importing more than one file at once
Testing plan in comments
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron unified_loader
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/432.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #432
----
commit 47d814ef95d67738d20ce5dc530ba7b05d418a96
Author: cstella <ce...@gmail.com>
Date: 2017-01-27T23:15:44Z
Multithreading the SimpleEnrichmentFlatFileLoader
commit 918d4ce4aea5d7dfde992f32bf049c70f35dd182
Author: cstella <ce...@gmail.com>
Date: 2017-01-27T23:23:19Z
doc changes.
commit c6ca3a86881eb77bc9598a61e3c0cf8280ccb03f
Author: cstella <ce...@gmail.com>
Date: 2017-01-27T23:39:56Z
Updating docs.
commit 8c9a79cdfa38ea2fbd161095d5e346147558ec5f
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T03:36:31Z
Investigating integration tests.
commit 315bd181aa634290ab987441d81c28addb7952e2
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T04:09:28Z
Update integration test to be a proper integration test.
commit 004c6f41b6c1cc3ecea70513e1a468501bd32e3c
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T04:49:37Z
Adding spliterator unit test for completeness
commit f8dd48ef920c948e1fc5ff736e386f641e551b2b
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T05:01:42Z
Updating test to use a proper file
commit 9b04f9723d442c8f4fb7a8bcaa1d733fc1305dc4
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T05:17:12Z
Updating docs and renaming a few things.
commit eb5b82cc35bd767a169f548ea8144dd9ae165f84
Author: cstella <ce...@gmail.com>
Date: 2017-01-28T05:23:25Z
Update one more test case.
commit 310c98bd946b2fdb320193cce85d368f016bf8c3
Author: cstella <ce...@gmail.com>
Date: 2017-01-30T20:36:23Z
Merge branch 'master' into unified_loader
commit 3f6e3ba4f30e41c94ff25027f1fd7c839ea6c9bf
Author: cstella <ce...@gmail.com>
Date: 2017-01-31T15:39:03Z
Updating simple enrichment flat file loader to be complete.
commit 79cfdb4fba5e82e9e170bfc77c7133e6646f9787
Author: cstella <ce...@gmail.com>
Date: 2017-01-31T22:12:05Z
Removing old threatintel_bulk_load.sh script and integrating into the flatfile load script
commit bf7756b52e66907ca23a576ba9be9ab40b33f77d
Author: cstella <ce...@gmail.com>
Date: 2017-01-31T22:22:17Z
Forgot licenses.
commit a104f464e6b882121c7ab44079a5570d282c8457
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T00:28:46Z
updating script.
commit b121e13d892834865847ddd806cbf10da63fa44e
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T00:34:28Z
Merge branch 'master' into unified_loader
commit b5a9e5a9243576b27d59e959dfab3e99d34eb761
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T00:57:02Z
Added gzip and zip to regular files
commit 323267ddfb52ab1aa7488e02643a8158044797e2
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T15:04:53Z
Fixed stupid zip issue.
commit bc26b5b3992b91097bb4fc4b214d4b6bacaddfbb
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T16:27:58Z
Updating readme and making progress bar optional and better.
commit 6cdf35d94f72be7da524fd5f854876f131ddb9f9
Author: cstella <ce...@gmail.com>
Date: 2017-02-01T17:39:59Z
updating tests to include gzip and zip
commit 799811ccd587d9fcd2135ea3970d5bbf1121a29b
Author: cstella <ce...@gmail.com>
Date: 2017-02-03T22:21:34Z
Reacted to @mmiklavcic
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/432
Also, I'll point out that you can make your life easier and kill pretty much everything on your vagrant and do this. The only reliance is on HBase and MR.
I would suggest killing:
* monit via `service monit stop`
* all the storm topologies via `storm kill bro && storm kill snort && storm kill enrichment && storm kill indexing`
* tcpreplay via `for i in $(ps -ef | grep tcpreplay | awk '{print $2}');do kill -9 $i;done`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/432#discussion_r99420193
--- Diff: metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/extractor/inputformat/InputFormatHandler.java ---
@@ -17,12 +17,17 @@
*/
package org.apache.metron.dataloads.extractor.inputformat;
+import com.google.common.collect.ImmutableList;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.Job;
import java.io.IOException;
+import java.util.List;
import java.util.Map;
public interface InputFormatHandler {
- void set(Job job, Path input, Map<String, Object> config) throws IOException;
+ void set(Job job, List<Path> input, Map<String, Object> config) throws IOException;
--- End diff --
I like
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/432
@ottobackwards your `config` field is missing a `{` on line 2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/432
I'll point out as well that it'd be nice to have a decent exception there, kinda like what you'd get from jsonlint.com:
```
Error: Parse error on line 2:
... "config": "columns": { "domain": 1,
-----------------------^
Expecting 'EOF', '}', ',', ']', got ':'
```
That might be worth a JIRA, honestly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/432
I am trying to test this out - my vm is not doing well at the moment however. Hopefully I can get it straight
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/432
[vagrant@node1 tmp]$ /usr/metron/0.3.0/bin/flatfile_loader.sh -i http://s3.amazonaws.com/alexa-static/top-1m.csv.zip -t enrichment -c t -e ./extractor.json -p 5 -b 128
Exception in thread "main" org.codehaus.jackson.map.JsonMappingException: Can not instantiate value of type [map type; class java.util.LinkedHashMap, [simple type, class java.lang.String] -> [simple type, class java.lang.Object]] from JSON String; no single-String constructor/factory method (through reference chain: org.apache.metron.dataloads.extractor.ExtractorHandler["config"])
at org.codehaus.jackson.map.deser.std.StdValueInstantiator._createFromStringFallbacks(StdValueInstantiator.java:379)
at org.codehaus.jackson.map.deser.std.StdValueInstantiator.createFromString(StdValueInstantiator.java:268)
at org.codehaus.jackson.map.deser.std.MapDeserializer.deserialize(MapDeserializer.java:244)
at org.codehaus.jackson.map.deser.std.MapDeserializer.deserialize(MapDeserializer.java:33)
at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:299)
at org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:414)
at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:697)
at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)
at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1909)
at org.apache.metron.dataloads.extractor.ExtractorHandler.load(ExtractorHandler.java:70)
at org.apache.metron.dataloads.extractor.ExtractorHandler.load(ExtractorHandler.java:75)
at org.apache.metron.dataloads.extractor.ExtractorHandler.load(ExtractorHandler.java:78)
at org.apache.metron.dataloads.nonbulk.flatfile.SimpleEnrichmentFlatFileLoader.main(SimpleEnrichmentFlatFileLoader.java:49)
at org.apache.metron.dataloads.nonbulk.flatfile.SimpleEnrichmentFlatFileLoader.main(SimpleEnrichmentFlatFileLoader.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
[vagrant@node1 tmp]$
with extractor.json of:
```
{
"config" :
"columns" : {
"domain" : 1,
"rank" : 0
}
,"indicator_column" : "domain"
,"type" : "alexa"
,"separator" : ","
},
"extractor" : "CSV"
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #432: METRON-682: Unify and Improve the Flat F...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/432
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---