You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by mmiklavc <gi...@git.apache.org> on 2018/08/07 20:08:35 UTC
[GitHub] metron pull request #1151: METRON-1728: Handle null values in config in Pcap...
GitHub user mmiklavc opened a pull request:
https://github.com/apache/metron/pull/1151
METRON-1728: Handle null values in config in Pcap backend more gracefully
## Contributor Comments
https://issues.apache.org/jira/browse/METRON-1728
I modified the ConfigOptions interface to provide the ability to handle default values similar to the Map interface. ie getOrDefault. For PcapJob, I only improved/changed the values we expect to be user-supplied, ie start time, end time, etc., so we don't throw NPE's if a value gets lost in translation. I think we should definitely still add some validation to users via both REST and the UI similar to what is done in PcapCLI.
**Note:** This PR is an improvement to existing functionality, not a regression fix.
## Pull Request Checklist
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
n/a
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mmiklavc/metron pcap-config-handle-nulls
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1151.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1151
----
commit 1fee540969b2f192e58805cdca1ce0a666759af6
Author: Michael Miklavcic <mi...@...>
Date: 2018-08-07T19:29:41Z
Handle null values in config backend more gracefully
----
---
[GitHub] metron pull request #1151: METRON-1728: Handle null values in config in Pcap...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1151#discussion_r208679310
--- Diff: metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java ---
@@ -216,20 +218,22 @@ public void setCompleteCheckInterval(long interval) {
Configuration hadoopConf = PcapOptions.HADOOP_CONF.get(configuration, Configuration.class);
FileSystem fileSystem = PcapOptions.FILESYSTEM.get(configuration, FileSystem.class);
Path basePath = PcapOptions.BASE_PATH.getTransformed(configuration, Path.class);
- Path baseInterimResultPath = PcapOptions.BASE_INTERIM_RESULT_PATH.getTransformed(configuration, Path.class);
+ Path baseInterimResultPath = PcapOptions.BASE_INTERIM_RESULT_PATH
+ .getTransformedOrDefault(configuration, Path.class,
+ new Path(PcapGlobalDefaults.BASE_INTERIM_RESULT_PATH_DEFAULT));
long startTime;
if (configuration.containsKey(PcapOptions.START_TIME_NS.getKey())) {
- startTime = PcapOptions.START_TIME_NS.get(configuration, Long.class);
+ startTime = PcapOptions.START_TIME_NS.getOrDefault(configuration, Long.class, 0L);
} else {
- startTime = PcapOptions.START_TIME_MS.get(configuration, Long.class) * 1000000;
+ startTime = PcapOptions.START_TIME_MS.getOrDefault(configuration, Long.class, 0L) * 1000000;
}
long endTime;
if (configuration.containsKey(PcapOptions.END_TIME_NS.getKey())) {
- endTime = PcapOptions.END_TIME_NS.get(configuration, Long.class);
+ endTime = PcapOptions.END_TIME_NS.getOrDefault(configuration, Long.class, System.nanoTime());
--- End diff --
Nanotime isn't actually current time in nanos or anything objectively meaningful (and can vary by jvm or be negative or whatever). Check out https://docs.oracle.com/javase/7/docs/api/java/lang/System.html#nanoTime()
Particularly,
> This method can only be used to measure elapsed time and is not related to any other notion of system or wall-clock time. The value returned represents nanoseconds since some fixed but arbitrary origin time (perhaps in the future, so values may be negative).
I'd just make it System.currentTimeMillis() and adjust, since I don't think there's a clean way to get a real nanos timestamp in Java (at least until 9, maybe)
---
[GitHub] metron issue #1151: METRON-1728: Handle null values in config in Pcap backen...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1151
+1 by inspection. Thanks for the improvement!
---
[GitHub] metron pull request #1151: METRON-1728: Handle null values in config in Pcap...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc closed the pull request at:
https://github.com/apache/metron/pull/1151
---
[GitHub] metron pull request #1151: METRON-1728: Handle null values in config in Pcap...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1151#discussion_r208697541
--- Diff: metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java ---
@@ -216,20 +218,22 @@ public void setCompleteCheckInterval(long interval) {
Configuration hadoopConf = PcapOptions.HADOOP_CONF.get(configuration, Configuration.class);
FileSystem fileSystem = PcapOptions.FILESYSTEM.get(configuration, FileSystem.class);
Path basePath = PcapOptions.BASE_PATH.getTransformed(configuration, Path.class);
- Path baseInterimResultPath = PcapOptions.BASE_INTERIM_RESULT_PATH.getTransformed(configuration, Path.class);
+ Path baseInterimResultPath = PcapOptions.BASE_INTERIM_RESULT_PATH
+ .getTransformedOrDefault(configuration, Path.class,
+ new Path(PcapGlobalDefaults.BASE_INTERIM_RESULT_PATH_DEFAULT));
long startTime;
if (configuration.containsKey(PcapOptions.START_TIME_NS.getKey())) {
- startTime = PcapOptions.START_TIME_NS.get(configuration, Long.class);
+ startTime = PcapOptions.START_TIME_NS.getOrDefault(configuration, Long.class, 0L);
} else {
- startTime = PcapOptions.START_TIME_MS.get(configuration, Long.class) * 1000000;
+ startTime = PcapOptions.START_TIME_MS.getOrDefault(configuration, Long.class, 0L) * 1000000;
}
long endTime;
if (configuration.containsKey(PcapOptions.END_TIME_NS.getKey())) {
- endTime = PcapOptions.END_TIME_NS.get(configuration, Long.class);
+ endTime = PcapOptions.END_TIME_NS.getOrDefault(configuration, Long.class, System.nanoTime());
--- End diff --
@justinleet good catch! Thanks.
---
[GitHub] metron issue #1151: METRON-1728: Handle null values in config in Pcap backen...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1151
Closing as merged
---