You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2016/10/11 17:52:36 UTC
[GitHub] incubator-metron pull request #304: METRON-496: Field transformations are ap...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/304
METRON-496: Field transformations are applied after parser validation
Field transformations are applied after validation, which means that the validation cannot be affected by the transformations. Consider a situation where you get a timestamp field in as a string and the parser validation expects a long. Conversion could be done as part of a field transformation, whereas now it would fail validation.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-496
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/304.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #304
----
commit fa40a3b496a251d55d7fe301b5a839435a4ade7d
Author: cstella <ce...@gmail.com>
Date: 2016-10-11T17:47:24Z
METRON-496: Field transformations are applied after parser validation
commit 329a13d2d676701f4284cf216f9d444de5a56986
Author: cstella <ce...@gmail.com>
Date: 2016-10-11T17:51:10Z
Merge branch 'master' into METRON-496
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #304: METRON-496: Field transformations are applied a...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/304
Haha, definitely @ottobackwards comment! Thanks for the feedback. :)
* I found a potential situation with the CSVParser wrt timestamps that would prevent users from retaining the timestamps passed in. If you pass in a timestamp and do NOT configure the CSVParser to parse the timestamp (via the `timestampFormat` config option), then it'll overwrite your timestamp field with null, which would mean you have no ability to parse the timestamp in stellar downstream.
Consider the following config:
```
{
"parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
,"parserConfig": {
"columns" : {
"data" : 0
,"timestamp" : 1
}
}
,"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : "timestamp"
,"config" : {
"timestamp" : "TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss', 'UTC')"
}
}
]
}
```
with the following data, `foo,2016-01-05 17:02:30`
As it stands, the CSVParser would return:
```
{
"original_string" : "foo,2016-01-05 17:02:30"
,"data" : "foo"
,"timestamp" : null
```
thereby dropping the timestamp field instead of passing it through if you choose to NOT have the parser manage the timestamps.
* The benefit is that you have the choice now to do it in Stellar, but I would prefer to stay backwards compatible. Since the CSVParser supported timestamp parsing before this PR, I left it in (and fixed a bug that would have made it impossible to pass timestamps through).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #304: METRON-496: Field transformations are ap...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/304
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #304: METRON-496: Field transformations are applied a...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/304
two things, if I may comment
* did you mean to bring in the CSV parser change?
* isn't the benefit of this that you can _remove_ the timestamp injection from the generic parsers and do it in stellar? Why add it into the csv?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #304: METRON-496: Field transformations are applied a...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/304
I'm +1 by inspection. I'm pretty agnostic on the question of leaving the capability in and would be inclined to do what Casey said and add another jira if we are interesting removing timestamp handling.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #304: METRON-496: Field transformations are applied a...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/304
So, for the JSONMapParser - we could remove it though?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #304: METRON-496: Field transformations are applied a...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/304
I'd prefer to keep the functionality for the CSVParser the way it is in this JIRA. I just wanted to make it so that the CSVParser's `parse` method wouldn't actively stomp on the data passed through if you don't choose to use the timestamp conversion functionality. If you want to add another JIRA around removing handling timestamps in parsers, then we can.
Ultimately though, I am not super offended by the capability remaining as long as the user has the choice to not use it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] incubator-metron pull request #304: METRON-496: Field
transformations are ap...
Posted by Nick Allen <ni...@nickallen.org>.
I am not aware of any discussions that have happened around that. I think
it would be very useful to have that capability though. Good thought,
Carolyn.
On Tue, Oct 11, 2016 at 2:38 PM, Carolyn Duby <cd...@hortonworks.com> wrote:
> Have we considered having precondition validation and postcondition
> validation. Preconditions could check that the fields are in good
> condition to be transformed. Postconditions could find errors in events
> after field transformations.
>
> Thanks
> Carolyn
>
>
>
> On 10/11/16, 1:52 PM, "cestella" <gi...@git.apache.org> wrote:
>
> >GitHub user cestella opened a pull request:
> >
> > https://github.com/apache/incubator-metron/pull/304
> >
> > METRON-496: Field transformations are applied after parser validation
> >
> > Field transformations are applied after validation, which means that
> the validation cannot be affected by the transformations. Consider a
> situation where you get a timestamp field in as a string and the parser
> validation expects a long. Conversion could be done as part of a field
> transformation, whereas now it would fail validation.
> >
> >You can merge this pull request into a Git repository by running:
> >
> > $ git pull https://github.com/cestella/incubator-metron METRON-496
> >
> >Alternatively you can review and apply these changes as the patch at:
> >
> > https://github.com/apache/incubator-metron/pull/304.patch
> >
> >To close this pull request, make a commit to your master/trunk branch
> >with (at least) the following in the commit message:
> >
> > This closes #304
> >
> >----
> >commit fa40a3b496a251d55d7fe301b5a839435a4ade7d
> >Author: cstella <ce...@gmail.com>
> >Date: 2016-10-11T17:47:24Z
> >
> > METRON-496: Field transformations are applied after parser validation
> >
> >commit 329a13d2d676701f4284cf216f9d444de5a56986
> >Author: cstella <ce...@gmail.com>
> >Date: 2016-10-11T17:51:10Z
> >
> > Merge branch 'master' into METRON-496
> >
> >----
> >
> >
> >---
> >If your project is set up for it, you can reply to this email and have
> your
> >reply appear on GitHub as well. If your project does not have this feature
> >enabled and wishes so, or if the feature is enabled but not working,
> please
> >contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> >with INFRA.
> >---
> >
>
--
Nick Allen <ni...@nickallen.org>
Re: [GitHub] incubator-metron pull request #304: METRON-496: Field
transformations are ap...
Posted by Carolyn Duby <cd...@hortonworks.com>.
Have we considered having precondition validation and postcondition validation. Preconditions could check that the fields are in good condition to be transformed. Postconditions could find errors in events after field transformations.
Thanks
Carolyn
On 10/11/16, 1:52 PM, "cestella" <gi...@git.apache.org> wrote:
>GitHub user cestella opened a pull request:
>
> https://github.com/apache/incubator-metron/pull/304
>
> METRON-496: Field transformations are applied after parser validation
>
> Field transformations are applied after validation, which means that the validation cannot be affected by the transformations. Consider a situation where you get a timestamp field in as a string and the parser validation expects a long. Conversion could be done as part of a field transformation, whereas now it would fail validation.
>
>You can merge this pull request into a Git repository by running:
>
> $ git pull https://github.com/cestella/incubator-metron METRON-496
>
>Alternatively you can review and apply these changes as the patch at:
>
> https://github.com/apache/incubator-metron/pull/304.patch
>
>To close this pull request, make a commit to your master/trunk branch
>with (at least) the following in the commit message:
>
> This closes #304
>
>----
>commit fa40a3b496a251d55d7fe301b5a839435a4ade7d
>Author: cstella <ce...@gmail.com>
>Date: 2016-10-11T17:47:24Z
>
> METRON-496: Field transformations are applied after parser validation
>
>commit 329a13d2d676701f4284cf216f9d444de5a56986
>Author: cstella <ce...@gmail.com>
>Date: 2016-10-11T17:51:10Z
>
> Merge branch 'master' into METRON-496
>
>----
>
>
>---
>If your project is set up for it, you can reply to this email and have your
>reply appear on GitHub as well. If your project does not have this feature
>enabled and wishes so, or if the feature is enabled but not working, please
>contact infrastructure at infrastructure@apache.org or file a JIRA ticket
>with INFRA.
>---
>