You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Mustafa Akmal <mu...@abcdata.org> on 2019/02/27 13:31:47 UTC
Raw Message Strategy Envelope not reading custom metadata
Hello
I am using a CSV parser. I have the following log
{"data": "1,john,23","AdditionField": "abcd","AdditionField2": "12345"}
Now I have set the raw message strategy to 'ENVELOPE' and the messageField to 'data'
However after the record is indexed in elasticsearch, the parser does parse the value inside data but it does not get the additional fields as shown in the original log that is 'AdditionField' and 'AdditionField2'. What am I doing wrong? Can anyone help?
Thanks!
---
This email has been checked for viruses by AVG.
https://www.avg.com
Re: Raw Message Strategy Envelope not reading custom metadata
Posted by Stefan Kupstaitis-Dunkler <st...@gmail.com>.
Hi Mustafa,
great. Let me know if it worked after the changes
Best,
Stefan
On Wed, Feb 27, 2019 at 6:12 AM Mustafa Akmal <mu...@abcdata.org>
wrote:
> Ok Thanks Stefan! I'll give it a try
> Regards,
> Mustafa
>
> On Feb 27 2019, at 7:10 pm, Stefan Kupstaitis-Dunkler <
> stefan.dun@gmail.com> wrote:
>
> Hi Mustafa,
>
> Looking at your parser config, you've set these values within the
> "rawMessageStrategyConfig"
>
> *"readMetadata": "true",*
> *"mergeMetadata": "true"*
>
> However, there they will be ignored. Have a look at the same properties
> just below the "*errorWriterClassName*" property. Both of them are set
> to false.
> You can remove those properties from the "rawMessageStrategyConfig"
> section and set them true in the upper part of your config! I'm confident
> that will make it work for you!
>
> Best,
> Stefan
>
> [image: Sent from Mailspring]
> On Wed, Feb 27, 2019 at 6:04 AM Mustafa Akmal <mu...@abcdata.org>
> wrote:
>
> Hi Stefan
> Thanks for the quick response.
> I am using Metron version 0.6.0.1.7.1.0
> <https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/0?redirect=0.6.0.1.7.1.0&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D>
> Here is my parser config
> {
> "parserClassName": "org.apache.metron.parsers.csv.CSVParser",
> "filterClassName": null,
> "sensorTopic": "test_syslog",
> "outputTopic": null,
> "errorTopic": null,
> "writerClassName": null,
> "errorWriterClassName": null,
> "readMetadata": false,
> "mergeMetadata": false,
> "numWorkers": null,
> "numAckers": null,
> "spoutParallelism": 1,
> "spoutNumTasks": 1,
> "parserParallelism": 1,
> "parserNumTasks": 1,
> "errorWriterParallelism": 1,
> "errorWriterNumTasks": 1,
> "spoutConfig": {},
> "securityProtocol": null,
> "stormConfig": {},
> "parserConfig": {
> "columns": {
> "Id": 0,
> "Name": 1,
> "Age": 2
> }
> },
> "fieldTransformations": [],
> "cacheConfig": {},
> "rawMessageStrategy": "ENVELOPE",
> "rawMessageStrategyConfig": {
> "messageField": "data",
> "readMetadata": "true",
> "mergeMetadata": "true"
> }
> }
>
> I have been trying to do this for a couple of days. Tried it on multiple
> different parsers after studying documentation. At first I was using a
> parser chaining usecase. But now I wrote a simple parser to test the
> problem. Set the readMetadata and mergeMetadata fields to true just in
> case. It just doesn't work.
> Regards,
> Mustafa Akmal
>
> On Feb 27 2019, at 6:49 pm, Stefan Kupstaitis-Dunkler <
> stefan.dun@gmail.com> wrote:
>
> Hi Mustafa,
>
>
> - can you verify if the "*mergeMetaData*" property in the parser json
> is set to "*true*".
> - If this property is set to false, other fields won't be merged as
> meta data.
> - If this property is not set, it should default to true for the
> message strategy "ENVELOPE".
> - Any other behaviour is probably a bug.
> - Also verify if you set "*rawMessageStrategy*" to "*true*" for the
> same reasons.
>
> Best,
> Stefan
>
> [image: Sent from Mailspring]
> On Wed, Feb 27, 2019 at 5:34 AM Mustafa Akmal <mu...@abcdata.org>
> wrote:
>
> Hello
> I am using a CSV parser. I have the following log
> {"data": "1,john,23","AdditionField": "abcd","AdditionField2": "12345"}
>
> Now I have set the raw message strategy to 'ENVELOPE' and the messageField
> to 'data'
> However after the record is indexed in elasticsearch, the parser does
> parse the value inside data but it does not get the additional fields as
> shown in the original log that is 'AdditionField' and 'AdditionField2'.
> What am I doing wrong? Can anyone help?
> Thanks!
>
> [image: Sent from Mailspring]
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
> --
> Stefan Kupstaitis-Dunkler
> https://datahovel.com/
> <https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/1?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D>
> https://twitter.com/StefanDunkler
> <https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/2?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D>
>
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
> --
> Stefan Kupstaitis-Dunkler
> https://datahovel.com/
> <https://link.getmailspring.com/link/CB6AFDA8-662C-4388-8B27-F4CEE1406977@getmailspring.com/0?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D>
> https://twitter.com/StefanDunkler
> <https://link.getmailspring.com/link/CB6AFDA8-662C-4388-8B27-F4CEE1406977@getmailspring.com/1?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D>
>
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> <#m_-8837950766558748118_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
--
Stefan Kupstaitis-Dunkler
https://datahovel.com/
https://twitter.com/StefanDunkler
Re: Raw Message Strategy Envelope not reading custom metadata
Posted by Mustafa Akmal <mu...@abcdata.org>.
Ok Thanks Stefan! I'll give it a try
Regards,
Mustafa
On Feb 27 2019, at 7:10 pm, Stefan Kupstaitis-Dunkler <st...@gmail.com> wrote:
> Hi Mustafa,
>
> Looking at your parser config, you've set these values within the "rawMessageStrategyConfig"
>
> "readMetadata": "true",
> "mergeMetadata": "true"
>
>
> However, there they will be ignored. Have a look at the same properties just below the "errorWriterClassName" property. Both of them are set to false.
> You can remove those properties from the "rawMessageStrategyConfig" section and set them true in the upper part of your config! I'm confident that will make it work for you!
>
> Best,
> Stefan
>
>
> On Wed, Feb 27, 2019 at 6:04 AM Mustafa Akmal <mustafa.akmal@abcdata.org (mailto:mustafa.akmal@abcdata.org)> wrote:
> > Hi Stefan
> > Thanks for the quick response.
> > I am using Metron version 0.6.0.1.7.1.0 (https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/0?redirect=0.6.0.1.7.1.0&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
> >
> > Here is my parser config
> > {
> > "parserClassName": "org.apache.metron.parsers.csv.CSVParser",
> > "filterClassName": null,
> > "sensorTopic": "test_syslog",
> > "outputTopic": null,
> > "errorTopic": null,
> > "writerClassName": null,
> > "errorWriterClassName": null,
> > "readMetadata": false,
> > "mergeMetadata": false,
> > "numWorkers": null,
> > "numAckers": null,
> > "spoutParallelism": 1,
> > "spoutNumTasks": 1,
> > "parserParallelism": 1,
> > "parserNumTasks": 1,
> > "errorWriterParallelism": 1,
> > "errorWriterNumTasks": 1,
> > "spoutConfig": {},
> > "securityProtocol": null,
> > "stormConfig": {},
> > "parserConfig": {
> > "columns": {
> > "Id": 0,
> > "Name": 1,
> > "Age": 2
> > }
> > },
> > "fieldTransformations": [],
> > "cacheConfig": {},
> > "rawMessageStrategy": "ENVELOPE",
> > "rawMessageStrategyConfig": {
> > "messageField": "data",
> > "readMetadata": "true",
> > "mergeMetadata": "true"
> > }
> > }
> >
> > I have been trying to do this for a couple of days. Tried it on multiple different parsers after studying documentation. At first I was using a parser chaining usecase. But now I wrote a simple parser to test the problem. Set the readMetadata and mergeMetadata fields to true just in case. It just doesn't work.
> > Regards,
> > Mustafa Akmal
> >
> > On Feb 27 2019, at 6:49 pm, Stefan Kupstaitis-Dunkler <stefan.dun@gmail.com (mailto:stefan.dun@gmail.com)> wrote:
> > > Hi Mustafa,
> > >
> > > can you verify if the "mergeMetaData" property in the parser json is set to "true".
> > > If this property is set to false, other fields won't be merged as meta data.
> > >
> > > If this property is not set, it should default to true for the message strategy "ENVELOPE".
> > >
> > > Any other behaviour is probably a bug.
> > >
> > > Also verify if you set "rawMessageStrategy" to "true" for the same reasons.
> > >
> > >
> > > Best,
> > >
> > > Stefan
> > >
> > >
> > > On Wed, Feb 27, 2019 at 5:34 AM Mustafa Akmal <mustafa.akmal@abcdata.org (mailto:mustafa.akmal@abcdata.org)> wrote:
> > > > Hello
> > > > I am using a CSV parser. I have the following log
> > > > {"data": "1,john,23","AdditionField": "abcd","AdditionField2": "12345"}
> > > >
> > > > Now I have set the raw message strategy to 'ENVELOPE' and the messageField to 'data'
> > > > However after the record is indexed in elasticsearch, the parser does parse the value inside data but it does not get the additional fields as shown in the original log that is 'AdditionField' and 'AdditionField2'. What am I doing wrong? Can anyone help?
> > > > Thanks!
> > > >
> > > >
> > > > Virus-free. www.avg.com (http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient)
> > >
> > > --
> > > Stefan Kupstaitis-Dunkler
> > >
> > > https://datahovel.com/ (https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/1?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
> > > https://twitter.com/StefanDunkler (https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/2?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
> > >
> > >
> > >
> > >
> >
> >
> > Virus-free. www.avg.com (http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient)
>
> --
> Stefan Kupstaitis-Dunkler
>
> https://datahovel.com/ (https://link.getmailspring.com/link/CB6AFDA8-662C-4388-8B27-F4CEE1406977@getmailspring.com/0?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
> https://twitter.com/StefanDunkler (https://link.getmailspring.com/link/CB6AFDA8-662C-4388-8B27-F4CEE1406977@getmailspring.com/1?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
>
>
>
>
---
This email has been checked for viruses by AVG.
https://www.avg.com
Re: Raw Message Strategy Envelope not reading custom metadata
Posted by Mustafa Akmal <mu...@abcdata.org>.
Such a careless mistake. Thanks for the help! :)
Regards,
Mustafa Akmal
On Feb 27 2019, at 7:10 pm, Stefan Kupstaitis-Dunkler <st...@gmail.com> wrote:
> Hi Mustafa,
>
> Looking at your parser config, you've set these values within the "rawMessageStrategyConfig"
>
> "readMetadata": "true",
> "mergeMetadata": "true"
>
>
> However, there they will be ignored. Have a look at the same properties just below the "errorWriterClassName" property. Both of them are set to false.
> You can remove those properties from the "rawMessageStrategyConfig" section and set them true in the upper part of your config! I'm confident that will make it work for you!
>
> Best,
> Stefan
>
>
> On Wed, Feb 27, 2019 at 6:04 AM Mustafa Akmal <mustafa.akmal@abcdata.org (mailto:mustafa.akmal@abcdata.org)> wrote:
> > Hi Stefan
> > Thanks for the quick response.
> > I am using Metron version 0.6.0.1.7.1.0 (https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/0?redirect=0.6.0.1.7.1.0&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
> >
> > Here is my parser config
> > {
> > "parserClassName": "org.apache.metron.parsers.csv.CSVParser",
> > "filterClassName": null,
> > "sensorTopic": "test_syslog",
> > "outputTopic": null,
> > "errorTopic": null,
> > "writerClassName": null,
> > "errorWriterClassName": null,
> > "readMetadata": false,
> > "mergeMetadata": false,
> > "numWorkers": null,
> > "numAckers": null,
> > "spoutParallelism": 1,
> > "spoutNumTasks": 1,
> > "parserParallelism": 1,
> > "parserNumTasks": 1,
> > "errorWriterParallelism": 1,
> > "errorWriterNumTasks": 1,
> > "spoutConfig": {},
> > "securityProtocol": null,
> > "stormConfig": {},
> > "parserConfig": {
> > "columns": {
> > "Id": 0,
> > "Name": 1,
> > "Age": 2
> > }
> > },
> > "fieldTransformations": [],
> > "cacheConfig": {},
> > "rawMessageStrategy": "ENVELOPE",
> > "rawMessageStrategyConfig": {
> > "messageField": "data",
> > "readMetadata": "true",
> > "mergeMetadata": "true"
> > }
> > }
> >
> > I have been trying to do this for a couple of days. Tried it on multiple different parsers after studying documentation. At first I was using a parser chaining usecase. But now I wrote a simple parser to test the problem. Set the readMetadata and mergeMetadata fields to true just in case. It just doesn't work.
> > Regards,
> > Mustafa Akmal
> >
> > On Feb 27 2019, at 6:49 pm, Stefan Kupstaitis-Dunkler <stefan.dun@gmail.com (mailto:stefan.dun@gmail.com)> wrote:
> > > Hi Mustafa,
> > >
> > > can you verify if the "mergeMetaData" property in the parser json is set to "true".
> > > If this property is set to false, other fields won't be merged as meta data.
> > >
> > > If this property is not set, it should default to true for the message strategy "ENVELOPE".
> > >
> > > Any other behaviour is probably a bug.
> > >
> > > Also verify if you set "rawMessageStrategy" to "true" for the same reasons.
> > >
> > >
> > > Best,
> > >
> > > Stefan
> > >
> > >
> > > On Wed, Feb 27, 2019 at 5:34 AM Mustafa Akmal <mustafa.akmal@abcdata.org (mailto:mustafa.akmal@abcdata.org)> wrote:
> > > > Hello
> > > > I am using a CSV parser. I have the following log
> > > > {"data": "1,john,23","AdditionField": "abcd","AdditionField2": "12345"}
> > > >
> > > > Now I have set the raw message strategy to 'ENVELOPE' and the messageField to 'data'
> > > > However after the record is indexed in elasticsearch, the parser does parse the value inside data but it does not get the additional fields as shown in the original log that is 'AdditionField' and 'AdditionField2'. What am I doing wrong? Can anyone help?
> > > > Thanks!
> > > >
> > > >
> > > > Virus-free. www.avg.com (http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient)
> > >
> > > --
> > > Stefan Kupstaitis-Dunkler
> > >
> > > https://datahovel.com/ (https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/1?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
> > > https://twitter.com/StefanDunkler (https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/2?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
> > >
> > >
> > >
> > >
> >
> >
> > Virus-free. www.avg.com (http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient)
>
> --
> Stefan Kupstaitis-Dunkler
>
> https://datahovel.com/ (https://link.getmailspring.com/link/65E266D3-8AC7-418B-AE85-A86E827C3DA7@getmailspring.com/0?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
> https://twitter.com/StefanDunkler (https://link.getmailspring.com/link/65E266D3-8AC7-418B-AE85-A86E827C3DA7@getmailspring.com/1?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
>
>
>
>
---
This email has been checked for viruses by AVG.
https://www.avg.com
Re: Raw Message Strategy Envelope not reading custom metadata
Posted by Stefan Kupstaitis-Dunkler <st...@gmail.com>.
Hi Mustafa,
Looking at your parser config, you've set these values within the
"rawMessageStrategyConfig"
*"readMetadata": "true",*
*"mergeMetadata": "true"*
However, there they will be ignored. Have a look at the same properties
just below the "*errorWriterClassName*" property. Both of them are set to
false.
You can remove those properties from the "rawMessageStrategyConfig" section
and set them true in the upper part of your config! I'm confident that will
make it work for you!
Best,
Stefan
On Wed, Feb 27, 2019 at 6:04 AM Mustafa Akmal <mu...@abcdata.org>
wrote:
> Hi Stefan
> Thanks for the quick response.
> I am using Metron version 0.6.0.1.7.1.0
> <https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/0?redirect=0.6.0.1.7.1.0&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D>
> Here is my parser config
> {
> "parserClassName": "org.apache.metron.parsers.csv.CSVParser",
> "filterClassName": null,
> "sensorTopic": "test_syslog",
> "outputTopic": null,
> "errorTopic": null,
> "writerClassName": null,
> "errorWriterClassName": null,
> "readMetadata": false,
> "mergeMetadata": false,
> "numWorkers": null,
> "numAckers": null,
> "spoutParallelism": 1,
> "spoutNumTasks": 1,
> "parserParallelism": 1,
> "parserNumTasks": 1,
> "errorWriterParallelism": 1,
> "errorWriterNumTasks": 1,
> "spoutConfig": {},
> "securityProtocol": null,
> "stormConfig": {},
> "parserConfig": {
> "columns": {
> "Id": 0,
> "Name": 1,
> "Age": 2
> }
> },
> "fieldTransformations": [],
> "cacheConfig": {},
> "rawMessageStrategy": "ENVELOPE",
> "rawMessageStrategyConfig": {
> "messageField": "data",
> "readMetadata": "true",
> "mergeMetadata": "true"
> }
> }
>
> I have been trying to do this for a couple of days. Tried it on multiple
> different parsers after studying documentation. At first I was using a
> parser chaining usecase. But now I wrote a simple parser to test the
> problem. Set the readMetadata and mergeMetadata fields to true just in
> case. It just doesn't work.
> Regards,
> Mustafa Akmal
>
> On Feb 27 2019, at 6:49 pm, Stefan Kupstaitis-Dunkler <
> stefan.dun@gmail.com> wrote:
>
> Hi Mustafa,
>
>
> - can you verify if the "*mergeMetaData*" property in the parser json
> is set to "*true*".
> - If this property is set to false, other fields won't be merged as
> meta data.
> - If this property is not set, it should default to true for the
> message strategy "ENVELOPE".
> - Any other behaviour is probably a bug.
> - Also verify if you set "*rawMessageStrategy*" to "*true*" for the
> same reasons.
>
> Best,
> Stefan
>
> [image: Sent from Mailspring]
> On Wed, Feb 27, 2019 at 5:34 AM Mustafa Akmal <mu...@abcdata.org>
> wrote:
>
> Hello
> I am using a CSV parser. I have the following log
> {"data": "1,john,23","AdditionField": "abcd","AdditionField2": "12345"}
>
> Now I have set the raw message strategy to 'ENVELOPE' and the messageField
> to 'data'
> However after the record is indexed in elasticsearch, the parser does
> parse the value inside data but it does not get the additional fields as
> shown in the original log that is 'AdditionField' and 'AdditionField2'.
> What am I doing wrong? Can anyone help?
> Thanks!
>
> [image: Sent from Mailspring]
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
> --
> Stefan Kupstaitis-Dunkler
> https://datahovel.com/
> <https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/1?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D>
> https://twitter.com/StefanDunkler
> <https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/2?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D>
>
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> <#m_-7107511858632782524_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
--
Stefan Kupstaitis-Dunkler
https://datahovel.com/
https://twitter.com/StefanDunkler
Re: Raw Message Strategy Envelope not reading custom metadata
Posted by Mustafa Akmal <mu...@abcdata.org>.
Hi Stefan
Thanks for the quick response.
I am using Metron version 0.6.0.1.7.1.0 (https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/0?redirect=0.6.0.1.7.1.0&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
Here is my parser config
{
"parserClassName": "org.apache.metron.parsers.csv.CSVParser",
"filterClassName": null,
"sensorTopic": "test_syslog",
"outputTopic": null,
"errorTopic": null,
"writerClassName": null,
"errorWriterClassName": null,
"readMetadata": false,
"mergeMetadata": false,
"numWorkers": null,
"numAckers": null,
"spoutParallelism": 1,
"spoutNumTasks": 1,
"parserParallelism": 1,
"parserNumTasks": 1,
"errorWriterParallelism": 1,
"errorWriterNumTasks": 1,
"spoutConfig": {},
"securityProtocol": null,
"stormConfig": {},
"parserConfig": {
"columns": {
"Id": 0,
"Name": 1,
"Age": 2
}
},
"fieldTransformations": [],
"cacheConfig": {},
"rawMessageStrategy": "ENVELOPE",
"rawMessageStrategyConfig": {
"messageField": "data",
"readMetadata": "true",
"mergeMetadata": "true"
}
}
I have been trying to do this for a couple of days. Tried it on multiple different parsers after studying documentation. At first I was using a parser chaining usecase. But now I wrote a simple parser to test the problem. Set the readMetadata and mergeMetadata fields to true just in case. It just doesn't work.
Regards,
Mustafa Akmal
On Feb 27 2019, at 6:49 pm, Stefan Kupstaitis-Dunkler <st...@gmail.com> wrote:
> Hi Mustafa,
>
> can you verify if the "mergeMetaData" property in the parser json is set to "true".
> If this property is set to false, other fields won't be merged as meta data.
>
> If this property is not set, it should default to true for the message strategy "ENVELOPE".
>
> Any other behaviour is probably a bug.
>
> Also verify if you set "rawMessageStrategy" to "true" for the same reasons.
>
>
> Best,
>
> Stefan
>
>
> On Wed, Feb 27, 2019 at 5:34 AM Mustafa Akmal <mustafa.akmal@abcdata.org (mailto:mustafa.akmal@abcdata.org)> wrote:
> > Hello
> > I am using a CSV parser. I have the following log
> > {"data": "1,john,23","AdditionField": "abcd","AdditionField2": "12345"}
> >
> > Now I have set the raw message strategy to 'ENVELOPE' and the messageField to 'data'
> > However after the record is indexed in elasticsearch, the parser does parse the value inside data but it does not get the additional fields as shown in the original log that is 'AdditionField' and 'AdditionField2'. What am I doing wrong? Can anyone help?
> > Thanks!
> >
> >
> > Virus-free. www.avg.com (http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient)
>
> --
> Stefan Kupstaitis-Dunkler
>
> https://datahovel.com/ (https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/1?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
> https://twitter.com/StefanDunkler (https://link.getmailspring.com/link/750F1435-C1CA-4F45-B67F-3A275105C1AE@getmailspring.com/2?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D)
>
>
>
>
---
This email has been checked for viruses by AVG.
https://www.avg.com
Re: Raw Message Strategy Envelope not reading custom metadata
Posted by Stefan Kupstaitis-Dunkler <st...@gmail.com>.
Hi Mustafa,
- can you verify if the "*mergeMetaData*" property in the parser json
is set to "*true*".
- If this property is set to false, other fields won't be merged as meta
data.
- If this property is not set, it should default to true for the message
strategy "ENVELOPE".
- Any other behaviour is probably a bug.
- Also verify if you set "*rawMessageStrategy*" to "*true*" for the same
reasons.
Best,
Stefan
On Wed, Feb 27, 2019 at 5:34 AM Mustafa Akmal <mu...@abcdata.org>
wrote:
> Hello
> I am using a CSV parser. I have the following log
> {"data": "1,john,23","AdditionField": "abcd","AdditionField2": "12345"}
>
> Now I have set the raw message strategy to 'ENVELOPE' and the messageField
> to 'data'
> However after the record is indexed in elasticsearch, the parser does
> parse the value inside data but it does not get the additional fields as
> shown in the original log that is 'AdditionField' and 'AdditionField2'.
> What am I doing wrong? Can anyone help?
> Thanks!
>
> [image: Sent from Mailspring]
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> <#m_-9076012844472627488_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
--
Stefan Kupstaitis-Dunkler
https://datahovel.com/
https://twitter.com/StefanDunkler