You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Hema malini <nh...@gmail.com> on 2019/11/16 13:51:29 UTC
Enable optional fields in csv parser
Hi all,
Is there any way to mark some columns as optional in column mapping in CSV
parser.
Thanks and Regards,
Hema
Re: Enable optional fields in csv parser
Posted by Hema malini <nh...@gmail.com>.
Thanks a lot for your timely response Simon and Otto .
On Sun, 17 Nov, 2019, 9:20 PM Otto Fowler, <ot...@gmail.com> wrote:
> I think what he is saying is that the csv files may not always have all
> the columns, in which case they won’t parse, which is before stellar can do
> anything.
>
>
>
>
> On November 16, 2019 at 11:30:25, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> A better way of doing this would be to use the fieldTransformation setting
> and the REMOVE method to get rid of the extraneous fields. Docs are
> included at
>
> https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html#
>
> That way you don’t need a separate preprocessing step.
>
> Simon
>
> On Sat, 16 Nov 2019 at 16:09, Hema malini <nh...@gmail.com> wrote:
>
>> Thanks ..will do preprocessing of data..
>>
>> On Sat, 16 Nov, 2019, 9:25 PM Otto Fowler, <ot...@gmail.com>
>> wrote:
>>
>>> No, there is no way to do this currently.
>>>
>>> The parser parses the line into and array of strings that must match the
>>> size of the columns.
>>>
>>> The underlying opencsv parser does not support this either. You may
>>> have to do some normalization work on your data if you need to account for
>>> this.
>>>
>>>
>>>
>>> On November 16, 2019 at 08:49:36, Hema malini (nhemamalini11@gmail.com)
>>> wrote:
>>>
>>> Hi all,
>>>
>>> Is there any way to mark some columns as optional in column mapping in
>>> CSV parser.
>>>
>>> Thanks and Regards,
>>> Hema
>>>
>>> --
> --
> simon elliston ball
> @sireb
>
>
Re: Enable optional fields in csv parser
Posted by Otto Fowler <ot...@gmail.com>.
I think what he is saying is that the csv files may not always have all the
columns, in which case they won’t parse, which is before stellar can do
anything.
On November 16, 2019 at 11:30:25, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
A better way of doing this would be to use the fieldTransformation setting
and the REMOVE method to get rid of the extraneous fields. Docs are
included at
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html#
That way you don’t need a separate preprocessing step.
Simon
On Sat, 16 Nov 2019 at 16:09, Hema malini <nh...@gmail.com> wrote:
> Thanks ..will do preprocessing of data..
>
> On Sat, 16 Nov, 2019, 9:25 PM Otto Fowler, <ot...@gmail.com>
> wrote:
>
>> No, there is no way to do this currently.
>>
>> The parser parses the line into and array of strings that must match the
>> size of the columns.
>>
>> The underlying opencsv parser does not support this either. You may have
>> to do some normalization work on your data if you need to account for this.
>>
>>
>>
>> On November 16, 2019 at 08:49:36, Hema malini (nhemamalini11@gmail.com)
>> wrote:
>>
>> Hi all,
>>
>> Is there any way to mark some columns as optional in column mapping in
>> CSV parser.
>>
>> Thanks and Regards,
>> Hema
>>
>> --
--
simon elliston ball
@sireb
Re: Enable optional fields in csv parser
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
A better way of doing this would be to use the fieldTransformation setting
and the REMOVE method to get rid of the extraneous fields. Docs are
included at
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html#
That way you don’t need a separate preprocessing step.
Simon
On Sat, 16 Nov 2019 at 16:09, Hema malini <nh...@gmail.com> wrote:
> Thanks ..will do preprocessing of data..
>
> On Sat, 16 Nov, 2019, 9:25 PM Otto Fowler, <ot...@gmail.com>
> wrote:
>
>> No, there is no way to do this currently.
>>
>> The parser parses the line into and array of strings that must match the
>> size of the columns.
>>
>> The underlying opencsv parser does not support this either. You may have
>> to do some normalization work on your data if you need to account for this.
>>
>>
>>
>>
>> On November 16, 2019 at 08:49:36, Hema malini (nhemamalini11@gmail.com)
>> wrote:
>>
>> Hi all,
>>
>> Is there any way to mark some columns as optional in column mapping in
>> CSV parser.
>>
>> Thanks and Regards,
>> Hema
>>
>> --
--
simon elliston ball
@sireb
Re: Enable optional fields in csv parser
Posted by Hema malini <nh...@gmail.com>.
Thanks ..will do preprocessing of data..
On Sat, 16 Nov, 2019, 9:25 PM Otto Fowler, <ot...@gmail.com> wrote:
> No, there is no way to do this currently.
>
> The parser parses the line into and array of strings that must match the
> size of the columns.
>
> The underlying opencsv parser does not support this either. You may have
> to do some normalization work on your data if you need to account for this.
>
>
>
>
> On November 16, 2019 at 08:49:36, Hema malini (nhemamalini11@gmail.com)
> wrote:
>
> Hi all,
>
> Is there any way to mark some columns as optional in column mapping in CSV
> parser.
>
> Thanks and Regards,
> Hema
>
>
Re: Enable optional fields in csv parser
Posted by Otto Fowler <ot...@gmail.com>.
No, there is no way to do this currently.
The parser parses the line into and array of strings that must match the
size of the columns.
The underlying opencsv parser does not support this either. You may have
to do some normalization work on your data if you need to account for this.
On November 16, 2019 at 08:49:36, Hema malini (nhemamalini11@gmail.com)
wrote:
Hi all,
Is there any way to mark some columns as optional in column mapping in CSV
parser.
Thanks and Regards,
Hema