You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@metron.apache.org by Simon Elliston Ball <si...@simonellistonball.com> on 2019/11/12 15:53:35 UTC

Re: CSV parser

You modify the column data in the parser config. I suggest checking the
docs for the csv parser.

On Tue, 12 Nov 2019 at 15:51, Hema malini <nh...@gmail.com> wrote:

> Hi,
>
> I enabled CSV parser and created Kafka topic for CSV. Where i need to
> configure column metadata. Similarly for json parser where i should enable
> the json key ,value. Do i need to modify the parser class and redeploy the
> jar again.
>
> Thanks and regards,
> Hema
>
-- 
--
simon elliston ball
@sireb

Re: CSV parser

Posted by Hema malini <nh...@gmail.com>.

Hi ,

Thanks for your reply. Issue got resolved after several restarts. I faced
issues like when it is a new index it is taking time to index the data.
After restarting to nearly five times, suddenly data appeared in the index.

Thanks and Regards,
Hema

On Wed, 13 Nov, 2019, 12:48 PM Hema malini, <nh...@gmail.com> wrote:

> Simon,
>
> Please find the sample config file
>
> {
>   "parserClassName":"org.apache.metron.parsers.csv.CSVParser",
>   "sensorTopic":"CSV",
>   "parserConfig": {
>     "deviceTimeZone": "UTC",
> "separator":",",
> "columns":{ 'A' : 1, 'B' : 2}
>   }
> }
> Thanks and Regards,
> Hema
>
> On Wed, 13 Nov, 2019, 3:56 AM Simon Elliston Ball, <
> simon@simonellistonball.com> wrote:
>
>> Perhaps you could post your config? You should have a dictionary in it
>> called columns which maps column name to index.
>>
>> Simon
>>
>> On Tue, 12 Nov 2019 at 16:05, Hema malini <nh...@gmail.com>
>> wrote:
>>
>>> Yes. I uploaded as mentioned in the document.
>>>
>>> On Tue, 12 Nov, 2019, 9:31 PM Simon Elliston Ball, <
>>> simon@simonellistonball.com> wrote:
>>>
>>>> Did you upload your configs to zookeeper?
>>>>
>>>> On Tue, 12 Nov 2019 at 16:00, Hema malini <nh...@gmail.com>
>>>> wrote:
>>>>
>>>>> I referred the document - .
>>>>> https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html..
>>>>> I modified the file $METRON_HOME/config/zookeeper/parsers/csv.json. i am
>>>>> getting still column metadata not defined.please let me know what I am
>>>>> missing in this.
>>>>>
>>>>> On Tue, 12 Nov, 2019, 9:23 PM Simon Elliston Ball, <
>>>>> simon@simonellistonball.com> wrote:
>>>>>
>>>>>> You modify the column data in the parser config. I suggest checking
>>>>>> the docs for the csv parser.
>>>>>>
>>>>>> On Tue, 12 Nov 2019 at 15:51, Hema malini <nh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I enabled CSV parser and created Kafka topic for CSV. Where i need
>>>>>>> to configure column metadata. Similarly for json parser where i should
>>>>>>> enable the json key ,value. Do i need to modify the parser class and
>>>>>>> redeploy the jar again.
>>>>>>>
>>>>>>> Thanks and regards,
>>>>>>> Hema
>>>>>>>
>>>>>> --
>>>>>> --
>>>>>> simon elliston ball
>>>>>> @sireb
>>>>>>
>>>>> --
>>>> --
>>>> simon elliston ball
>>>> @sireb
>>>>
>>> --
>> --
>> simon elliston ball
>> @sireb
>>
>

Re: CSV parser

Posted by Hema malini <nh...@gmail.com>.

Simon,

Please find the sample config file

{
  "parserClassName":"org.apache.metron.parsers.csv.CSVParser",
  "sensorTopic":"CSV",
  "parserConfig": {
    "deviceTimeZone": "UTC",
"separator":",",
"columns":{ 'A' : 1, 'B' : 2}
  }
}
Thanks and Regards,
Hema

On Wed, 13 Nov, 2019, 3:56 AM Simon Elliston Ball, <
simon@simonellistonball.com> wrote:

> Perhaps you could post your config? You should have a dictionary in it
> called columns which maps column name to index.
>
> Simon
>
> On Tue, 12 Nov 2019 at 16:05, Hema malini <nh...@gmail.com> wrote:
>
>> Yes. I uploaded as mentioned in the document.
>>
>> On Tue, 12 Nov, 2019, 9:31 PM Simon Elliston Ball, <
>> simon@simonellistonball.com> wrote:
>>
>>> Did you upload your configs to zookeeper?
>>>
>>> On Tue, 12 Nov 2019 at 16:00, Hema malini <nh...@gmail.com>
>>> wrote:
>>>
>>>> I referred the document - .
>>>> https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html..
>>>> I modified the file $METRON_HOME/config/zookeeper/parsers/csv.json. i am
>>>> getting still column metadata not defined.please let me know what I am
>>>> missing in this.
>>>>
>>>> On Tue, 12 Nov, 2019, 9:23 PM Simon Elliston Ball, <
>>>> simon@simonellistonball.com> wrote:
>>>>
>>>>> You modify the column data in the parser config. I suggest checking
>>>>> the docs for the csv parser.
>>>>>
>>>>> On Tue, 12 Nov 2019 at 15:51, Hema malini <nh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I enabled CSV parser and created Kafka topic for CSV. Where i need to
>>>>>> configure column metadata. Similarly for json parser where i should enable
>>>>>> the json key ,value. Do i need to modify the parser class and redeploy the
>>>>>> jar again.
>>>>>>
>>>>>> Thanks and regards,
>>>>>> Hema
>>>>>>
>>>>> --
>>>>> --
>>>>> simon elliston ball
>>>>> @sireb
>>>>>
>>>> --
>>> --
>>> simon elliston ball
>>> @sireb
>>>
>> --
> --
> simon elliston ball
> @sireb
>

Re: CSV parser

Posted by Simon Elliston Ball <si...@simonellistonball.com>.

Perhaps you could post your config? You should have a dictionary in it
called columns which maps column name to index.

Simon

On Tue, 12 Nov 2019 at 16:05, Hema malini <nh...@gmail.com> wrote:

> Yes. I uploaded as mentioned in the document.
>
> On Tue, 12 Nov, 2019, 9:31 PM Simon Elliston Ball, <
> simon@simonellistonball.com> wrote:
>
>> Did you upload your configs to zookeeper?
>>
>> On Tue, 12 Nov 2019 at 16:00, Hema malini <nh...@gmail.com>
>> wrote:
>>
>>> I referred the document - .
>>> https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html..
>>> I modified the file $METRON_HOME/config/zookeeper/parsers/csv.json. i am
>>> getting still column metadata not defined.please let me know what I am
>>> missing in this.
>>>
>>> On Tue, 12 Nov, 2019, 9:23 PM Simon Elliston Ball, <
>>> simon@simonellistonball.com> wrote:
>>>
>>>> You modify the column data in the parser config. I suggest checking the
>>>> docs for the csv parser.
>>>>
>>>> On Tue, 12 Nov 2019 at 15:51, Hema malini <nh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I enabled CSV parser and created Kafka topic for CSV. Where i need to
>>>>> configure column metadata. Similarly for json parser where i should enable
>>>>> the json key ,value. Do i need to modify the parser class and redeploy the
>>>>> jar again.
>>>>>
>>>>> Thanks and regards,
>>>>> Hema
>>>>>
>>>> --
>>>> --
>>>> simon elliston ball
>>>> @sireb
>>>>
>>> --
>> --
>> simon elliston ball
>> @sireb
>>
> --
--
simon elliston ball
@sireb

Re: CSV parser

Posted by Hema malini <nh...@gmail.com>.

Yes. I uploaded as mentioned in the document.

On Tue, 12 Nov, 2019, 9:31 PM Simon Elliston Ball, <
simon@simonellistonball.com> wrote:

> Did you upload your configs to zookeeper?
>
> On Tue, 12 Nov 2019 at 16:00, Hema malini <nh...@gmail.com> wrote:
>
>> I referred the document - .
>> https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html..
>> I modified the file $METRON_HOME/config/zookeeper/parsers/csv.json. i am
>> getting still column metadata not defined.please let me know what I am
>> missing in this.
>>
>> On Tue, 12 Nov, 2019, 9:23 PM Simon Elliston Ball, <
>> simon@simonellistonball.com> wrote:
>>
>>> You modify the column data in the parser config. I suggest checking the
>>> docs for the csv parser.
>>>
>>> On Tue, 12 Nov 2019 at 15:51, Hema malini <nh...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I enabled CSV parser and created Kafka topic for CSV. Where i need to
>>>> configure column metadata. Similarly for json parser where i should enable
>>>> the json key ,value. Do i need to modify the parser class and redeploy the
>>>> jar again.
>>>>
>>>> Thanks and regards,
>>>> Hema
>>>>
>>> --
>>> --
>>> simon elliston ball
>>> @sireb
>>>
>> --
> --
> simon elliston ball
> @sireb
>

Re: CSV parser

Posted by Simon Elliston Ball <si...@simonellistonball.com>.

Did you upload your configs to zookeeper?

On Tue, 12 Nov 2019 at 16:00, Hema malini <nh...@gmail.com> wrote:

> I referred the document - .
> https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html..
> I modified the file $METRON_HOME/config/zookeeper/parsers/csv.json. i am
> getting still column metadata not defined.please let me know what I am
> missing in this.
>
> On Tue, 12 Nov, 2019, 9:23 PM Simon Elliston Ball, <
> simon@simonellistonball.com> wrote:
>
>> You modify the column data in the parser config. I suggest checking the
>> docs for the csv parser.
>>
>> On Tue, 12 Nov 2019 at 15:51, Hema malini <nh...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I enabled CSV parser and created Kafka topic for CSV. Where i need to
>>> configure column metadata. Similarly for json parser where i should enable
>>> the json key ,value. Do i need to modify the parser class and redeploy the
>>> jar again.
>>>
>>> Thanks and regards,
>>> Hema
>>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
> --
--
simon elliston ball
@sireb

Re: CSV parser

Posted by Hema malini <nh...@gmail.com>.

I referred the document - .
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html..
I modified the file $METRON_HOME/config/zookeeper/parsers/csv.json. i am
getting still column metadata not defined.please let me know what I am
missing in this.

On Tue, 12 Nov, 2019, 9:23 PM Simon Elliston Ball, <
simon@simonellistonball.com> wrote:

> You modify the column data in the parser config. I suggest checking the
> docs for the csv parser.
>
> On Tue, 12 Nov 2019 at 15:51, Hema malini <nh...@gmail.com> wrote:
>
>> Hi,
>>
>> I enabled CSV parser and created Kafka topic for CSV. Where i need to
>> configure column metadata. Similarly for json parser where i should enable
>> the json key ,value. Do i need to modify the parser class and redeploy the
>> jar again.
>>
>> Thanks and regards,
>> Hema
>>
> --
> --
> simon elliston ball
> @sireb
>