You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Pariksheet Barapatre <pa...@gmail.com> on 2016/06/20 09:06:44 UTC
Kafka HDFS Connector
Hello All,
I have data coming from sensors into kafka cluster in text format delimited
by comma.
How to offload this data to Hive periodically from Kafka. I guess, Kafka
Connect should solve my problem but when I checked documentation, examples
have only avro formatted data. Can you please provide some knowledge on
this.
Many Thanks
Pari
Re: Kafka HDFS Connector
Posted by Pariksheet Barapatre <pa...@gmail.com>.
Many Thanks Dave and Dustin for your inputs. I will check code and try to
implement proposed solution.
Cheers
Pari
On 22 June 2016 at 23:25, Dustin Cote <du...@confluent.io> wrote:
> Yes, I believe what you're looking for is what Dave described. Here's the
> source of that interface
>
> https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/Format.java
> There
> already exists a StringConverter that should handle the conversion in and
> out of the connect data format in your case
>
> https://kafka.apache.org/0100/javadoc/org/apache/kafka/connect/storage/StringConverter.html
> .
> I think that's what you are looking for in terms of a Converter. It looks
> like your bigger need is the output format for HDFS.
>
> FYI -- you are most welcome to add your request at the GitHub issues page
> for the HDFS Connector
> https://github.com/confluentinc/kafka-connect-hdfs/issues
>
> On Wed, Jun 22, 2016 at 1:26 PM, Tauzell, Dave <
> Dave.Tauzell@surescripts.com
> > wrote:
>
> > I don't see any built-in support for this but I think that you can write
> a
> > class that implements io.confluent.connect.hdfs.Format
> >
> > public interface Format {
> > RecordWriterProvider getRecordWriterProvider();
> > SchemaFileReader getSchemaFileReader(AvroData avroData);
> > HiveUtil getHiveUtil(HdfsSinkConnectorConfig config, AvroData avroData,
> > HiveMetaStore hiveMetaStore);
> > }
> >
> > You would still have to register a schema in the Schema Registry and the
> > "SchemaFileReader" that you return would have to return the same Schema.
> >
> > -Dave
> >
> > Dave Tauzell | Senior Software Engineer | Surescripts
> > O: 651.855.3042 | www.surescripts.com | Dave.Tauzell@surescripts.com
> > Connect with us: Twitter I LinkedIn I Facebook I YouTube
> >
> >
> > -----Original Message-----
> > From: Pariksheet Barapatre [mailto:pari.datagen@gmail.com]
> > Sent: Wednesday, June 22, 2016 11:49 AM
> > To: users@kafka.apache.org
> > Cc: dev@kafka.apache.org
> > Subject: Re: Kafka HDFS Connector
> >
> > Hi Dustin,
> >
> > I am looking for option 1.
> >
> > Looking at Kafka Connect code, I guess we need to write converter code if
> > not available.
> >
> >
> > Thanks in advance.
> >
> > Regards
> > Pari
> >
> >
> > On 22 June 2016 at 18:50, Dustin Cote <du...@confluent.io> wrote:
> >
> > > Hi Pari,
> > >
> > > Can you clarify which scenario you are looking to implement?
> > > 1) plaintext Kafka data --> plaintext HDFS data readable by hive
> > > 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
> > >
> > > Regards,
> > >
> > >
> > >
> > > On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre <
> > > pari.datagen@gmail.com> wrote:
> > >
> > > > Thanks for your suggestions. I think if kafka connect provides the
> > > > same functionality as flume and storm, why should we go for another
> > > > infrastructure investment.
> > > >
> > > > Kafka Connect effectively copies data from Kafka topic to HDFS
> > > > through connector. It supports avro as well as parquet, I am looking
> > > > if we can
> > > use
> > > > it to load plain text data.
> > > >
> > > > Cheers
> > > > Pari
> > > >
> > > >
> > > >
> > > > On 22 June 2016 at 12:34, Lohith Samaga M
> > > > <Lo...@mphasis.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > You can use Storm also, Here you have the option of
> > > > > rotating
> > > the
> > > > > file. You can also write to Hive directly.
> > > > >
> > > > > Best regards / Mit freundlichen Grüßen / Sincères salutations M.
> > > > > Lohith Samaga
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> > > > > Sent: Wednesday, June 22, 2016 12.32
> > > > > To: users@kafka.apache.org; dev@kafka.apache.org
> > > > > Subject: Re: Kafka HDFS Connector
> > > > >
> > > > > I think you can use flume also.
> > > > >
> > > > > Thanks,
> > > > > Mudit
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre"
> > > > > <pa...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > >Anybody have any idea on this?
> > > > > >
> > > > > >Thanks
> > > > > >Pari
> > > > > >
> > > > > >On 20 June 2016 at 14:36, Pariksheet Barapatre <
> > > pari.datagen@gmail.com>
> > > > > >wrote:
> > > > > >
> > > > > >> Hello All,
> > > > > >>
> > > > > >> I have data coming from sensors into kafka cluster in text
> > > > > >> format delimited by comma.
> > > > > >>
> > > > > >> How to offload this data to Hive periodically from Kafka. I
> > > > > >> guess, Kafka Connect should solve my problem but when I checked
> > > > > >> documentation, examples have only avro formatted data. Can you
> > > please
> > > > > >> provide some knowledge on this.
> > > > > >>
> > > > > >> Many Thanks
> > > > > >> Pari
> > > > > >>
> > > > >
> > > > > Information transmitted by this e-mail is proprietary to Mphasis,
> > > > > its associated companies and/ or its customers and is intended for
> > > > > use only by the individual or entity to which it is addressed, and
> > > > may
> > > > > contain information that is privileged, confidential or exempt
> > > > > from disclosure under applicable law. If you are not the
> > > intended
> > > > > recipient or it appears that this mail has been forwarded to you
> > > > > without proper authority, you are notified that any use or
> > > > > dissemination of this information in any manner is strictly
> > > > > prohibited. In such cases, please notify us immediately at
> > > > > mailmaster@mphasis.com and delete this mail from your records.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Dustin Cote
> > > confluent.io
> > >
> > This e-mail and any files transmitted with it are confidential, may
> > contain sensitive information, and are intended solely for the use of the
> > individual or entity to whom they are addressed. If you have received
> this
> > e-mail in error, please notify the sender by reply e-mail immediately and
> > destroy all copies of the e-mail and any attachments.
> >
>
>
>
> --
> Dustin Cote
> confluent.io
>
Re: Kafka HDFS Connector
Posted by Pariksheet Barapatre <pa...@gmail.com>.
Many Thanks Dave and Dustin for your inputs. I will check code and try to
implement proposed solution.
Cheers
Pari
On 22 June 2016 at 23:25, Dustin Cote <du...@confluent.io> wrote:
> Yes, I believe what you're looking for is what Dave described. Here's the
> source of that interface
>
> https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/Format.java
> There
> already exists a StringConverter that should handle the conversion in and
> out of the connect data format in your case
>
> https://kafka.apache.org/0100/javadoc/org/apache/kafka/connect/storage/StringConverter.html
> .
> I think that's what you are looking for in terms of a Converter. It looks
> like your bigger need is the output format for HDFS.
>
> FYI -- you are most welcome to add your request at the GitHub issues page
> for the HDFS Connector
> https://github.com/confluentinc/kafka-connect-hdfs/issues
>
> On Wed, Jun 22, 2016 at 1:26 PM, Tauzell, Dave <
> Dave.Tauzell@surescripts.com
> > wrote:
>
> > I don't see any built-in support for this but I think that you can write
> a
> > class that implements io.confluent.connect.hdfs.Format
> >
> > public interface Format {
> > RecordWriterProvider getRecordWriterProvider();
> > SchemaFileReader getSchemaFileReader(AvroData avroData);
> > HiveUtil getHiveUtil(HdfsSinkConnectorConfig config, AvroData avroData,
> > HiveMetaStore hiveMetaStore);
> > }
> >
> > You would still have to register a schema in the Schema Registry and the
> > "SchemaFileReader" that you return would have to return the same Schema.
> >
> > -Dave
> >
> > Dave Tauzell | Senior Software Engineer | Surescripts
> > O: 651.855.3042 | www.surescripts.com | Dave.Tauzell@surescripts.com
> > Connect with us: Twitter I LinkedIn I Facebook I YouTube
> >
> >
> > -----Original Message-----
> > From: Pariksheet Barapatre [mailto:pari.datagen@gmail.com]
> > Sent: Wednesday, June 22, 2016 11:49 AM
> > To: users@kafka.apache.org
> > Cc: dev@kafka.apache.org
> > Subject: Re: Kafka HDFS Connector
> >
> > Hi Dustin,
> >
> > I am looking for option 1.
> >
> > Looking at Kafka Connect code, I guess we need to write converter code if
> > not available.
> >
> >
> > Thanks in advance.
> >
> > Regards
> > Pari
> >
> >
> > On 22 June 2016 at 18:50, Dustin Cote <du...@confluent.io> wrote:
> >
> > > Hi Pari,
> > >
> > > Can you clarify which scenario you are looking to implement?
> > > 1) plaintext Kafka data --> plaintext HDFS data readable by hive
> > > 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
> > >
> > > Regards,
> > >
> > >
> > >
> > > On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre <
> > > pari.datagen@gmail.com> wrote:
> > >
> > > > Thanks for your suggestions. I think if kafka connect provides the
> > > > same functionality as flume and storm, why should we go for another
> > > > infrastructure investment.
> > > >
> > > > Kafka Connect effectively copies data from Kafka topic to HDFS
> > > > through connector. It supports avro as well as parquet, I am looking
> > > > if we can
> > > use
> > > > it to load plain text data.
> > > >
> > > > Cheers
> > > > Pari
> > > >
> > > >
> > > >
> > > > On 22 June 2016 at 12:34, Lohith Samaga M
> > > > <Lo...@mphasis.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > You can use Storm also, Here you have the option of
> > > > > rotating
> > > the
> > > > > file. You can also write to Hive directly.
> > > > >
> > > > > Best regards / Mit freundlichen Grüßen / Sincères salutations M.
> > > > > Lohith Samaga
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> > > > > Sent: Wednesday, June 22, 2016 12.32
> > > > > To: users@kafka.apache.org; dev@kafka.apache.org
> > > > > Subject: Re: Kafka HDFS Connector
> > > > >
> > > > > I think you can use flume also.
> > > > >
> > > > > Thanks,
> > > > > Mudit
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre"
> > > > > <pa...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > >Anybody have any idea on this?
> > > > > >
> > > > > >Thanks
> > > > > >Pari
> > > > > >
> > > > > >On 20 June 2016 at 14:36, Pariksheet Barapatre <
> > > pari.datagen@gmail.com>
> > > > > >wrote:
> > > > > >
> > > > > >> Hello All,
> > > > > >>
> > > > > >> I have data coming from sensors into kafka cluster in text
> > > > > >> format delimited by comma.
> > > > > >>
> > > > > >> How to offload this data to Hive periodically from Kafka. I
> > > > > >> guess, Kafka Connect should solve my problem but when I checked
> > > > > >> documentation, examples have only avro formatted data. Can you
> > > please
> > > > > >> provide some knowledge on this.
> > > > > >>
> > > > > >> Many Thanks
> > > > > >> Pari
> > > > > >>
> > > > >
> > > > > Information transmitted by this e-mail is proprietary to Mphasis,
> > > > > its associated companies and/ or its customers and is intended for
> > > > > use only by the individual or entity to which it is addressed, and
> > > > may
> > > > > contain information that is privileged, confidential or exempt
> > > > > from disclosure under applicable law. If you are not the
> > > intended
> > > > > recipient or it appears that this mail has been forwarded to you
> > > > > without proper authority, you are notified that any use or
> > > > > dissemination of this information in any manner is strictly
> > > > > prohibited. In such cases, please notify us immediately at
> > > > > mailmaster@mphasis.com and delete this mail from your records.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Dustin Cote
> > > confluent.io
> > >
> > This e-mail and any files transmitted with it are confidential, may
> > contain sensitive information, and are intended solely for the use of the
> > individual or entity to whom they are addressed. If you have received
> this
> > e-mail in error, please notify the sender by reply e-mail immediately and
> > destroy all copies of the e-mail and any attachments.
> >
>
>
>
> --
> Dustin Cote
> confluent.io
>
Re: Kafka HDFS Connector
Posted by Dustin Cote <du...@confluent.io>.
Yes, I believe what you're looking for is what Dave described. Here's the
source of that interface
https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/Format.java
There
already exists a StringConverter that should handle the conversion in and
out of the connect data format in your case
https://kafka.apache.org/0100/javadoc/org/apache/kafka/connect/storage/StringConverter.html.
I think that's what you are looking for in terms of a Converter. It looks
like your bigger need is the output format for HDFS.
FYI -- you are most welcome to add your request at the GitHub issues page
for the HDFS Connector
https://github.com/confluentinc/kafka-connect-hdfs/issues
On Wed, Jun 22, 2016 at 1:26 PM, Tauzell, Dave <Dave.Tauzell@surescripts.com
> wrote:
> I don't see any built-in support for this but I think that you can write a
> class that implements io.confluent.connect.hdfs.Format
>
> public interface Format {
> RecordWriterProvider getRecordWriterProvider();
> SchemaFileReader getSchemaFileReader(AvroData avroData);
> HiveUtil getHiveUtil(HdfsSinkConnectorConfig config, AvroData avroData,
> HiveMetaStore hiveMetaStore);
> }
>
> You would still have to register a schema in the Schema Registry and the
> "SchemaFileReader" that you return would have to return the same Schema.
>
> -Dave
>
> Dave Tauzell | Senior Software Engineer | Surescripts
> O: 651.855.3042 | www.surescripts.com | Dave.Tauzell@surescripts.com
> Connect with us: Twitter I LinkedIn I Facebook I YouTube
>
>
> -----Original Message-----
> From: Pariksheet Barapatre [mailto:pari.datagen@gmail.com]
> Sent: Wednesday, June 22, 2016 11:49 AM
> To: users@kafka.apache.org
> Cc: dev@kafka.apache.org
> Subject: Re: Kafka HDFS Connector
>
> Hi Dustin,
>
> I am looking for option 1.
>
> Looking at Kafka Connect code, I guess we need to write converter code if
> not available.
>
>
> Thanks in advance.
>
> Regards
> Pari
>
>
> On 22 June 2016 at 18:50, Dustin Cote <du...@confluent.io> wrote:
>
> > Hi Pari,
> >
> > Can you clarify which scenario you are looking to implement?
> > 1) plaintext Kafka data --> plaintext HDFS data readable by hive
> > 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
> >
> > Regards,
> >
> >
> >
> > On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre <
> > pari.datagen@gmail.com> wrote:
> >
> > > Thanks for your suggestions. I think if kafka connect provides the
> > > same functionality as flume and storm, why should we go for another
> > > infrastructure investment.
> > >
> > > Kafka Connect effectively copies data from Kafka topic to HDFS
> > > through connector. It supports avro as well as parquet, I am looking
> > > if we can
> > use
> > > it to load plain text data.
> > >
> > > Cheers
> > > Pari
> > >
> > >
> > >
> > > On 22 June 2016 at 12:34, Lohith Samaga M
> > > <Lo...@mphasis.com>
> > > wrote:
> > >
> > > > Hi,
> > > > You can use Storm also, Here you have the option of
> > > > rotating
> > the
> > > > file. You can also write to Hive directly.
> > > >
> > > > Best regards / Mit freundlichen Grüßen / Sincères salutations M.
> > > > Lohith Samaga
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> > > > Sent: Wednesday, June 22, 2016 12.32
> > > > To: users@kafka.apache.org; dev@kafka.apache.org
> > > > Subject: Re: Kafka HDFS Connector
> > > >
> > > > I think you can use flume also.
> > > >
> > > > Thanks,
> > > > Mudit
> > > >
> > > >
> > > >
> > > >
> > > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre"
> > > > <pa...@gmail.com>
> > > > wrote:
> > > >
> > > > >Anybody have any idea on this?
> > > > >
> > > > >Thanks
> > > > >Pari
> > > > >
> > > > >On 20 June 2016 at 14:36, Pariksheet Barapatre <
> > pari.datagen@gmail.com>
> > > > >wrote:
> > > > >
> > > > >> Hello All,
> > > > >>
> > > > >> I have data coming from sensors into kafka cluster in text
> > > > >> format delimited by comma.
> > > > >>
> > > > >> How to offload this data to Hive periodically from Kafka. I
> > > > >> guess, Kafka Connect should solve my problem but when I checked
> > > > >> documentation, examples have only avro formatted data. Can you
> > please
> > > > >> provide some knowledge on this.
> > > > >>
> > > > >> Many Thanks
> > > > >> Pari
> > > > >>
> > > >
> > > > Information transmitted by this e-mail is proprietary to Mphasis,
> > > > its associated companies and/ or its customers and is intended for
> > > > use only by the individual or entity to which it is addressed, and
> > > may
> > > > contain information that is privileged, confidential or exempt
> > > > from disclosure under applicable law. If you are not the
> > intended
> > > > recipient or it appears that this mail has been forwarded to you
> > > > without proper authority, you are notified that any use or
> > > > dissemination of this information in any manner is strictly
> > > > prohibited. In such cases, please notify us immediately at
> > > > mailmaster@mphasis.com and delete this mail from your records.
> > > >
> > >
> >
> >
> >
> > --
> > Dustin Cote
> > confluent.io
> >
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>
--
Dustin Cote
confluent.io
Re: Kafka HDFS Connector
Posted by Dustin Cote <du...@confluent.io>.
Yes, I believe what you're looking for is what Dave described. Here's the
source of that interface
https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/Format.java
There
already exists a StringConverter that should handle the conversion in and
out of the connect data format in your case
https://kafka.apache.org/0100/javadoc/org/apache/kafka/connect/storage/StringConverter.html.
I think that's what you are looking for in terms of a Converter. It looks
like your bigger need is the output format for HDFS.
FYI -- you are most welcome to add your request at the GitHub issues page
for the HDFS Connector
https://github.com/confluentinc/kafka-connect-hdfs/issues
On Wed, Jun 22, 2016 at 1:26 PM, Tauzell, Dave <Dave.Tauzell@surescripts.com
> wrote:
> I don't see any built-in support for this but I think that you can write a
> class that implements io.confluent.connect.hdfs.Format
>
> public interface Format {
> RecordWriterProvider getRecordWriterProvider();
> SchemaFileReader getSchemaFileReader(AvroData avroData);
> HiveUtil getHiveUtil(HdfsSinkConnectorConfig config, AvroData avroData,
> HiveMetaStore hiveMetaStore);
> }
>
> You would still have to register a schema in the Schema Registry and the
> "SchemaFileReader" that you return would have to return the same Schema.
>
> -Dave
>
> Dave Tauzell | Senior Software Engineer | Surescripts
> O: 651.855.3042 | www.surescripts.com | Dave.Tauzell@surescripts.com
> Connect with us: Twitter I LinkedIn I Facebook I YouTube
>
>
> -----Original Message-----
> From: Pariksheet Barapatre [mailto:pari.datagen@gmail.com]
> Sent: Wednesday, June 22, 2016 11:49 AM
> To: users@kafka.apache.org
> Cc: dev@kafka.apache.org
> Subject: Re: Kafka HDFS Connector
>
> Hi Dustin,
>
> I am looking for option 1.
>
> Looking at Kafka Connect code, I guess we need to write converter code if
> not available.
>
>
> Thanks in advance.
>
> Regards
> Pari
>
>
> On 22 June 2016 at 18:50, Dustin Cote <du...@confluent.io> wrote:
>
> > Hi Pari,
> >
> > Can you clarify which scenario you are looking to implement?
> > 1) plaintext Kafka data --> plaintext HDFS data readable by hive
> > 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
> >
> > Regards,
> >
> >
> >
> > On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre <
> > pari.datagen@gmail.com> wrote:
> >
> > > Thanks for your suggestions. I think if kafka connect provides the
> > > same functionality as flume and storm, why should we go for another
> > > infrastructure investment.
> > >
> > > Kafka Connect effectively copies data from Kafka topic to HDFS
> > > through connector. It supports avro as well as parquet, I am looking
> > > if we can
> > use
> > > it to load plain text data.
> > >
> > > Cheers
> > > Pari
> > >
> > >
> > >
> > > On 22 June 2016 at 12:34, Lohith Samaga M
> > > <Lo...@mphasis.com>
> > > wrote:
> > >
> > > > Hi,
> > > > You can use Storm also, Here you have the option of
> > > > rotating
> > the
> > > > file. You can also write to Hive directly.
> > > >
> > > > Best regards / Mit freundlichen Grüßen / Sincères salutations M.
> > > > Lohith Samaga
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> > > > Sent: Wednesday, June 22, 2016 12.32
> > > > To: users@kafka.apache.org; dev@kafka.apache.org
> > > > Subject: Re: Kafka HDFS Connector
> > > >
> > > > I think you can use flume also.
> > > >
> > > > Thanks,
> > > > Mudit
> > > >
> > > >
> > > >
> > > >
> > > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre"
> > > > <pa...@gmail.com>
> > > > wrote:
> > > >
> > > > >Anybody have any idea on this?
> > > > >
> > > > >Thanks
> > > > >Pari
> > > > >
> > > > >On 20 June 2016 at 14:36, Pariksheet Barapatre <
> > pari.datagen@gmail.com>
> > > > >wrote:
> > > > >
> > > > >> Hello All,
> > > > >>
> > > > >> I have data coming from sensors into kafka cluster in text
> > > > >> format delimited by comma.
> > > > >>
> > > > >> How to offload this data to Hive periodically from Kafka. I
> > > > >> guess, Kafka Connect should solve my problem but when I checked
> > > > >> documentation, examples have only avro formatted data. Can you
> > please
> > > > >> provide some knowledge on this.
> > > > >>
> > > > >> Many Thanks
> > > > >> Pari
> > > > >>
> > > >
> > > > Information transmitted by this e-mail is proprietary to Mphasis,
> > > > its associated companies and/ or its customers and is intended for
> > > > use only by the individual or entity to which it is addressed, and
> > > may
> > > > contain information that is privileged, confidential or exempt
> > > > from disclosure under applicable law. If you are not the
> > intended
> > > > recipient or it appears that this mail has been forwarded to you
> > > > without proper authority, you are notified that any use or
> > > > dissemination of this information in any manner is strictly
> > > > prohibited. In such cases, please notify us immediately at
> > > > mailmaster@mphasis.com and delete this mail from your records.
> > > >
> > >
> >
> >
> >
> > --
> > Dustin Cote
> > confluent.io
> >
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>
--
Dustin Cote
confluent.io
RE: Kafka HDFS Connector
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
I don't see any built-in support for this but I think that you can write a class that implements io.confluent.connect.hdfs.Format
public interface Format {
RecordWriterProvider getRecordWriterProvider();
SchemaFileReader getSchemaFileReader(AvroData avroData);
HiveUtil getHiveUtil(HdfsSinkConnectorConfig config, AvroData avroData, HiveMetaStore hiveMetaStore);
}
You would still have to register a schema in the Schema Registry and the "SchemaFileReader" that you return would have to return the same Schema.
-Dave
Dave Tauzell | Senior Software Engineer | Surescripts
O: 651.855.3042 | www.surescripts.com | Dave.Tauzell@surescripts.com
Connect with us: Twitter I LinkedIn I Facebook I YouTube
-----Original Message-----
From: Pariksheet Barapatre [mailto:pari.datagen@gmail.com]
Sent: Wednesday, June 22, 2016 11:49 AM
To: users@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: Kafka HDFS Connector
Hi Dustin,
I am looking for option 1.
Looking at Kafka Connect code, I guess we need to write converter code if not available.
Thanks in advance.
Regards
Pari
On 22 June 2016 at 18:50, Dustin Cote <du...@confluent.io> wrote:
> Hi Pari,
>
> Can you clarify which scenario you are looking to implement?
> 1) plaintext Kafka data --> plaintext HDFS data readable by hive
> 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
>
> Regards,
>
>
>
> On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre <
> pari.datagen@gmail.com> wrote:
>
> > Thanks for your suggestions. I think if kafka connect provides the
> > same functionality as flume and storm, why should we go for another
> > infrastructure investment.
> >
> > Kafka Connect effectively copies data from Kafka topic to HDFS
> > through connector. It supports avro as well as parquet, I am looking
> > if we can
> use
> > it to load plain text data.
> >
> > Cheers
> > Pari
> >
> >
> >
> > On 22 June 2016 at 12:34, Lohith Samaga M
> > <Lo...@mphasis.com>
> > wrote:
> >
> > > Hi,
> > > You can use Storm also, Here you have the option of
> > > rotating
> the
> > > file. You can also write to Hive directly.
> > >
> > > Best regards / Mit freundlichen Grüßen / Sincères salutations M.
> > > Lohith Samaga
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> > > Sent: Wednesday, June 22, 2016 12.32
> > > To: users@kafka.apache.org; dev@kafka.apache.org
> > > Subject: Re: Kafka HDFS Connector
> > >
> > > I think you can use flume also.
> > >
> > > Thanks,
> > > Mudit
> > >
> > >
> > >
> > >
> > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre"
> > > <pa...@gmail.com>
> > > wrote:
> > >
> > > >Anybody have any idea on this?
> > > >
> > > >Thanks
> > > >Pari
> > > >
> > > >On 20 June 2016 at 14:36, Pariksheet Barapatre <
> pari.datagen@gmail.com>
> > > >wrote:
> > > >
> > > >> Hello All,
> > > >>
> > > >> I have data coming from sensors into kafka cluster in text
> > > >> format delimited by comma.
> > > >>
> > > >> How to offload this data to Hive periodically from Kafka. I
> > > >> guess, Kafka Connect should solve my problem but when I checked
> > > >> documentation, examples have only avro formatted data. Can you
> please
> > > >> provide some knowledge on this.
> > > >>
> > > >> Many Thanks
> > > >> Pari
> > > >>
> > >
> > > Information transmitted by this e-mail is proprietary to Mphasis,
> > > its associated companies and/ or its customers and is intended for
> > > use only by the individual or entity to which it is addressed, and
> > may
> > > contain information that is privileged, confidential or exempt
> > > from disclosure under applicable law. If you are not the
> intended
> > > recipient or it appears that this mail has been forwarded to you
> > > without proper authority, you are notified that any use or
> > > dissemination of this information in any manner is strictly
> > > prohibited. In such cases, please notify us immediately at
> > > mailmaster@mphasis.com and delete this mail from your records.
> > >
> >
>
>
>
> --
> Dustin Cote
> confluent.io
>
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
RE: Kafka HDFS Connector
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
I don't see any built-in support for this but I think that you can write a class that implements io.confluent.connect.hdfs.Format
public interface Format {
RecordWriterProvider getRecordWriterProvider();
SchemaFileReader getSchemaFileReader(AvroData avroData);
HiveUtil getHiveUtil(HdfsSinkConnectorConfig config, AvroData avroData, HiveMetaStore hiveMetaStore);
}
You would still have to register a schema in the Schema Registry and the "SchemaFileReader" that you return would have to return the same Schema.
-Dave
Dave Tauzell | Senior Software Engineer | Surescripts
O: 651.855.3042 | www.surescripts.com | Dave.Tauzell@surescripts.com
Connect with us: Twitter I LinkedIn I Facebook I YouTube
-----Original Message-----
From: Pariksheet Barapatre [mailto:pari.datagen@gmail.com]
Sent: Wednesday, June 22, 2016 11:49 AM
To: users@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: Kafka HDFS Connector
Hi Dustin,
I am looking for option 1.
Looking at Kafka Connect code, I guess we need to write converter code if not available.
Thanks in advance.
Regards
Pari
On 22 June 2016 at 18:50, Dustin Cote <du...@confluent.io> wrote:
> Hi Pari,
>
> Can you clarify which scenario you are looking to implement?
> 1) plaintext Kafka data --> plaintext HDFS data readable by hive
> 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
>
> Regards,
>
>
>
> On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre <
> pari.datagen@gmail.com> wrote:
>
> > Thanks for your suggestions. I think if kafka connect provides the
> > same functionality as flume and storm, why should we go for another
> > infrastructure investment.
> >
> > Kafka Connect effectively copies data from Kafka topic to HDFS
> > through connector. It supports avro as well as parquet, I am looking
> > if we can
> use
> > it to load plain text data.
> >
> > Cheers
> > Pari
> >
> >
> >
> > On 22 June 2016 at 12:34, Lohith Samaga M
> > <Lo...@mphasis.com>
> > wrote:
> >
> > > Hi,
> > > You can use Storm also, Here you have the option of
> > > rotating
> the
> > > file. You can also write to Hive directly.
> > >
> > > Best regards / Mit freundlichen Grüßen / Sincères salutations M.
> > > Lohith Samaga
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> > > Sent: Wednesday, June 22, 2016 12.32
> > > To: users@kafka.apache.org; dev@kafka.apache.org
> > > Subject: Re: Kafka HDFS Connector
> > >
> > > I think you can use flume also.
> > >
> > > Thanks,
> > > Mudit
> > >
> > >
> > >
> > >
> > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre"
> > > <pa...@gmail.com>
> > > wrote:
> > >
> > > >Anybody have any idea on this?
> > > >
> > > >Thanks
> > > >Pari
> > > >
> > > >On 20 June 2016 at 14:36, Pariksheet Barapatre <
> pari.datagen@gmail.com>
> > > >wrote:
> > > >
> > > >> Hello All,
> > > >>
> > > >> I have data coming from sensors into kafka cluster in text
> > > >> format delimited by comma.
> > > >>
> > > >> How to offload this data to Hive periodically from Kafka. I
> > > >> guess, Kafka Connect should solve my problem but when I checked
> > > >> documentation, examples have only avro formatted data. Can you
> please
> > > >> provide some knowledge on this.
> > > >>
> > > >> Many Thanks
> > > >> Pari
> > > >>
> > >
> > > Information transmitted by this e-mail is proprietary to Mphasis,
> > > its associated companies and/ or its customers and is intended for
> > > use only by the individual or entity to which it is addressed, and
> > may
> > > contain information that is privileged, confidential or exempt
> > > from disclosure under applicable law. If you are not the
> intended
> > > recipient or it appears that this mail has been forwarded to you
> > > without proper authority, you are notified that any use or
> > > dissemination of this information in any manner is strictly
> > > prohibited. In such cases, please notify us immediately at
> > > mailmaster@mphasis.com and delete this mail from your records.
> > >
> >
>
>
>
> --
> Dustin Cote
> confluent.io
>
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Kafka HDFS Connector
Posted by Pariksheet Barapatre <pa...@gmail.com>.
Hi Dustin,
I am looking for option 1.
Looking at Kafka Connect code, I guess we need to write converter code if
not available.
Thanks in advance.
Regards
Pari
On 22 June 2016 at 18:50, Dustin Cote <du...@confluent.io> wrote:
> Hi Pari,
>
> Can you clarify which scenario you are looking to implement?
> 1) plaintext Kafka data --> plaintext HDFS data readable by hive
> 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
>
> Regards,
>
>
>
> On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre <
> pari.datagen@gmail.com> wrote:
>
> > Thanks for your suggestions. I think if kafka connect provides the same
> > functionality as flume and storm, why should we go for another
> > infrastructure investment.
> >
> > Kafka Connect effectively copies data from Kafka topic to HDFS through
> > connector. It supports avro as well as parquet, I am looking if we can
> use
> > it to load plain text data.
> >
> > Cheers
> > Pari
> >
> >
> >
> > On 22 June 2016 at 12:34, Lohith Samaga M <Lo...@mphasis.com>
> > wrote:
> >
> > > Hi,
> > > You can use Storm also, Here you have the option of rotating
> the
> > > file. You can also write to Hive directly.
> > >
> > > Best regards / Mit freundlichen Grüßen / Sincères salutations
> > > M. Lohith Samaga
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> > > Sent: Wednesday, June 22, 2016 12.32
> > > To: users@kafka.apache.org; dev@kafka.apache.org
> > > Subject: Re: Kafka HDFS Connector
> > >
> > > I think you can use flume also.
> > >
> > > Thanks,
> > > Mudit
> > >
> > >
> > >
> > >
> > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pa...@gmail.com>
> > > wrote:
> > >
> > > >Anybody have any idea on this?
> > > >
> > > >Thanks
> > > >Pari
> > > >
> > > >On 20 June 2016 at 14:36, Pariksheet Barapatre <
> pari.datagen@gmail.com>
> > > >wrote:
> > > >
> > > >> Hello All,
> > > >>
> > > >> I have data coming from sensors into kafka cluster in text format
> > > >> delimited by comma.
> > > >>
> > > >> How to offload this data to Hive periodically from Kafka. I guess,
> > > >> Kafka Connect should solve my problem but when I checked
> > > >> documentation, examples have only avro formatted data. Can you
> please
> > > >> provide some knowledge on this.
> > > >>
> > > >> Many Thanks
> > > >> Pari
> > > >>
> > >
> > > Information transmitted by this e-mail is proprietary to Mphasis, its
> > > associated companies and/ or its customers and is intended
> > > for use only by the individual or entity to which it is addressed, and
> > may
> > > contain information that is privileged, confidential or
> > > exempt from disclosure under applicable law. If you are not the
> intended
> > > recipient or it appears that this mail has been forwarded
> > > to you without proper authority, you are notified that any use or
> > > dissemination of this information in any manner is strictly
> > > prohibited. In such cases, please notify us immediately at
> > > mailmaster@mphasis.com and delete this mail from your records.
> > >
> >
>
>
>
> --
> Dustin Cote
> confluent.io
>
Re: Kafka HDFS Connector
Posted by Pariksheet Barapatre <pa...@gmail.com>.
Hi Dustin,
I am looking for option 1.
Looking at Kafka Connect code, I guess we need to write converter code if
not available.
Thanks in advance.
Regards
Pari
On 22 June 2016 at 18:50, Dustin Cote <du...@confluent.io> wrote:
> Hi Pari,
>
> Can you clarify which scenario you are looking to implement?
> 1) plaintext Kafka data --> plaintext HDFS data readable by hive
> 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
>
> Regards,
>
>
>
> On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre <
> pari.datagen@gmail.com> wrote:
>
> > Thanks for your suggestions. I think if kafka connect provides the same
> > functionality as flume and storm, why should we go for another
> > infrastructure investment.
> >
> > Kafka Connect effectively copies data from Kafka topic to HDFS through
> > connector. It supports avro as well as parquet, I am looking if we can
> use
> > it to load plain text data.
> >
> > Cheers
> > Pari
> >
> >
> >
> > On 22 June 2016 at 12:34, Lohith Samaga M <Lo...@mphasis.com>
> > wrote:
> >
> > > Hi,
> > > You can use Storm also, Here you have the option of rotating
> the
> > > file. You can also write to Hive directly.
> > >
> > > Best regards / Mit freundlichen Grüßen / Sincères salutations
> > > M. Lohith Samaga
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> > > Sent: Wednesday, June 22, 2016 12.32
> > > To: users@kafka.apache.org; dev@kafka.apache.org
> > > Subject: Re: Kafka HDFS Connector
> > >
> > > I think you can use flume also.
> > >
> > > Thanks,
> > > Mudit
> > >
> > >
> > >
> > >
> > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pa...@gmail.com>
> > > wrote:
> > >
> > > >Anybody have any idea on this?
> > > >
> > > >Thanks
> > > >Pari
> > > >
> > > >On 20 June 2016 at 14:36, Pariksheet Barapatre <
> pari.datagen@gmail.com>
> > > >wrote:
> > > >
> > > >> Hello All,
> > > >>
> > > >> I have data coming from sensors into kafka cluster in text format
> > > >> delimited by comma.
> > > >>
> > > >> How to offload this data to Hive periodically from Kafka. I guess,
> > > >> Kafka Connect should solve my problem but when I checked
> > > >> documentation, examples have only avro formatted data. Can you
> please
> > > >> provide some knowledge on this.
> > > >>
> > > >> Many Thanks
> > > >> Pari
> > > >>
> > >
> > > Information transmitted by this e-mail is proprietary to Mphasis, its
> > > associated companies and/ or its customers and is intended
> > > for use only by the individual or entity to which it is addressed, and
> > may
> > > contain information that is privileged, confidential or
> > > exempt from disclosure under applicable law. If you are not the
> intended
> > > recipient or it appears that this mail has been forwarded
> > > to you without proper authority, you are notified that any use or
> > > dissemination of this information in any manner is strictly
> > > prohibited. In such cases, please notify us immediately at
> > > mailmaster@mphasis.com and delete this mail from your records.
> > >
> >
>
>
>
> --
> Dustin Cote
> confluent.io
>
Re: Kafka HDFS Connector
Posted by Dustin Cote <du...@confluent.io>.
Hi Pari,
Can you clarify which scenario you are looking to implement?
1) plaintext Kafka data --> plaintext HDFS data readable by hive
2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
Regards,
On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre <
pari.datagen@gmail.com> wrote:
> Thanks for your suggestions. I think if kafka connect provides the same
> functionality as flume and storm, why should we go for another
> infrastructure investment.
>
> Kafka Connect effectively copies data from Kafka topic to HDFS through
> connector. It supports avro as well as parquet, I am looking if we can use
> it to load plain text data.
>
> Cheers
> Pari
>
>
>
> On 22 June 2016 at 12:34, Lohith Samaga M <Lo...@mphasis.com>
> wrote:
>
> > Hi,
> > You can use Storm also, Here you have the option of rotating the
> > file. You can also write to Hive directly.
> >
> > Best regards / Mit freundlichen Grüßen / Sincères salutations
> > M. Lohith Samaga
> >
> >
> >
> >
> > -----Original Message-----
> > From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> > Sent: Wednesday, June 22, 2016 12.32
> > To: users@kafka.apache.org; dev@kafka.apache.org
> > Subject: Re: Kafka HDFS Connector
> >
> > I think you can use flume also.
> >
> > Thanks,
> > Mudit
> >
> >
> >
> >
> > On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pa...@gmail.com>
> > wrote:
> >
> > >Anybody have any idea on this?
> > >
> > >Thanks
> > >Pari
> > >
> > >On 20 June 2016 at 14:36, Pariksheet Barapatre <pa...@gmail.com>
> > >wrote:
> > >
> > >> Hello All,
> > >>
> > >> I have data coming from sensors into kafka cluster in text format
> > >> delimited by comma.
> > >>
> > >> How to offload this data to Hive periodically from Kafka. I guess,
> > >> Kafka Connect should solve my problem but when I checked
> > >> documentation, examples have only avro formatted data. Can you please
> > >> provide some knowledge on this.
> > >>
> > >> Many Thanks
> > >> Pari
> > >>
> >
> > Information transmitted by this e-mail is proprietary to Mphasis, its
> > associated companies and/ or its customers and is intended
> > for use only by the individual or entity to which it is addressed, and
> may
> > contain information that is privileged, confidential or
> > exempt from disclosure under applicable law. If you are not the intended
> > recipient or it appears that this mail has been forwarded
> > to you without proper authority, you are notified that any use or
> > dissemination of this information in any manner is strictly
> > prohibited. In such cases, please notify us immediately at
> > mailmaster@mphasis.com and delete this mail from your records.
> >
>
--
Dustin Cote
confluent.io
Re: Kafka HDFS Connector
Posted by Dustin Cote <du...@confluent.io>.
Hi Pari,
Can you clarify which scenario you are looking to implement?
1) plaintext Kafka data --> plaintext HDFS data readable by hive
2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
Regards,
On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre <
pari.datagen@gmail.com> wrote:
> Thanks for your suggestions. I think if kafka connect provides the same
> functionality as flume and storm, why should we go for another
> infrastructure investment.
>
> Kafka Connect effectively copies data from Kafka topic to HDFS through
> connector. It supports avro as well as parquet, I am looking if we can use
> it to load plain text data.
>
> Cheers
> Pari
>
>
>
> On 22 June 2016 at 12:34, Lohith Samaga M <Lo...@mphasis.com>
> wrote:
>
> > Hi,
> > You can use Storm also, Here you have the option of rotating the
> > file. You can also write to Hive directly.
> >
> > Best regards / Mit freundlichen Grüßen / Sincères salutations
> > M. Lohith Samaga
> >
> >
> >
> >
> > -----Original Message-----
> > From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> > Sent: Wednesday, June 22, 2016 12.32
> > To: users@kafka.apache.org; dev@kafka.apache.org
> > Subject: Re: Kafka HDFS Connector
> >
> > I think you can use flume also.
> >
> > Thanks,
> > Mudit
> >
> >
> >
> >
> > On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pa...@gmail.com>
> > wrote:
> >
> > >Anybody have any idea on this?
> > >
> > >Thanks
> > >Pari
> > >
> > >On 20 June 2016 at 14:36, Pariksheet Barapatre <pa...@gmail.com>
> > >wrote:
> > >
> > >> Hello All,
> > >>
> > >> I have data coming from sensors into kafka cluster in text format
> > >> delimited by comma.
> > >>
> > >> How to offload this data to Hive periodically from Kafka. I guess,
> > >> Kafka Connect should solve my problem but when I checked
> > >> documentation, examples have only avro formatted data. Can you please
> > >> provide some knowledge on this.
> > >>
> > >> Many Thanks
> > >> Pari
> > >>
> >
> > Information transmitted by this e-mail is proprietary to Mphasis, its
> > associated companies and/ or its customers and is intended
> > for use only by the individual or entity to which it is addressed, and
> may
> > contain information that is privileged, confidential or
> > exempt from disclosure under applicable law. If you are not the intended
> > recipient or it appears that this mail has been forwarded
> > to you without proper authority, you are notified that any use or
> > dissemination of this information in any manner is strictly
> > prohibited. In such cases, please notify us immediately at
> > mailmaster@mphasis.com and delete this mail from your records.
> >
>
--
Dustin Cote
confluent.io
Re: Kafka HDFS Connector
Posted by Pariksheet Barapatre <pa...@gmail.com>.
Thanks for your suggestions. I think if kafka connect provides the same
functionality as flume and storm, why should we go for another
infrastructure investment.
Kafka Connect effectively copies data from Kafka topic to HDFS through
connector. It supports avro as well as parquet, I am looking if we can use
it to load plain text data.
Cheers
Pari
On 22 June 2016 at 12:34, Lohith Samaga M <Lo...@mphasis.com> wrote:
> Hi,
> You can use Storm also, Here you have the option of rotating the
> file. You can also write to Hive directly.
>
> Best regards / Mit freundlichen Grüßen / Sincères salutations
> M. Lohith Samaga
>
>
>
>
> -----Original Message-----
> From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> Sent: Wednesday, June 22, 2016 12.32
> To: users@kafka.apache.org; dev@kafka.apache.org
> Subject: Re: Kafka HDFS Connector
>
> I think you can use flume also.
>
> Thanks,
> Mudit
>
>
>
>
> On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pa...@gmail.com>
> wrote:
>
> >Anybody have any idea on this?
> >
> >Thanks
> >Pari
> >
> >On 20 June 2016 at 14:36, Pariksheet Barapatre <pa...@gmail.com>
> >wrote:
> >
> >> Hello All,
> >>
> >> I have data coming from sensors into kafka cluster in text format
> >> delimited by comma.
> >>
> >> How to offload this data to Hive periodically from Kafka. I guess,
> >> Kafka Connect should solve my problem but when I checked
> >> documentation, examples have only avro formatted data. Can you please
> >> provide some knowledge on this.
> >>
> >> Many Thanks
> >> Pari
> >>
>
> Information transmitted by this e-mail is proprietary to Mphasis, its
> associated companies and/ or its customers and is intended
> for use only by the individual or entity to which it is addressed, and may
> contain information that is privileged, confidential or
> exempt from disclosure under applicable law. If you are not the intended
> recipient or it appears that this mail has been forwarded
> to you without proper authority, you are notified that any use or
> dissemination of this information in any manner is strictly
> prohibited. In such cases, please notify us immediately at
> mailmaster@mphasis.com and delete this mail from your records.
>
Re: Kafka HDFS Connector
Posted by Pariksheet Barapatre <pa...@gmail.com>.
Thanks for your suggestions. I think if kafka connect provides the same
functionality as flume and storm, why should we go for another
infrastructure investment.
Kafka Connect effectively copies data from Kafka topic to HDFS through
connector. It supports avro as well as parquet, I am looking if we can use
it to load plain text data.
Cheers
Pari
On 22 June 2016 at 12:34, Lohith Samaga M <Lo...@mphasis.com> wrote:
> Hi,
> You can use Storm also, Here you have the option of rotating the
> file. You can also write to Hive directly.
>
> Best regards / Mit freundlichen Grüßen / Sincères salutations
> M. Lohith Samaga
>
>
>
>
> -----Original Message-----
> From: Mudit Kumar [mailto:mudit.kumar@askme.in]
> Sent: Wednesday, June 22, 2016 12.32
> To: users@kafka.apache.org; dev@kafka.apache.org
> Subject: Re: Kafka HDFS Connector
>
> I think you can use flume also.
>
> Thanks,
> Mudit
>
>
>
>
> On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pa...@gmail.com>
> wrote:
>
> >Anybody have any idea on this?
> >
> >Thanks
> >Pari
> >
> >On 20 June 2016 at 14:36, Pariksheet Barapatre <pa...@gmail.com>
> >wrote:
> >
> >> Hello All,
> >>
> >> I have data coming from sensors into kafka cluster in text format
> >> delimited by comma.
> >>
> >> How to offload this data to Hive periodically from Kafka. I guess,
> >> Kafka Connect should solve my problem but when I checked
> >> documentation, examples have only avro formatted data. Can you please
> >> provide some knowledge on this.
> >>
> >> Many Thanks
> >> Pari
> >>
>
> Information transmitted by this e-mail is proprietary to Mphasis, its
> associated companies and/ or its customers and is intended
> for use only by the individual or entity to which it is addressed, and may
> contain information that is privileged, confidential or
> exempt from disclosure under applicable law. If you are not the intended
> recipient or it appears that this mail has been forwarded
> to you without proper authority, you are notified that any use or
> dissemination of this information in any manner is strictly
> prohibited. In such cases, please notify us immediately at
> mailmaster@mphasis.com and delete this mail from your records.
>
RE: Kafka HDFS Connector
Posted by Lohith Samaga M <Lo...@mphasis.com>.
Hi,
You can use Storm also, Here you have the option of rotating the file. You can also write to Hive directly.
Best regards / Mit freundlichen Grüßen / Sincères salutations
M. Lohith Samaga
-----Original Message-----
From: Mudit Kumar [mailto:mudit.kumar@askme.in]
Sent: Wednesday, June 22, 2016 12.32
To: users@kafka.apache.org; dev@kafka.apache.org
Subject: Re: Kafka HDFS Connector
I think you can use flume also.
Thanks,
Mudit
On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pa...@gmail.com> wrote:
>Anybody have any idea on this?
>
>Thanks
>Pari
>
>On 20 June 2016 at 14:36, Pariksheet Barapatre <pa...@gmail.com>
>wrote:
>
>> Hello All,
>>
>> I have data coming from sensors into kafka cluster in text format
>> delimited by comma.
>>
>> How to offload this data to Hive periodically from Kafka. I guess,
>> Kafka Connect should solve my problem but when I checked
>> documentation, examples have only avro formatted data. Can you please
>> provide some knowledge on this.
>>
>> Many Thanks
>> Pari
>>
Information transmitted by this e-mail is proprietary to Mphasis, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete this mail from your records.
RE: Kafka HDFS Connector
Posted by Lohith Samaga M <Lo...@mphasis.com>.
Hi,
You can use Storm also, Here you have the option of rotating the file. You can also write to Hive directly.
Best regards / Mit freundlichen Grüßen / Sincères salutations
M. Lohith Samaga
-----Original Message-----
From: Mudit Kumar [mailto:mudit.kumar@askme.in]
Sent: Wednesday, June 22, 2016 12.32
To: users@kafka.apache.org; dev@kafka.apache.org
Subject: Re: Kafka HDFS Connector
I think you can use flume also.
Thanks,
Mudit
On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pa...@gmail.com> wrote:
>Anybody have any idea on this?
>
>Thanks
>Pari
>
>On 20 June 2016 at 14:36, Pariksheet Barapatre <pa...@gmail.com>
>wrote:
>
>> Hello All,
>>
>> I have data coming from sensors into kafka cluster in text format
>> delimited by comma.
>>
>> How to offload this data to Hive periodically from Kafka. I guess,
>> Kafka Connect should solve my problem but when I checked
>> documentation, examples have only avro formatted data. Can you please
>> provide some knowledge on this.
>>
>> Many Thanks
>> Pari
>>
Information transmitted by this e-mail is proprietary to Mphasis, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete this mail from your records.
Re: Kafka HDFS Connector
Posted by Mudit Kumar <mu...@askme.in>.
I think you can use flume also.
Thanks,
Mudit
On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pa...@gmail.com> wrote:
>Anybody have any idea on this?
>
>Thanks
>Pari
>
>On 20 June 2016 at 14:36, Pariksheet Barapatre <pa...@gmail.com>
>wrote:
>
>> Hello All,
>>
>> I have data coming from sensors into kafka cluster in text format
>> delimited by comma.
>>
>> How to offload this data to Hive periodically from Kafka. I guess, Kafka
>> Connect should solve my problem but when I checked documentation, examples
>> have only avro formatted data. Can you please provide some knowledge on
>> this.
>>
>> Many Thanks
>> Pari
>>
Re: Kafka HDFS Connector
Posted by Mudit Kumar <mu...@askme.in>.
I think you can use flume also.
Thanks,
Mudit
On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pa...@gmail.com> wrote:
>Anybody have any idea on this?
>
>Thanks
>Pari
>
>On 20 June 2016 at 14:36, Pariksheet Barapatre <pa...@gmail.com>
>wrote:
>
>> Hello All,
>>
>> I have data coming from sensors into kafka cluster in text format
>> delimited by comma.
>>
>> How to offload this data to Hive periodically from Kafka. I guess, Kafka
>> Connect should solve my problem but when I checked documentation, examples
>> have only avro formatted data. Can you please provide some knowledge on
>> this.
>>
>> Many Thanks
>> Pari
>>
Re: Kafka HDFS Connector
Posted by Pariksheet Barapatre <pa...@gmail.com>.
Anybody have any idea on this?
Thanks
Pari
On 20 June 2016 at 14:36, Pariksheet Barapatre <pa...@gmail.com>
wrote:
> Hello All,
>
> I have data coming from sensors into kafka cluster in text format
> delimited by comma.
>
> How to offload this data to Hive periodically from Kafka. I guess, Kafka
> Connect should solve my problem but when I checked documentation, examples
> have only avro formatted data. Can you please provide some knowledge on
> this.
>
> Many Thanks
> Pari
>
Re: Kafka HDFS Connector
Posted by Pariksheet Barapatre <pa...@gmail.com>.
Anybody have any idea on this?
Thanks
Pari
On 20 June 2016 at 14:36, Pariksheet Barapatre <pa...@gmail.com>
wrote:
> Hello All,
>
> I have data coming from sensors into kafka cluster in text format
> delimited by comma.
>
> How to offload this data to Hive periodically from Kafka. I guess, Kafka
> Connect should solve my problem but when I checked documentation, examples
> have only avro formatted data. Can you please provide some knowledge on
> this.
>
> Many Thanks
> Pari
>