You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by kishore kumar <ak...@gmail.com> on 2014/12/14 07:39:53 UTC
csv files to kafka
HI Experts,
I want to load csv files to kafka, anybody help me to write javacode for
this?
--
Thanks,
Kishore.
Re: csv files to kafka
Posted by Joe Stein <jo...@stealth.ly>.
In that project the protobuf
https://github.com/stealthly/f2k/blob/master/src/main/proto/FileType.proto
is compiled/generated in the gradle task using the plugin
https://github.com/stealthly/f2k/blob/master/build.gradle#L17
You need to have protobuf https://github.com/google/protobuf/ installed and
have it setup for your environment appropriately (see protoBuf {} gradle
task in build.gradle
https://github.com/stealthly/f2k/blob/master/build.gradle#L19-L24 )
I left the issue you created on github open so we can get around to
updating the README and adding a vagrant up implementation so it works
right from the repo better without having to-do much local for others so
they don't bump into this and then work around it locally by installing
protobuf.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Sun, Dec 28, 2014 at 2:56 AM, kishore kumar <ak...@gmail.com>
wrote:
> Hi Joe,
>
> I am cloned this repo with
>
> git clone https://github.com/stealthly/f2k.git
>
> cd f2k
>
> when I run .
> #./gradlew clean fatJar
>
> the error is "Execution failed for task ':compileProto'."
>
> when I run with --debug option it gives the info
>
> Gradle user home: /root/.gradle
> Current dir: /root/f2k
> java.io.IOException: Cannot run program "/usr/bin/protoc" (in directory
> "/root/f2k"): error=2, No such file or directory
>
> but when I run from /root
>
> ./f2k/gradlew clean fatJar --stacktrace
>
> the error is "Task 'clean' not found in root project 'root'."
>
> Please help me how to build this.
>
> Thanks.
>
> On Sun, Dec 14, 2014 at 12:53 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Here is a non-production example of a file load to Kafka using Scala
> >
> >
> https://github.com/stealthly/f2k/blob/master/src/main/scala/ly/stealth/f2k/KafkaUploader.scala
> >
> >
> > If you wanted to convert that to javacode the Scala compiler does that
> for
> > you :) and makes it importable as jar to your java program, or you can
> use
> > as reference and rewrite it in Java.
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
> > On Sun, Dec 14, 2014 at 1:39 AM, kishore kumar <ak...@gmail.com>
> > wrote:
> > >
> > > HI Experts,
> > >
> > > I want to load csv files to kafka, anybody help me to write javacode
> for
> > > this?
> > >
> > > --
> > > Thanks,
> > > Kishore.
> > >
> >
>
>
>
> --
> Thanks,
> Kishore.
>
Re: csv files to kafka
Posted by kishore kumar <ak...@gmail.com>.
Hi Joe,
I am cloned this repo with
git clone https://github.com/stealthly/f2k.git
cd f2k
when I run .
#./gradlew clean fatJar
the error is "Execution failed for task ':compileProto'."
when I run with --debug option it gives the info
Gradle user home: /root/.gradle
Current dir: /root/f2k
java.io.IOException: Cannot run program "/usr/bin/protoc" (in directory
"/root/f2k"): error=2, No such file or directory
but when I run from /root
./f2k/gradlew clean fatJar --stacktrace
the error is "Task 'clean' not found in root project 'root'."
Please help me how to build this.
Thanks.
On Sun, Dec 14, 2014 at 12:53 PM, Joe Stein <jo...@stealth.ly> wrote:
> Here is a non-production example of a file load to Kafka using Scala
>
> https://github.com/stealthly/f2k/blob/master/src/main/scala/ly/stealth/f2k/KafkaUploader.scala
>
>
> If you wanted to convert that to javacode the Scala compiler does that for
> you :) and makes it importable as jar to your java program, or you can use
> as reference and rewrite it in Java.
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
> On Sun, Dec 14, 2014 at 1:39 AM, kishore kumar <ak...@gmail.com>
> wrote:
> >
> > HI Experts,
> >
> > I want to load csv files to kafka, anybody help me to write javacode for
> > this?
> >
> > --
> > Thanks,
> > Kishore.
> >
>
--
Thanks,
Kishore.
Re: csv files to kafka
Posted by Joe Stein <jo...@stealth.ly>.
Here is a non-production example of a file load to Kafka using Scala
https://github.com/stealthly/f2k/blob/master/src/main/scala/ly/stealth/f2k/KafkaUploader.scala
If you wanted to convert that to javacode the Scala compiler does that for
you :) and makes it importable as jar to your java program, or you can use
as reference and rewrite it in Java.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Sun, Dec 14, 2014 at 1:39 AM, kishore kumar <ak...@gmail.com>
wrote:
>
> HI Experts,
>
> I want to load csv files to kafka, anybody help me to write javacode for
> this?
>
> --
> Thanks,
> Kishore.
>