You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Mohit Anchlia <mo...@gmail.com> on 2012/06/07 23:00:53 UTC
Maven repo
I see entried to flume-ng-sdk in maven repo but when I try this it fails. I
tried with different versions but it doesn't work. Am I missing something?
<dependency>
<groupId>org.apache.flume</groupId>
<artifactId>*flume*-*ng*-*sdk*</artifactId>
<version>1.2.0</version>
</dependency>
</dependencies>
6/7/12 1:58:34 PM PDT: Missing artifact
org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
Re: Maven repo
Posted by Mohammad Tariq <do...@gmail.com>.
Thank you so much Mike..I'll proceed as you have suggested and let you
know the progress..And I know you guys can never be harsh ;)..earlier
also I got many of my queries answered by you guys...:)
Regards,
Mohammad Tariq
On Fri, Jun 8, 2012 at 3:41 AM, Mike Percy <mp...@cloudera.com> wrote:
> Hi Mohammad,
> Please feel free to share your experiences and information freely on this
> list. If my reply sounded harsh it was not intended that way :)
>
> If you want to send data from a local file (using tail via exec source) to
> HDFS then you can do something like the following:
>
> Use an exec
> source: https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/FlumeUserGuide.html#exec-source
> Use an HDFS
> sink: https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/FlumeUserGuide.html#hdfs-sink
>
> You can use the exec source to run "tail -f" on the file you are tailing.
>
> Alex has a nice blog post here showing a full example (although he isn't
> using an exec source
> there): http://mapredit.blogspot.com/2012/03/flumeng-evolution.html
>
> Please let us know as you make progress or if you run into any issues.
>
> Thanks,
> Mike
>
> On Thursday, June 7, 2012 at 2:49 PM, Mohammad Tariq wrote:
>
> sorry, it was for sending data from local file to hdfs.
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Jun 8, 2012 at 3:18 AM, Mohammad Tariq <do...@gmail.com> wrote:
>
> Hello Mike,
>
> You are right..but somehow it worked for me, so thought of sharing
> it..Actually I am totally new to all this and trying to learn, so
> might be wrong at places.Pardon me for that..Also I would like to ask,
> once I have flume built and ready to use how can I use it to aggregate
> data..I have used OG but totally unaware of how to achieve this in
> NG..For example in OG I would use following configuration from
> master's web page to send data to hdfs from a local file -
>
> node1 : tail("/var/log/apache2/error.log") | agentSink("localhost",35853) ;
> node2 : collectorSource(35853) |
> collectorSink("hdfs://localhost:9000/flume/webdata/%Y-%m-%d/%H00/",
> "web-");
>
> Provided I have already started master and node1 and node 2 using -
>
> bin/flume master no_watch
> bin/flume node no_watch -n node1
> bin/flume node no_watch -n node1
>
> How can this be achieved in NG???Do we have any web interface in NG???
> Many thanks.
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Jun 8, 2012 at 3:07 AM, Mike Percy <mp...@cloudera.com> wrote:
>
> Sudo should not be necessary at all... For Subversion, as long as you are in
> your home directory you should be fine from a file permissions perspective…
> and for Maven it's not needed since your local repository typically resides
> at $HOME/.m2/repository so you should have write access to run mvn install.
>
> HTH,
> Mike
>
> On Thursday, June 7, 2012 at 2:29 PM, Mohammad Tariq wrote:
>
> Hello Mohit,
>
> If you are facing problems in building flume-ng properly, give a
> try to the steps shown below..actually I have also been struggling
> since last few days..it worked for me..
>
> $ sudi svn co https://svn.apache.org/repos/asf/incubator/flume/trunk flume
> $ cd flume
> $ sudo mvn3 install -DskipTests
>
> I am using Apache Maven 3.0.3.
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Jun 8, 2012 at 2:30 AM, Mohit Anchlia <mo...@gmail.com>
> wrote:
>
> I see entried to flume-ng-sdk in maven repo but when I try this it fails. I
> tried with different versions but it doesn't work. Am I missing something?
>
> <dependency>
>
> <groupId>org.apache.flume</groupId>
>
> <artifactId>flume-ng-sdk</artifactId>
>
> <version>1.2.0</version>
>
> </dependency>
>
> </dependencies>
>
>
>
> 6/7/12 1:58:34 PM PDT: Missing artifact
> org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
>
>
Re: Maven repo
Posted by Mike Percy <mp...@cloudera.com>.
Hi Mohammad,
Please feel free to share your experiences and information freely on this list. If my reply sounded harsh it was not intended that way :)
If you want to send data from a local file (using tail via exec source) to HDFS then you can do something like the following:
Use an exec source: https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/FlumeUserGuide.html#exec-source
Use an HDFS sink: https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/FlumeUserGuide.html#hdfs-sink
You can use the exec source to run "tail -f" on the file you are tailing.
Alex has a nice blog post here showing a full example (although he isn't using an exec source there): http://mapredit.blogspot.com/2012/03/flumeng-evolution.html
Please let us know as you make progress or if you run into any issues.
Thanks,
Mike
On Thursday, June 7, 2012 at 2:49 PM, Mohammad Tariq wrote:
> sorry, it was for sending data from local file to hdfs.
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Jun 8, 2012 at 3:18 AM, Mohammad Tariq <dontariq@gmail.com (mailto:dontariq@gmail.com)> wrote:
> > Hello Mike,
> >
> > You are right..but somehow it worked for me, so thought of sharing
> > it..Actually I am totally new to all this and trying to learn, so
> > might be wrong at places.Pardon me for that..Also I would like to ask,
> > once I have flume built and ready to use how can I use it to aggregate
> > data..I have used OG but totally unaware of how to achieve this in
> > NG..For example in OG I would use following configuration from
> > master's web page to send data to hdfs from a local file -
> >
> > node1 : tail("/var/log/apache2/error.log") | agentSink("localhost",35853) ;
> > node2 : collectorSource(35853) |
> > collectorSink("hdfs://localhost:9000/flume/webdata/%Y-%m-%d/%H00/",
> > "web-");
> >
> > Provided I have already started master and node1 and node 2 using -
> >
> > bin/flume master no_watch
> > bin/flume node no_watch -n node1
> > bin/flume node no_watch -n node1
> >
> > How can this be achieved in NG???Do we have any web interface in NG???
> > Many thanks.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Fri, Jun 8, 2012 at 3:07 AM, Mike Percy <mpercy@cloudera.com (mailto:mpercy@cloudera.com)> wrote:
> > > Sudo should not be necessary at all... For Subversion, as long as you are in
> > > your home directory you should be fine from a file permissions perspective…
> > > and for Maven it's not needed since your local repository typically resides
> > > at $HOME/.m2/repository so you should have write access to run mvn install.
> > >
> > > HTH,
> > > Mike
> > >
> > > On Thursday, June 7, 2012 at 2:29 PM, Mohammad Tariq wrote:
> > >
> > > Hello Mohit,
> > >
> > > If you are facing problems in building flume-ng properly, give a
> > > try to the steps shown below..actually I have also been struggling
> > > since last few days..it worked for me..
> > >
> > > $ sudi svn co https://svn.apache.org/repos/asf/incubator/flume/trunk flume
> > > $ cd flume
> > > $ sudo mvn3 install -DskipTests
> > >
> > > I am using Apache Maven 3.0.3.
> > >
> > > Regards,
> > > Mohammad Tariq
> > >
> > >
> > > On Fri, Jun 8, 2012 at 2:30 AM, Mohit Anchlia <mohitanchlia@gmail.com (mailto:mohitanchlia@gmail.com)>
> > > wrote:
> > >
> > > I see entried to flume-ng-sdk in maven repo but when I try this it fails. I
> > > tried with different versions but it doesn't work. Am I missing something?
> > >
> > > <dependency>
> > >
> > > <groupId>org.apache.flume</groupId>
> > >
> > > <artifactId>flume-ng-sdk</artifactId>
> > >
> > > <version>1.2.0</version>
> > >
> > > </dependency>
> > >
> > > </dependencies>
> > >
> > >
> > >
> > > 6/7/12 1:58:34 PM PDT: Missing artifact
> > > org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
> > >
> >
> >
>
>
>
Re: Maven repo
Posted by Mohammad Tariq <do...@gmail.com>.
sorry, it was for sending data from local file to hdfs.
Regards,
Mohammad Tariq
On Fri, Jun 8, 2012 at 3:18 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Mike,
>
> You are right..but somehow it worked for me, so thought of sharing
> it..Actually I am totally new to all this and trying to learn, so
> might be wrong at places.Pardon me for that..Also I would like to ask,
> once I have flume built and ready to use how can I use it to aggregate
> data..I have used OG but totally unaware of how to achieve this in
> NG..For example in OG I would use following configuration from
> master's web page to send data to hdfs from a local file -
>
> node1 : tail("/var/log/apache2/error.log") | agentSink("localhost",35853) ;
> node2 : collectorSource(35853) |
> collectorSink("hdfs://localhost:9000/flume/webdata/%Y-%m-%d/%H00/",
> "web-");
>
> Provided I have already started master and node1 and node 2 using -
>
> bin/flume master no_watch
> bin/flume node no_watch -n node1
> bin/flume node no_watch -n node1
>
> How can this be achieved in NG???Do we have any web interface in NG???
> Many thanks.
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Jun 8, 2012 at 3:07 AM, Mike Percy <mp...@cloudera.com> wrote:
>> Sudo should not be necessary at all... For Subversion, as long as you are in
>> your home directory you should be fine from a file permissions perspective…
>> and for Maven it's not needed since your local repository typically resides
>> at $HOME/.m2/repository so you should have write access to run mvn install.
>>
>> HTH,
>> Mike
>>
>> On Thursday, June 7, 2012 at 2:29 PM, Mohammad Tariq wrote:
>>
>> Hello Mohit,
>>
>> If you are facing problems in building flume-ng properly, give a
>> try to the steps shown below..actually I have also been struggling
>> since last few days..it worked for me..
>>
>> $ sudi svn co https://svn.apache.org/repos/asf/incubator/flume/trunk flume
>> $ cd flume
>> $ sudo mvn3 install -DskipTests
>>
>> I am using Apache Maven 3.0.3.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Fri, Jun 8, 2012 at 2:30 AM, Mohit Anchlia <mo...@gmail.com>
>> wrote:
>>
>> I see entried to flume-ng-sdk in maven repo but when I try this it fails. I
>> tried with different versions but it doesn't work. Am I missing something?
>>
>> <dependency>
>>
>> <groupId>org.apache.flume</groupId>
>>
>> <artifactId>flume-ng-sdk</artifactId>
>>
>> <version>1.2.0</version>
>>
>> </dependency>
>>
>> </dependencies>
>>
>>
>>
>> 6/7/12 1:58:34 PM PDT: Missing artifact
>> org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
>>
>>
Re: Maven repo
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Mike,
You are right..but somehow it worked for me, so thought of sharing
it..Actually I am totally new to all this and trying to learn, so
might be wrong at places.Pardon me for that..Also I would like to ask,
once I have flume built and ready to use how can I use it to aggregate
data..I have used OG but totally unaware of how to achieve this in
NG..For example in OG I would use following configuration from
master's web page to send data to hdfs from a local file -
node1 : tail("/var/log/apache2/error.log") | agentSink("localhost",35853) ;
node2 : collectorSource(35853) |
collectorSink("hdfs://localhost:9000/flume/webdata/%Y-%m-%d/%H00/",
"web-");
Provided I have already started master and node1 and node 2 using -
bin/flume master no_watch
bin/flume node no_watch -n node1
bin/flume node no_watch -n node1
How can this be achieved in NG???Do we have any web interface in NG???
Many thanks.
Regards,
Mohammad Tariq
On Fri, Jun 8, 2012 at 3:07 AM, Mike Percy <mp...@cloudera.com> wrote:
> Sudo should not be necessary at all... For Subversion, as long as you are in
> your home directory you should be fine from a file permissions perspective…
> and for Maven it's not needed since your local repository typically resides
> at $HOME/.m2/repository so you should have write access to run mvn install.
>
> HTH,
> Mike
>
> On Thursday, June 7, 2012 at 2:29 PM, Mohammad Tariq wrote:
>
> Hello Mohit,
>
> If you are facing problems in building flume-ng properly, give a
> try to the steps shown below..actually I have also been struggling
> since last few days..it worked for me..
>
> $ sudi svn co https://svn.apache.org/repos/asf/incubator/flume/trunk flume
> $ cd flume
> $ sudo mvn3 install -DskipTests
>
> I am using Apache Maven 3.0.3.
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Jun 8, 2012 at 2:30 AM, Mohit Anchlia <mo...@gmail.com>
> wrote:
>
> I see entried to flume-ng-sdk in maven repo but when I try this it fails. I
> tried with different versions but it doesn't work. Am I missing something?
>
> <dependency>
>
> <groupId>org.apache.flume</groupId>
>
> <artifactId>flume-ng-sdk</artifactId>
>
> <version>1.2.0</version>
>
> </dependency>
>
> </dependencies>
>
>
>
> 6/7/12 1:58:34 PM PDT: Missing artifact
> org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
>
>
Re: Maven repo
Posted by Mike Percy <mp...@cloudera.com>.
Sudo should not be necessary at all... For Subversion, as long as you are in your home directory you should be fine from a file permissions perspective… and for Maven it's not needed since your local repository typically resides at $HOME/.m2/repository so you should have write access to run mvn install.
HTH,
Mike
On Thursday, June 7, 2012 at 2:29 PM, Mohammad Tariq wrote:
> Hello Mohit,
>
> If you are facing problems in building flume-ng properly, give a
> try to the steps shown below..actually I have also been struggling
> since last few days..it worked for me..
>
> $ sudi svn co https://svn.apache.org/repos/asf/incubator/flume/trunk flume
> $ cd flume
> $ sudo mvn3 install -DskipTests
>
> I am using Apache Maven 3.0.3.
>
> Regards,
> Mohammad Tariq
>
>
> On Fri, Jun 8, 2012 at 2:30 AM, Mohit Anchlia <mohitanchlia@gmail.com (mailto:mohitanchlia@gmail.com)> wrote:
> > I see entried to flume-ng-sdk in maven repo but when I try this it fails. I
> > tried with different versions but it doesn't work. Am I missing something?
> >
> > <dependency>
> >
> > <groupId>org.apache.flume</groupId>
> >
> > <artifactId>flume-ng-sdk</artifactId>
> >
> > <version>1.2.0</version>
> >
> > </dependency>
> >
> > </dependencies>
> >
> >
> >
> > 6/7/12 1:58:34 PM PDT: Missing artifact
> > org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
> >
>
>
>
Re: Maven repo
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Mohit,
If you are facing problems in building flume-ng properly, give a
try to the steps shown below..actually I have also been struggling
since last few days..it worked for me..
$ sudi svn co https://svn.apache.org/repos/asf/incubator/flume/trunk flume
$ cd flume
$ sudo mvn3 install -DskipTests
I am using Apache Maven 3.0.3.
Regards,
Mohammad Tariq
On Fri, Jun 8, 2012 at 2:30 AM, Mohit Anchlia <mo...@gmail.com> wrote:
> I see entried to flume-ng-sdk in maven repo but when I try this it fails. I
> tried with different versions but it doesn't work. Am I missing something?
>
> <dependency>
>
> <groupId>org.apache.flume</groupId>
>
> <artifactId>flume-ng-sdk</artifactId>
>
> <version>1.2.0</version>
>
> </dependency>
>
> </dependencies>
>
>
>
> 6/7/12 1:58:34 PM PDT: Missing artifact
> org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
Re: Maven repo
Posted by Mohit Anchlia <mo...@gmail.com>.
Thanks. I'll populate local maven and do it that way
On Thu, Jun 7, 2012 at 2:24 PM, Mike Percy <mp...@cloudera.com> wrote:
> I just checked on search.maven.org, and I guess the Flume
> 1.1.0-incubating artifacts never got deployed to Maven Central. I think it
> should still be possible to do that. I have filed
> https://issues.apache.org/jira/browse/FLUME-1263 to track doing that, we
> just need to find someone who knows how to do it.
>
> In the meantime, you can simply install the 1.1.0 release jars in your
> local maven repository as a workaround, or you can do an mvn install on the
> trunk sources to achieve the same result with the latest 1.2.0-SNAPSHOT
> bits.
>
> Thanks,
> Mike
>
> On Thursday, June 7, 2012 at 2:00 PM, Mohit Anchlia wrote:
>
> I see entried to flume-ng-sdk in maven repo but when I try this it
> fails. I tried with different versions but it doesn't work. Am I missing
> something?
>
> <dependency>
>
> <groupId>org.apache.flume</groupId>
>
> <artifactId>*flume*-*ng*-*sdk*</artifactId>
>
> <version>1.2.0</version>
>
> </dependency>
>
> </dependencies>
>
>
>
> 6/7/12 1:58:34 PM PDT: Missing artifact
> org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
>
>
>
Re: Maven repo
Posted by Mohit Anchlia <mo...@gmail.com>.
I finally got it working. Now I can ask my next set of questions :) But in
different thread.
On Tue, Jun 12, 2012 at 5:55 PM, Mohit Anchlia <mo...@gmail.com>wrote:
> I looked at the maven depedency tree and it looks like one of my other
> packages that is using pig .8.1 is including an older version of avro. I
> tries using "exclusions" but didn't work. Do you have any suggestions on
> what I can do here with maven in this case?
>
>
> On Tue, Jun 12, 2012 at 4:53 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>
>> I upgraded flume but I am still getting same error. Not sure if this is a
>> server problem or a client problem. Any pointers on how to resolve this
>> willbe very helpful
>>
>> FAILED: testAvroClient
>> java.lang.NoSuchMethodError:
>> org.apache.avro.ipc.specific.SpecificRequestor.request(Ljava/lang/String;Ljava/lang/Object;Lorg/apache/avro/ipc/Callback;)V
>> at
>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>> at $Proxy7.append(Unknown Source)
>> at
>> org.apache.flume.api.NettyAvroRpcClient.append(NettyAvroRpcClient.java:173)
>> at
>> org.apache.flume.api.NettyAvroRpcClient.append(NettyAvroRpcClient.java:153)
>> at
>> com.intuit.cg.services.dp.analytics.flume.AvroClient.sendDataToFlume(AvroClient.java:25)
>> at
>> com.intuit.cg.services.dp.analytics.flume.AvroClientTest.testAvroClient(AvroClientTest.java:10)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
>> at org.testng.internal.Invoker.invokeMethod(Invoker.java:702)
>> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:894)
>> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1219)
>> at
>> org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
>> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
>>
>> On Sat, Jun 9, 2012 at 1:50 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>>
>>> I tried Avro 1.6.3 in the client but it still failed. On the server
>>> console I see this: Client side jar was taken from the flum-ng build that
>>> is on the server so I don't really understand.
>>>
>>>
>>> 12/06/09 13:49:53 WARN ipc.NettyServer: Unexpected exception from
>>> downstream.
>>>
>>> java.io.IOException: Connection reset by peer
>>>
>>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>>>
>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>>>
>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
>>>
>>> at sun.nio.ch.IOUtil.read(IOUtil.java:169)
>>>
>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
>>>
>>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
>>>
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>> On Fri, Jun 8, 2012 at 9:25 PM, Mike Percy <mp...@cloudera.com> wrote:
>>>
>>>> Mohit,
>>>> You might want to try with Avro 1.6.3, I think we are using Avro 1.6.3
>>>> now (the Avro dependency should be in the pom).
>>>>
>>>> Thanks,
>>>> Mike
>>>>
>>>> On Friday, June 8, 2012 at 5:45 PM, Mohit Anchlia wrote:
>>>>
>>>> I installed flume-ng-sdk 1.2.0 jar and it's dependencies. Now when I
>>>> run my test I get. I am using avro 1.6.1 as dependency. Could someone
>>>> please help out?
>>>>
>>>>
>>>> FAILED: testAvroClient
>>>>
>>>> java.lang.NoSuchMethodError:
>>>> org.apache.avro.ipc.specific.SpecificRequestor.request(Ljava/lang/String;Ljava/lang/Object;Lorg/apache/avro/ipc/Callback;)V
>>>>
>>>> at org.apache.avro.ipc.specific.SpecificRequestor.invoke(
>>>> *SpecificRequestor.java:68*)
>>>>
>>>> at $Proxy7.append(Unknown Source)
>>>>
>>>> at org.apache.flume.api.NettyAvroRpcClient.append(
>>>> *NettyAvroRpcClient.java:169*)
>>>>
>>>> at org.apache.flume.api.NettyAvroRpcClient.append(
>>>> *NettyAvroRpcClient.java:149*)
>>>>
>>>> On Thu, Jun 7, 2012 at 2:24 PM, Mike Percy <mp...@cloudera.com> wrote:
>>>>
>>>> I just checked on search.maven.org, and I guess the Flume
>>>> 1.1.0-incubating artifacts never got deployed to Maven Central. I think it
>>>> should still be possible to do that. I have filed
>>>> https://issues.apache.org/jira/browse/FLUME-1263 to track doing that,
>>>> we just need to find someone who knows how to do it.
>>>>
>>>> In the meantime, you can simply install the 1.1.0 release jars in your
>>>> local maven repository as a workaround, or you can do an mvn install on the
>>>> trunk sources to achieve the same result with the latest 1.2.0-SNAPSHOT
>>>> bits.
>>>>
>>>> Thanks,
>>>> Mike
>>>>
>>>> On Thursday, June 7, 2012 at 2:00 PM, Mohit Anchlia wrote:
>>>>
>>>> I see entried to flume-ng-sdk in maven repo but when I try this it
>>>> fails. I tried with different versions but it doesn't work. Am I missing
>>>> something?
>>>>
>>>> <dependency>
>>>>
>>>> <groupId>org.apache.flume</groupId>
>>>>
>>>> <artifactId>*flume*-*ng*-*sdk*</artifactId>
>>>>
>>>> <version>1.2.0</version>
>>>>
>>>> </dependency>
>>>>
>>>> </dependencies>
>>>>
>>>>
>>>>
>>>> 6/7/12 1:58:34 PM PDT: Missing artifact
>>>> org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Maven repo
Posted by Mohit Anchlia <mo...@gmail.com>.
I looked at the maven depedency tree and it looks like one of my other
packages that is using pig .8.1 is including an older version of avro. I
tries using "exclusions" but didn't work. Do you have any suggestions on
what I can do here with maven in this case?
On Tue, Jun 12, 2012 at 4:53 PM, Mohit Anchlia <mo...@gmail.com>wrote:
> I upgraded flume but I am still getting same error. Not sure if this is a
> server problem or a client problem. Any pointers on how to resolve this
> willbe very helpful
>
> FAILED: testAvroClient
> java.lang.NoSuchMethodError:
> org.apache.avro.ipc.specific.SpecificRequestor.request(Ljava/lang/String;Ljava/lang/Object;Lorg/apache/avro/ipc/Callback;)V
> at
> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
> at $Proxy7.append(Unknown Source)
> at
> org.apache.flume.api.NettyAvroRpcClient.append(NettyAvroRpcClient.java:173)
> at
> org.apache.flume.api.NettyAvroRpcClient.append(NettyAvroRpcClient.java:153)
> at
> com.intuit.cg.services.dp.analytics.flume.AvroClient.sendDataToFlume(AvroClient.java:25)
> at
> com.intuit.cg.services.dp.analytics.flume.AvroClientTest.testAvroClient(AvroClientTest.java:10)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:702)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:894)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1219)
> at
> org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
>
> On Sat, Jun 9, 2012 at 1:50 PM, Mohit Anchlia <mo...@gmail.com>wrote:
>
>> I tried Avro 1.6.3 in the client but it still failed. On the server
>> console I see this: Client side jar was taken from the flum-ng build that
>> is on the server so I don't really understand.
>>
>>
>> 12/06/09 13:49:53 WARN ipc.NettyServer: Unexpected exception from
>> downstream.
>>
>> java.io.IOException: Connection reset by peer
>>
>> at sun.nio.ch.FileDispatcher.read0(Native Method)
>>
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>>
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
>>
>> at sun.nio.ch.IOUtil.read(IOUtil.java:169)
>>
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
>>
>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
>>
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> at java.lang.Thread.run(Thread.java:662)
>>
>>
>> On Fri, Jun 8, 2012 at 9:25 PM, Mike Percy <mp...@cloudera.com> wrote:
>>
>>> Mohit,
>>> You might want to try with Avro 1.6.3, I think we are using Avro 1.6.3
>>> now (the Avro dependency should be in the pom).
>>>
>>> Thanks,
>>> Mike
>>>
>>> On Friday, June 8, 2012 at 5:45 PM, Mohit Anchlia wrote:
>>>
>>> I installed flume-ng-sdk 1.2.0 jar and it's dependencies. Now when I
>>> run my test I get. I am using avro 1.6.1 as dependency. Could someone
>>> please help out?
>>>
>>>
>>> FAILED: testAvroClient
>>>
>>> java.lang.NoSuchMethodError:
>>> org.apache.avro.ipc.specific.SpecificRequestor.request(Ljava/lang/String;Ljava/lang/Object;Lorg/apache/avro/ipc/Callback;)V
>>>
>>> at org.apache.avro.ipc.specific.SpecificRequestor.invoke(
>>> *SpecificRequestor.java:68*)
>>>
>>> at $Proxy7.append(Unknown Source)
>>>
>>> at org.apache.flume.api.NettyAvroRpcClient.append(
>>> *NettyAvroRpcClient.java:169*)
>>>
>>> at org.apache.flume.api.NettyAvroRpcClient.append(
>>> *NettyAvroRpcClient.java:149*)
>>>
>>> On Thu, Jun 7, 2012 at 2:24 PM, Mike Percy <mp...@cloudera.com> wrote:
>>>
>>> I just checked on search.maven.org, and I guess the Flume
>>> 1.1.0-incubating artifacts never got deployed to Maven Central. I think it
>>> should still be possible to do that. I have filed
>>> https://issues.apache.org/jira/browse/FLUME-1263 to track doing that,
>>> we just need to find someone who knows how to do it.
>>>
>>> In the meantime, you can simply install the 1.1.0 release jars in your
>>> local maven repository as a workaround, or you can do an mvn install on the
>>> trunk sources to achieve the same result with the latest 1.2.0-SNAPSHOT
>>> bits.
>>>
>>> Thanks,
>>> Mike
>>>
>>> On Thursday, June 7, 2012 at 2:00 PM, Mohit Anchlia wrote:
>>>
>>> I see entried to flume-ng-sdk in maven repo but when I try this it
>>> fails. I tried with different versions but it doesn't work. Am I missing
>>> something?
>>>
>>> <dependency>
>>>
>>> <groupId>org.apache.flume</groupId>
>>>
>>> <artifactId>*flume*-*ng*-*sdk*</artifactId>
>>>
>>> <version>1.2.0</version>
>>>
>>> </dependency>
>>>
>>> </dependencies>
>>>
>>>
>>>
>>> 6/7/12 1:58:34 PM PDT: Missing artifact
>>> org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
>>>
>>>
>>>
>>>
>>>
>>
>
Re: Maven repo
Posted by Mohit Anchlia <mo...@gmail.com>.
I upgraded flume but I am still getting same error. Not sure if this is a
server problem or a client problem. Any pointers on how to resolve this
willbe very helpful
FAILED: testAvroClient
java.lang.NoSuchMethodError:
org.apache.avro.ipc.specific.SpecificRequestor.request(Ljava/lang/String;Ljava/lang/Object;Lorg/apache/avro/ipc/Callback;)V
at
org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
at $Proxy7.append(Unknown Source)
at
org.apache.flume.api.NettyAvroRpcClient.append(NettyAvroRpcClient.java:173)
at
org.apache.flume.api.NettyAvroRpcClient.append(NettyAvroRpcClient.java:153)
at
com.intuit.cg.services.dp.analytics.flume.AvroClient.sendDataToFlume(AvroClient.java:25)
at
com.intuit.cg.services.dp.analytics.flume.AvroClientTest.testAvroClient(AvroClientTest.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:702)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:894)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1219)
at
org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
On Sat, Jun 9, 2012 at 1:50 PM, Mohit Anchlia <mo...@gmail.com>wrote:
> I tried Avro 1.6.3 in the client but it still failed. On the server
> console I see this: Client side jar was taken from the flum-ng build that
> is on the server so I don't really understand.
>
>
> 12/06/09 13:49:53 WARN ipc.NettyServer: Unexpected exception from
> downstream.
>
> java.io.IOException: Connection reset by peer
>
> at sun.nio.ch.FileDispatcher.read0(Native Method)
>
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
>
> at sun.nio.ch.IOUtil.read(IOUtil.java:169)
>
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
>
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>
> at
> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
>
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
>
> On Fri, Jun 8, 2012 at 9:25 PM, Mike Percy <mp...@cloudera.com> wrote:
>
>> Mohit,
>> You might want to try with Avro 1.6.3, I think we are using Avro 1.6.3
>> now (the Avro dependency should be in the pom).
>>
>> Thanks,
>> Mike
>>
>> On Friday, June 8, 2012 at 5:45 PM, Mohit Anchlia wrote:
>>
>> I installed flume-ng-sdk 1.2.0 jar and it's dependencies. Now when I
>> run my test I get. I am using avro 1.6.1 as dependency. Could someone
>> please help out?
>>
>>
>> FAILED: testAvroClient
>>
>> java.lang.NoSuchMethodError:
>> org.apache.avro.ipc.specific.SpecificRequestor.request(Ljava/lang/String;Ljava/lang/Object;Lorg/apache/avro/ipc/Callback;)V
>>
>> at org.apache.avro.ipc.specific.SpecificRequestor.invoke(
>> *SpecificRequestor.java:68*)
>>
>> at $Proxy7.append(Unknown Source)
>>
>> at org.apache.flume.api.NettyAvroRpcClient.append(
>> *NettyAvroRpcClient.java:169*)
>>
>> at org.apache.flume.api.NettyAvroRpcClient.append(
>> *NettyAvroRpcClient.java:149*)
>>
>> On Thu, Jun 7, 2012 at 2:24 PM, Mike Percy <mp...@cloudera.com> wrote:
>>
>> I just checked on search.maven.org, and I guess the Flume
>> 1.1.0-incubating artifacts never got deployed to Maven Central. I think it
>> should still be possible to do that. I have filed
>> https://issues.apache.org/jira/browse/FLUME-1263 to track doing that, we
>> just need to find someone who knows how to do it.
>>
>> In the meantime, you can simply install the 1.1.0 release jars in your
>> local maven repository as a workaround, or you can do an mvn install on the
>> trunk sources to achieve the same result with the latest 1.2.0-SNAPSHOT
>> bits.
>>
>> Thanks,
>> Mike
>>
>> On Thursday, June 7, 2012 at 2:00 PM, Mohit Anchlia wrote:
>>
>> I see entried to flume-ng-sdk in maven repo but when I try this it
>> fails. I tried with different versions but it doesn't work. Am I missing
>> something?
>>
>> <dependency>
>>
>> <groupId>org.apache.flume</groupId>
>>
>> <artifactId>*flume*-*ng*-*sdk*</artifactId>
>>
>> <version>1.2.0</version>
>>
>> </dependency>
>>
>> </dependencies>
>>
>>
>>
>> 6/7/12 1:58:34 PM PDT: Missing artifact
>> org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
>>
>>
>>
>>
>>
>
Re: Maven repo
Posted by Mohit Anchlia <mo...@gmail.com>.
I tried Avro 1.6.3 in the client but it still failed. On the server console
I see this: Client side jar was taken from the flum-ng build that is on the
server so I don't really understand.
12/06/09 13:49:53 WARN ipc.NettyServer: Unexpected exception from
downstream.
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
On Fri, Jun 8, 2012 at 9:25 PM, Mike Percy <mp...@cloudera.com> wrote:
> Mohit,
> You might want to try with Avro 1.6.3, I think we are using Avro 1.6.3 now
> (the Avro dependency should be in the pom).
>
> Thanks,
> Mike
>
> On Friday, June 8, 2012 at 5:45 PM, Mohit Anchlia wrote:
>
> I installed flume-ng-sdk 1.2.0 jar and it's dependencies. Now when I run
> my test I get. I am using avro 1.6.1 as dependency. Could someone please
> help out?
>
>
> FAILED: testAvroClient
>
> java.lang.NoSuchMethodError:
> org.apache.avro.ipc.specific.SpecificRequestor.request(Ljava/lang/String;Ljava/lang/Object;Lorg/apache/avro/ipc/Callback;)V
>
> at org.apache.avro.ipc.specific.SpecificRequestor.invoke(
> *SpecificRequestor.java:68*)
>
> at $Proxy7.append(Unknown Source)
>
> at org.apache.flume.api.NettyAvroRpcClient.append(
> *NettyAvroRpcClient.java:169*)
>
> at org.apache.flume.api.NettyAvroRpcClient.append(
> *NettyAvroRpcClient.java:149*)
>
> On Thu, Jun 7, 2012 at 2:24 PM, Mike Percy <mp...@cloudera.com> wrote:
>
> I just checked on search.maven.org, and I guess the Flume
> 1.1.0-incubating artifacts never got deployed to Maven Central. I think it
> should still be possible to do that. I have filed
> https://issues.apache.org/jira/browse/FLUME-1263 to track doing that, we
> just need to find someone who knows how to do it.
>
> In the meantime, you can simply install the 1.1.0 release jars in your
> local maven repository as a workaround, or you can do an mvn install on the
> trunk sources to achieve the same result with the latest 1.2.0-SNAPSHOT
> bits.
>
> Thanks,
> Mike
>
> On Thursday, June 7, 2012 at 2:00 PM, Mohit Anchlia wrote:
>
> I see entried to flume-ng-sdk in maven repo but when I try this it
> fails. I tried with different versions but it doesn't work. Am I missing
> something?
>
> <dependency>
>
> <groupId>org.apache.flume</groupId>
>
> <artifactId>*flume*-*ng*-*sdk*</artifactId>
>
> <version>1.2.0</version>
>
> </dependency>
>
> </dependencies>
>
>
>
> 6/7/12 1:58:34 PM PDT: Missing artifact
> org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
>
>
>
>
>
Re: Maven repo
Posted by Mike Percy <mp...@cloudera.com>.
Mohit,
You might want to try with Avro 1.6.3, I think we are using Avro 1.6.3 now (the Avro dependency should be in the pom).
Thanks,
Mike
On Friday, June 8, 2012 at 5:45 PM, Mohit Anchlia wrote:
> I installed flume-ng-sdk 1.2.0 jar and it's dependencies. Now when I run my test I get. I am using avro 1.6.1 as dependency. Could someone please help out?
>
> FAILED: testAvroClient
> java.lang.NoSuchMethodError: org.apache.avro.ipc.specific.SpecificRequestor.request(Ljava/lang/String;Ljava/lang/Object;Lorg/apache/avro/ipc/Callback;)V
> at org.apache.avro.ipc.specific.SpecificRequestor.invoke(
> SpecificRequestor.java:68)
> at $Proxy7.append(Unknown Source)
> at org.apache.flume.api.NettyAvroRpcClient.append(
> NettyAvroRpcClient.java:169)
> at org.apache.flume.api.NettyAvroRpcClient.append(
> NettyAvroRpcClient.java:149)
>
> On Thu, Jun 7, 2012 at 2:24 PM, Mike Percy <mpercy@cloudera.com (mailto:mpercy@cloudera.com)> wrote:
> > I just checked on search.maven.org (http://search.maven.org/), and I guess the Flume 1.1.0-incubating artifacts never got deployed to Maven Central. I think it should still be possible to do that. I have filed https://issues.apache.org/jira/browse/FLUME-1263 to track doing that, we just need to find someone who knows how to do it.
> >
> > In the meantime, you can simply install the 1.1.0 release jars in your local maven repository as a workaround, or you can do an mvn install on the trunk sources to achieve the same result with the latest 1.2.0-SNAPSHOT bits.
> >
> > Thanks,
> > Mike
> >
> >
> > On Thursday, June 7, 2012 at 2:00 PM, Mohit Anchlia wrote:
> >
> > > I see entried to flume-ng-sdk in maven repo but when I try this it fails. I tried with different versions but it doesn't work. Am I missing something?
> > >
> > > <dependency>
> > >
> > > <groupId>org.apache.flume</groupId>
> > >
> > > <artifactId>flume-ng-sdk</artifactId>
> > >
> > > <version>1.2.0</version>
> > >
> > > </dependency>
> > >
> > > </dependencies>
> > >
> > > 6/7/12 1:58:34 PM PDT: Missing artifact org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
> > >
> > >
> > >
> > >
> >
> >
>
Re: Maven repo
Posted by Mohit Anchlia <mo...@gmail.com>.
I installed flume-ng-sdk 1.2.0 jar and it's dependencies. Now when I run my
test I get. I am using avro 1.6.1 as dependency. Could someone please help
out?
FAILED: testAvroClient
java.lang.NoSuchMethodError:
org.apache.avro.ipc.specific.SpecificRequestor.request(Ljava/lang/String;Ljava/lang/Object;Lorg/apache/avro/ipc/Callback;)V
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(*
SpecificRequestor.java:68*)
at $Proxy7.append(Unknown Source)
at org.apache.flume.api.NettyAvroRpcClient.append(*
NettyAvroRpcClient.java:169*)
at org.apache.flume.api.NettyAvroRpcClient.append(*
NettyAvroRpcClient.java:149*)
On Thu, Jun 7, 2012 at 2:24 PM, Mike Percy <mp...@cloudera.com> wrote:
> I just checked on search.maven.org, and I guess the Flume
> 1.1.0-incubating artifacts never got deployed to Maven Central. I think it
> should still be possible to do that. I have filed
> https://issues.apache.org/jira/browse/FLUME-1263 to track doing that, we
> just need to find someone who knows how to do it.
>
> In the meantime, you can simply install the 1.1.0 release jars in your
> local maven repository as a workaround, or you can do an mvn install on the
> trunk sources to achieve the same result with the latest 1.2.0-SNAPSHOT
> bits.
>
> Thanks,
> Mike
>
> On Thursday, June 7, 2012 at 2:00 PM, Mohit Anchlia wrote:
>
> I see entried to flume-ng-sdk in maven repo but when I try this it
> fails. I tried with different versions but it doesn't work. Am I missing
> something?
>
> <dependency>
>
> <groupId>org.apache.flume</groupId>
>
> <artifactId>*flume*-*ng*-*sdk*</artifactId>
>
> <version>1.2.0</version>
>
> </dependency>
>
> </dependencies>
>
>
>
> 6/7/12 1:58:34 PM PDT: Missing artifact
> org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
>
>
>
Re: Maven repo
Posted by Mike Percy <mp...@cloudera.com>.
I just checked on search.maven.org, and I guess the Flume 1.1.0-incubating artifacts never got deployed to Maven Central. I think it should still be possible to do that. I have filed https://issues.apache.org/jira/browse/FLUME-1263 to track doing that, we just need to find someone who knows how to do it.
In the meantime, you can simply install the 1.1.0 release jars in your local maven repository as a workaround, or you can do an mvn install on the trunk sources to achieve the same result with the latest 1.2.0-SNAPSHOT bits.
Thanks,
Mike
On Thursday, June 7, 2012 at 2:00 PM, Mohit Anchlia wrote:
> I see entried to flume-ng-sdk in maven repo but when I try this it fails. I tried with different versions but it doesn't work. Am I missing something?
>
> <dependency>
>
> <groupId>org.apache.flume</groupId>
>
> <artifactId>flume-ng-sdk</artifactId>
>
> <version>1.2.0</version>
>
> </dependency>
>
> </dependencies>
>
> 6/7/12 1:58:34 PM PDT: Missing artifact org.apache.flume:flume-ng-sdk:jar:1.1.0-cdh3u3:compile
>
>
>