You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Radu Gheorghe <ra...@sematext.com> on 2015/09/17 12:31:26 UTC
Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Hello flume users!
I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did that
with 1.5 but now I seem to fail :(
I got this error in the log (full exception below):
java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
information does not match signer information of other classes in the same
package
Basically, what I did was to install Flume 1.6, install Solr 5.3, compile
Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to Flume's
classpath. Here's the full command line:
exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
-Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
'/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
-Djava.library.path= org.apache.flume.node.Application -n agent -f
conf/flume-conf.properties
Here's my Java version, if it matters:
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>From what I can understand out off the Internet (thanks, Internet!) this
may be caused by multiple jars having different versions of the same thing.
But I can only find it in Morphlines (via grep -RHls in all the
classpaths). Maybe I'm bumping into
https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to verify
this by compiling Kite 0.12.0 but it failed with:
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.9:check
(default) on project kite-data: Too many files with unapproved license: 2
See RAT report....
So then I compiled only morphlines and that worked, only to bump into the
same error.
As promised, below is the full exception. For completeness, I'm also
including the main config and the morphline config:
17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
(org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) -
Unable to start SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585 counterGroup:{
name:null counters:{} } } - Exception follows.
java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
information does not match signer information of other classes in the same
package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
at
org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
at
org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
at
org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
at
org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
at
org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
> cat conf/flume-conf.properties
agent.sources = spoolDir
agent.channels = spillableMem
agent.sinks = solr
agent.sources.spoolDir.type = spooldir
agent.sources.spoolDir.spoolDir = /opt/logs
agent.sources.spoolDir.batchSize = 1000
agent.sources.spoolDir.channels = spillableMem
agent.channels.spillableMem.type = SPILLABLEMEMORY
agent.channels.spillableMem.memoryCapacity = 100000
agent.channels.spillableMem.overflowCapacity = 1000000
agent.channels.spillableMem.checkpointDir = /var/run/flume
agent.channels.spillableMem.dataDirs = /var/run/flume-data
agent.sinks.solr.type =
org.apache.flume.sink.solr.morphline.MorphlineSolrSink
agent.sinks.solr.channel = spillableMem
agent.sinks.solr.morphlineFile = conf/morphline.conf
agent.sinks.solr.morphlineId = 1
agent.sinks.solr.batchSize = 10000
> cat conf/morphline.conf
SOLR_LOCATOR : {
collection : gettingstarted
solrUrl : "http://192.168.1.3:8983/solr/"
batchSize : 5000
}
morphlines : [
{ id : 1
commands : [
{ readLine { charset : UTF-8 } }
{
grok {
dictionaryFiles : [conf/grok-patterns]
expressions : {
message : """%{COMBINEDAPACHELOG}"""
}
}
}
# { generateUUID {
# field : id
# }
# }
{
loadSolr {
solrLocator : ${SOLR_LOCATOR}
}
}
]
}
]
I also have a grok-patterns file there, but I don't think it's relevant.
Any ideas are much appreciated!
Thanks and best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Posted by Radu Gheorghe <ra...@sematext.com>.
Thanks, Gonzalo, for all your replies. I'm pretty sure the solution is
within reach, I'm just running out of time. I did build flume from the repo
you indicated but again it fails to start with some missing dependencies. I
guess that indeed I should look at what are the required dependencies and
provide them and only them.
I will try that later on and come back if I have more issues.
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Thu, Sep 17, 2015 at 4:28 PM, Gonzalo Herreros <gh...@gmail.com>
wrote:
> Yes, it sounds like a dependency library version error.
> You can take their the Cloudera Flume binaries, build from their branch of
> Flume or just see how the manage the dependencies and do the same.
>
> https://github.com/cloudera/flume-ng (look at their poms)
> You can download binaries here:
> https://repository.cloudera.com/cloudera/cloudera-repos/
>
> Gonzalo
>
>
> On 17 September 2015 at 14:21, Radu Gheorghe <ra...@sematext.com>
> wrote:
>
>> Thanks, Gonzalo! That actually got me further. A Solr exception later
>> (got around it by removing 4.10 Solr libs which I assume don't work with my
>> 5.3 for which I already have libs in CP), then I got another error pointing
>> the same sort of issue.
>>
>> So I just went ahead and removed all duplicated jars (though many were
>> din different versions) from kite. With something as ugly as this:
>> for lib in `ls -1 | sed s/-[0-9].*.jar$//`; do ls
>> /opt/apache-flume-1.6.0-bin/lib/$lib* 2>/dev/null; if [ $? -eq 0 ]; then rm
>> $lib*; fi; done
>>
>> Then Flume complained about some codahale metrics class missing, so I
>> moved back the metrics-core library to kite. And now I get this:
>>
>> java.lang.VerifyError: Cannot inherit from final class
>>
>> I'm not sure if you need the whole stacktrace (I will paste it below) but
>> I have again the suspicion of duplicated stuff or mismatched versions.
>>
>> I'm not sure I'm on the right path, so you were pointing me to different
>> ones:
>> - add the integration with Solr. Can you tell me which are the jars
>> needed? From which subproject?
>> - look at the version of Flume distributed by Cloudera. I can't find just
>> Flume, I suppose I need to get the whole CDH or something like that?
>>
>> Thanks again for all your help!
>>
>> Best regards,
>> Radu
>>
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>> On Thu, Sep 17, 2015 at 2:50 PM, Gonzalo Herreros <gh...@gmail.com>
>> wrote:
>>
>>> That's it.
>>> Remove that jar from the Flume classpath so it doesn't interfere with
>>> the one in Flume (version 2.5)
>>>
>>> Note Flume already has the kite libraries. I would just add the
>>> integration with Solr (I suggest you have a look at the version of Flume
>>> distributed by Cloudera)
>>>
>>> Gonzalo
>>>
>>>
>>> On 17 September 2015 at 12:39, Radu Gheorghe <radu.gheorghe@sematext.com
>>> > wrote:
>>>
>>>> Hi Gonzalo,
>>>>
>>>> Thanks for your reply. Unfortunately, I've already went that route with
>>>> no success. I can only find out one jar holding that, located in Morphlines:
>>>>
>>>> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all>
>>>> grep -RHls javax.servlet.AsyncContext .
>>>> ./target/lib/javax.servlet-3.0.0.v201112011016.jar
>>>>
>>>> grepping in the Solr and Flume directories got me no result. Do you
>>>> have more ideas?
>>>>
>>>> Best regards,
>>>> Radu
>>>>
>>>> --
>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>
>>>> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Usually that means you are loading different version of the servlet
>>>>> API.
>>>>> You need to do a bit of classpath troubleshooting to find which jars
>>>>> contain javax.servlet.AsyncContext and keep just the one included in
>>>>> Flume/lib
>>>>>
>>>>> Regards,
>>>>> Gonzalo
>>>>>
>>>>> On 17 September 2015 at 11:31, Radu Gheorghe <
>>>>> radu.gheorghe@sematext.com> wrote:
>>>>>
>>>>>> Hello flume users!
>>>>>>
>>>>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I
>>>>>> did that with 1.5 but now I seem to fail :(
>>>>>>
>>>>>> I got this error in the log (full exception below):
>>>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>>>> signer information does not match signer information of other classes in
>>>>>> the same package
>>>>>>
>>>>>> Basically, what I did was to install Flume 1.6, install Solr 5.3,
>>>>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to
>>>>>> Flume's classpath. Here's the full command line:
>>>>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
>>>>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
>>>>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
>>>>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f
>>>>>> conf/flume-conf.properties
>>>>>>
>>>>>> Here's my Java version, if it matters:
>>>>>> java version "1.7.0_79"
>>>>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
>>>>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>>>>>>
>>>>>> From what I can understand out off the Internet (thanks, Internet!)
>>>>>> this may be caused by multiple jars having different versions of the same
>>>>>> thing. But I can only find it in Morphlines (via grep -RHls in all the
>>>>>> classpaths). Maybe I'm bumping into
>>>>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to
>>>>>> verify this by compiling Kite 0.12.0 but it failed with:
>>>>>> [ERROR] Failed to execute goal
>>>>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-data:
>>>>>> Too many files with unapproved license: 2 See RAT report....
>>>>>>
>>>>>> So then I compiled only morphlines and that worked, only to bump into
>>>>>> the same error.
>>>>>>
>>>>>> As promised, below is the full exception. For completeness, I'm also
>>>>>> including the main config and the morphline config:
>>>>>>
>>>>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
>>>>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) -
>>>>>> Unable to start SinkRunner: {
>>>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585
>>>>>> counterGroup:{ name:null counters:{} } } - Exception follows.
>>>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>>>> signer information does not match signer information of other classes in
>>>>>> the same package
>>>>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
>>>>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
>>>>>> at
>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>>>>> at
>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>>> at
>>>>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
>>>>>> at
>>>>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
>>>>>> at
>>>>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
>>>>>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
>>>>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>>>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>>>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>>>>>> at
>>>>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>>>>>> at
>>>>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>>>>>> at
>>>>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>>>>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>>>>>> at
>>>>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>>
>>>>>> > cat conf/flume-conf.properties
>>>>>> agent.sources = spoolDir
>>>>>> agent.channels = spillableMem
>>>>>> agent.sinks = solr
>>>>>>
>>>>>> agent.sources.spoolDir.type = spooldir
>>>>>> agent.sources.spoolDir.spoolDir = /opt/logs
>>>>>> agent.sources.spoolDir.batchSize = 1000
>>>>>> agent.sources.spoolDir.channels = spillableMem
>>>>>>
>>>>>> agent.channels.spillableMem.type = SPILLABLEMEMORY
>>>>>> agent.channels.spillableMem.memoryCapacity = 100000
>>>>>> agent.channels.spillableMem.overflowCapacity = 1000000
>>>>>> agent.channels.spillableMem.checkpointDir = /var/run/flume
>>>>>> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>>>>>>
>>>>>> agent.sinks.solr.type =
>>>>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>>>>>> agent.sinks.solr.channel = spillableMem
>>>>>> agent.sinks.solr.morphlineFile = conf/morphline.conf
>>>>>> agent.sinks.solr.morphlineId = 1
>>>>>> agent.sinks.solr.batchSize = 10000
>>>>>>
>>>>>> > cat conf/morphline.conf
>>>>>> SOLR_LOCATOR : {
>>>>>> collection : gettingstarted
>>>>>> solrUrl : "http://192.168.1.3:8983/solr/"
>>>>>> batchSize : 5000
>>>>>> }
>>>>>>
>>>>>> morphlines : [
>>>>>> { id : 1
>>>>>> commands : [
>>>>>> { readLine { charset : UTF-8 } }
>>>>>> {
>>>>>> grok {
>>>>>> dictionaryFiles : [conf/grok-patterns]
>>>>>> expressions : {
>>>>>> message : """%{COMBINEDAPACHELOG}"""
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> # { generateUUID {
>>>>>> # field : id
>>>>>> # }
>>>>>> # }
>>>>>> {
>>>>>> loadSolr {
>>>>>> solrLocator : ${SOLR_LOCATOR}
>>>>>> }
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> ]
>>>>>>
>>>>>> I also have a grok-patterns file there, but I don't think it's
>>>>>> relevant.
>>>>>>
>>>>>> Any ideas are much appreciated!
>>>>>>
>>>>>> Thanks and best regards,
>>>>>> Radu
>>>>>> --
>>>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Posted by Gonzalo Herreros <gh...@gmail.com>.
Yes, it sounds like a dependency library version error.
You can take their the Cloudera Flume binaries, build from their branch of
Flume or just see how the manage the dependencies and do the same.
https://github.com/cloudera/flume-ng (look at their poms)
You can download binaries here:
https://repository.cloudera.com/cloudera/cloudera-repos/
Gonzalo
On 17 September 2015 at 14:21, Radu Gheorghe <ra...@sematext.com>
wrote:
> Thanks, Gonzalo! That actually got me further. A Solr exception later (got
> around it by removing 4.10 Solr libs which I assume don't work with my 5.3
> for which I already have libs in CP), then I got another error pointing the
> same sort of issue.
>
> So I just went ahead and removed all duplicated jars (though many were din
> different versions) from kite. With something as ugly as this:
> for lib in `ls -1 | sed s/-[0-9].*.jar$//`; do ls
> /opt/apache-flume-1.6.0-bin/lib/$lib* 2>/dev/null; if [ $? -eq 0 ]; then rm
> $lib*; fi; done
>
> Then Flume complained about some codahale metrics class missing, so I
> moved back the metrics-core library to kite. And now I get this:
>
> java.lang.VerifyError: Cannot inherit from final class
>
> I'm not sure if you need the whole stacktrace (I will paste it below) but
> I have again the suspicion of duplicated stuff or mismatched versions.
>
> I'm not sure I'm on the right path, so you were pointing me to different
> ones:
> - add the integration with Solr. Can you tell me which are the jars
> needed? From which subproject?
> - look at the version of Flume distributed by Cloudera. I can't find just
> Flume, I suppose I need to get the whole CDH or something like that?
>
> Thanks again for all your help!
>
> Best regards,
> Radu
>
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
> On Thu, Sep 17, 2015 at 2:50 PM, Gonzalo Herreros <gh...@gmail.com>
> wrote:
>
>> That's it.
>> Remove that jar from the Flume classpath so it doesn't interfere with the
>> one in Flume (version 2.5)
>>
>> Note Flume already has the kite libraries. I would just add the
>> integration with Solr (I suggest you have a look at the version of Flume
>> distributed by Cloudera)
>>
>> Gonzalo
>>
>>
>> On 17 September 2015 at 12:39, Radu Gheorghe <ra...@sematext.com>
>> wrote:
>>
>>> Hi Gonzalo,
>>>
>>> Thanks for your reply. Unfortunately, I've already went that route with
>>> no success. I can only find out one jar holding that, located in Morphlines:
>>>
>>> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all>
>>> grep -RHls javax.servlet.AsyncContext .
>>> ./target/lib/javax.servlet-3.0.0.v201112011016.jar
>>>
>>> grepping in the Solr and Flume directories got me no result. Do you have
>>> more ideas?
>>>
>>> Best regards,
>>> Radu
>>>
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gh...@gmail.com>
>>> wrote:
>>>
>>>> Usually that means you are loading different version of the servlet API.
>>>> You need to do a bit of classpath troubleshooting to find which jars
>>>> contain javax.servlet.AsyncContext and keep just the one included in
>>>> Flume/lib
>>>>
>>>> Regards,
>>>> Gonzalo
>>>>
>>>> On 17 September 2015 at 11:31, Radu Gheorghe <
>>>> radu.gheorghe@sematext.com> wrote:
>>>>
>>>>> Hello flume users!
>>>>>
>>>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did
>>>>> that with 1.5 but now I seem to fail :(
>>>>>
>>>>> I got this error in the log (full exception below):
>>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>>> signer information does not match signer information of other classes in
>>>>> the same package
>>>>>
>>>>> Basically, what I did was to install Flume 1.6, install Solr 5.3,
>>>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to
>>>>> Flume's classpath. Here's the full command line:
>>>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
>>>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
>>>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
>>>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f
>>>>> conf/flume-conf.properties
>>>>>
>>>>> Here's my Java version, if it matters:
>>>>> java version "1.7.0_79"
>>>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
>>>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>>>>>
>>>>> From what I can understand out off the Internet (thanks, Internet!)
>>>>> this may be caused by multiple jars having different versions of the same
>>>>> thing. But I can only find it in Morphlines (via grep -RHls in all the
>>>>> classpaths). Maybe I'm bumping into
>>>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to
>>>>> verify this by compiling Kite 0.12.0 but it failed with:
>>>>> [ERROR] Failed to execute goal
>>>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-data:
>>>>> Too many files with unapproved license: 2 See RAT report....
>>>>>
>>>>> So then I compiled only morphlines and that worked, only to bump into
>>>>> the same error.
>>>>>
>>>>> As promised, below is the full exception. For completeness, I'm also
>>>>> including the main config and the morphline config:
>>>>>
>>>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
>>>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) -
>>>>> Unable to start SinkRunner: {
>>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585
>>>>> counterGroup:{ name:null counters:{} } } - Exception follows.
>>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>>> signer information does not match signer information of other classes in
>>>>> the same package
>>>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
>>>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
>>>>> at
>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>>>> at
>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>>> at
>>>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
>>>>> at
>>>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
>>>>> at
>>>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
>>>>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
>>>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>>>>> at
>>>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>>>>> at
>>>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>>>>> at
>>>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>>>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>>>>> at
>>>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>>
>>>>> > cat conf/flume-conf.properties
>>>>> agent.sources = spoolDir
>>>>> agent.channels = spillableMem
>>>>> agent.sinks = solr
>>>>>
>>>>> agent.sources.spoolDir.type = spooldir
>>>>> agent.sources.spoolDir.spoolDir = /opt/logs
>>>>> agent.sources.spoolDir.batchSize = 1000
>>>>> agent.sources.spoolDir.channels = spillableMem
>>>>>
>>>>> agent.channels.spillableMem.type = SPILLABLEMEMORY
>>>>> agent.channels.spillableMem.memoryCapacity = 100000
>>>>> agent.channels.spillableMem.overflowCapacity = 1000000
>>>>> agent.channels.spillableMem.checkpointDir = /var/run/flume
>>>>> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>>>>>
>>>>> agent.sinks.solr.type =
>>>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>>>>> agent.sinks.solr.channel = spillableMem
>>>>> agent.sinks.solr.morphlineFile = conf/morphline.conf
>>>>> agent.sinks.solr.morphlineId = 1
>>>>> agent.sinks.solr.batchSize = 10000
>>>>>
>>>>> > cat conf/morphline.conf
>>>>> SOLR_LOCATOR : {
>>>>> collection : gettingstarted
>>>>> solrUrl : "http://192.168.1.3:8983/solr/"
>>>>> batchSize : 5000
>>>>> }
>>>>>
>>>>> morphlines : [
>>>>> { id : 1
>>>>> commands : [
>>>>> { readLine { charset : UTF-8 } }
>>>>> {
>>>>> grok {
>>>>> dictionaryFiles : [conf/grok-patterns]
>>>>> expressions : {
>>>>> message : """%{COMBINEDAPACHELOG}"""
>>>>> }
>>>>> }
>>>>> }
>>>>> # { generateUUID {
>>>>> # field : id
>>>>> # }
>>>>> # }
>>>>> {
>>>>> loadSolr {
>>>>> solrLocator : ${SOLR_LOCATOR}
>>>>> }
>>>>> }
>>>>> ]
>>>>> }
>>>>> ]
>>>>>
>>>>> I also have a grok-patterns file there, but I don't think it's
>>>>> relevant.
>>>>>
>>>>> Any ideas are much appreciated!
>>>>>
>>>>> Thanks and best regards,
>>>>> Radu
>>>>> --
>>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Posted by Radu Gheorghe <ra...@sematext.com>.
Thanks, Gonzalo! That actually got me further. A Solr exception later (got
around it by removing 4.10 Solr libs which I assume don't work with my 5.3
for which I already have libs in CP), then I got another error pointing the
same sort of issue.
So I just went ahead and removed all duplicated jars (though many were din
different versions) from kite. With something as ugly as this:
for lib in `ls -1 | sed s/-[0-9].*.jar$//`; do ls
/opt/apache-flume-1.6.0-bin/lib/$lib* 2>/dev/null; if [ $? -eq 0 ]; then rm
$lib*; fi; done
Then Flume complained about some codahale metrics class missing, so I moved
back the metrics-core library to kite. And now I get this:
java.lang.VerifyError: Cannot inherit from final class
I'm not sure if you need the whole stacktrace (I will paste it below) but I
have again the suspicion of duplicated stuff or mismatched versions.
I'm not sure I'm on the right path, so you were pointing me to different
ones:
- add the integration with Solr. Can you tell me which are the jars needed?
>From which subproject?
- look at the version of Flume distributed by Cloudera. I can't find just
Flume, I suppose I need to get the whole CDH or something like that?
Thanks again for all your help!
Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Thu, Sep 17, 2015 at 2:50 PM, Gonzalo Herreros <gh...@gmail.com>
wrote:
> That's it.
> Remove that jar from the Flume classpath so it doesn't interfere with the
> one in Flume (version 2.5)
>
> Note Flume already has the kite libraries. I would just add the
> integration with Solr (I suggest you have a look at the version of Flume
> distributed by Cloudera)
>
> Gonzalo
>
>
> On 17 September 2015 at 12:39, Radu Gheorghe <ra...@sematext.com>
> wrote:
>
>> Hi Gonzalo,
>>
>> Thanks for your reply. Unfortunately, I've already went that route with
>> no success. I can only find out one jar holding that, located in Morphlines:
>>
>> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all>
>> grep -RHls javax.servlet.AsyncContext .
>> ./target/lib/javax.servlet-3.0.0.v201112011016.jar
>>
>> grepping in the Solr and Flume directories got me no result. Do you have
>> more ideas?
>>
>> Best regards,
>> Radu
>>
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gh...@gmail.com>
>> wrote:
>>
>>> Usually that means you are loading different version of the servlet API.
>>> You need to do a bit of classpath troubleshooting to find which jars
>>> contain javax.servlet.AsyncContext and keep just the one included in
>>> Flume/lib
>>>
>>> Regards,
>>> Gonzalo
>>>
>>> On 17 September 2015 at 11:31, Radu Gheorghe <radu.gheorghe@sematext.com
>>> > wrote:
>>>
>>>> Hello flume users!
>>>>
>>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did
>>>> that with 1.5 but now I seem to fail :(
>>>>
>>>> I got this error in the log (full exception below):
>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>> signer information does not match signer information of other classes in
>>>> the same package
>>>>
>>>> Basically, what I did was to install Flume 1.6, install Solr 5.3,
>>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to
>>>> Flume's classpath. Here's the full command line:
>>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
>>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
>>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
>>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f
>>>> conf/flume-conf.properties
>>>>
>>>> Here's my Java version, if it matters:
>>>> java version "1.7.0_79"
>>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
>>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>>>>
>>>> From what I can understand out off the Internet (thanks, Internet!)
>>>> this may be caused by multiple jars having different versions of the same
>>>> thing. But I can only find it in Morphlines (via grep -RHls in all the
>>>> classpaths). Maybe I'm bumping into
>>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to
>>>> verify this by compiling Kite 0.12.0 but it failed with:
>>>> [ERROR] Failed to execute goal
>>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-data:
>>>> Too many files with unapproved license: 2 See RAT report....
>>>>
>>>> So then I compiled only morphlines and that worked, only to bump into
>>>> the same error.
>>>>
>>>> As promised, below is the full exception. For completeness, I'm also
>>>> including the main config and the morphline config:
>>>>
>>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
>>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) -
>>>> Unable to start SinkRunner: {
>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585
>>>> counterGroup:{ name:null counters:{} } } - Exception follows.
>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s
>>>> signer information does not match signer information of other classes in
>>>> the same package
>>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
>>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
>>>> at
>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>>> at
>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>> at
>>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
>>>> at
>>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
>>>> at
>>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
>>>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
>>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>>>> at
>>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>>>> at
>>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>>>> at
>>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>>>> at
>>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>>> > cat conf/flume-conf.properties
>>>> agent.sources = spoolDir
>>>> agent.channels = spillableMem
>>>> agent.sinks = solr
>>>>
>>>> agent.sources.spoolDir.type = spooldir
>>>> agent.sources.spoolDir.spoolDir = /opt/logs
>>>> agent.sources.spoolDir.batchSize = 1000
>>>> agent.sources.spoolDir.channels = spillableMem
>>>>
>>>> agent.channels.spillableMem.type = SPILLABLEMEMORY
>>>> agent.channels.spillableMem.memoryCapacity = 100000
>>>> agent.channels.spillableMem.overflowCapacity = 1000000
>>>> agent.channels.spillableMem.checkpointDir = /var/run/flume
>>>> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>>>>
>>>> agent.sinks.solr.type =
>>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>>>> agent.sinks.solr.channel = spillableMem
>>>> agent.sinks.solr.morphlineFile = conf/morphline.conf
>>>> agent.sinks.solr.morphlineId = 1
>>>> agent.sinks.solr.batchSize = 10000
>>>>
>>>> > cat conf/morphline.conf
>>>> SOLR_LOCATOR : {
>>>> collection : gettingstarted
>>>> solrUrl : "http://192.168.1.3:8983/solr/"
>>>> batchSize : 5000
>>>> }
>>>>
>>>> morphlines : [
>>>> { id : 1
>>>> commands : [
>>>> { readLine { charset : UTF-8 } }
>>>> {
>>>> grok {
>>>> dictionaryFiles : [conf/grok-patterns]
>>>> expressions : {
>>>> message : """%{COMBINEDAPACHELOG}"""
>>>> }
>>>> }
>>>> }
>>>> # { generateUUID {
>>>> # field : id
>>>> # }
>>>> # }
>>>> {
>>>> loadSolr {
>>>> solrLocator : ${SOLR_LOCATOR}
>>>> }
>>>> }
>>>> ]
>>>> }
>>>> ]
>>>>
>>>> I also have a grok-patterns file there, but I don't think it's relevant.
>>>>
>>>> Any ideas are much appreciated!
>>>>
>>>> Thanks and best regards,
>>>> Radu
>>>> --
>>>> Performance Monitoring * Log Analytics * Search Analytics
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>
>>>
>>>
>>
>
Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Posted by Gonzalo Herreros <gh...@gmail.com>.
That's it.
Remove that jar from the Flume classpath so it doesn't interfere with the
one in Flume (version 2.5)
Note Flume already has the kite libraries. I would just add the integration
with Solr (I suggest you have a look at the version of Flume distributed by
Cloudera)
Gonzalo
On 17 September 2015 at 12:39, Radu Gheorghe <ra...@sematext.com>
wrote:
> Hi Gonzalo,
>
> Thanks for your reply. Unfortunately, I've already went that route with no
> success. I can only find out one jar holding that, located in Morphlines:
>
> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all> grep
> -RHls javax.servlet.AsyncContext .
> ./target/lib/javax.servlet-3.0.0.v201112011016.jar
>
> grepping in the Solr and Flume directories got me no result. Do you have
> more ideas?
>
> Best regards,
> Radu
>
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gh...@gmail.com>
> wrote:
>
>> Usually that means you are loading different version of the servlet API.
>> You need to do a bit of classpath troubleshooting to find which jars
>> contain javax.servlet.AsyncContext and keep just the one included in
>> Flume/lib
>>
>> Regards,
>> Gonzalo
>>
>> On 17 September 2015 at 11:31, Radu Gheorghe <ra...@sematext.com>
>> wrote:
>>
>>> Hello flume users!
>>>
>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did
>>> that with 1.5 but now I seem to fail :(
>>>
>>> I got this error in the log (full exception below):
>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
>>> information does not match signer information of other classes in the same
>>> package
>>>
>>> Basically, what I did was to install Flume 1.6, install Solr 5.3,
>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to
>>> Flume's classpath. Here's the full command line:
>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f
>>> conf/flume-conf.properties
>>>
>>> Here's my Java version, if it matters:
>>> java version "1.7.0_79"
>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>>>
>>> From what I can understand out off the Internet (thanks, Internet!) this
>>> may be caused by multiple jars having different versions of the same thing.
>>> But I can only find it in Morphlines (via grep -RHls in all the
>>> classpaths). Maybe I'm bumping into
>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to
>>> verify this by compiling Kite 0.12.0 but it failed with:
>>> [ERROR] Failed to execute goal
>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-data:
>>> Too many files with unapproved license: 2 See RAT report....
>>>
>>> So then I compiled only morphlines and that worked, only to bump into
>>> the same error.
>>>
>>> As promised, below is the full exception. For completeness, I'm also
>>> including the main config and the morphline config:
>>>
>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) -
>>> Unable to start SinkRunner: {
>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585
>>> counterGroup:{ name:null counters:{} } } - Exception follows.
>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
>>> information does not match signer information of other classes in the same
>>> package
>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
>>> at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>> at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>> at
>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
>>> at
>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
>>> at
>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
>>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>>> at
>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>>> at
>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>>> at
>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>>> at
>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> > cat conf/flume-conf.properties
>>> agent.sources = spoolDir
>>> agent.channels = spillableMem
>>> agent.sinks = solr
>>>
>>> agent.sources.spoolDir.type = spooldir
>>> agent.sources.spoolDir.spoolDir = /opt/logs
>>> agent.sources.spoolDir.batchSize = 1000
>>> agent.sources.spoolDir.channels = spillableMem
>>>
>>> agent.channels.spillableMem.type = SPILLABLEMEMORY
>>> agent.channels.spillableMem.memoryCapacity = 100000
>>> agent.channels.spillableMem.overflowCapacity = 1000000
>>> agent.channels.spillableMem.checkpointDir = /var/run/flume
>>> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>>>
>>> agent.sinks.solr.type =
>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>>> agent.sinks.solr.channel = spillableMem
>>> agent.sinks.solr.morphlineFile = conf/morphline.conf
>>> agent.sinks.solr.morphlineId = 1
>>> agent.sinks.solr.batchSize = 10000
>>>
>>> > cat conf/morphline.conf
>>> SOLR_LOCATOR : {
>>> collection : gettingstarted
>>> solrUrl : "http://192.168.1.3:8983/solr/"
>>> batchSize : 5000
>>> }
>>>
>>> morphlines : [
>>> { id : 1
>>> commands : [
>>> { readLine { charset : UTF-8 } }
>>> {
>>> grok {
>>> dictionaryFiles : [conf/grok-patterns]
>>> expressions : {
>>> message : """%{COMBINEDAPACHELOG}"""
>>> }
>>> }
>>> }
>>> # { generateUUID {
>>> # field : id
>>> # }
>>> # }
>>> {
>>> loadSolr {
>>> solrLocator : ${SOLR_LOCATOR}
>>> }
>>> }
>>> ]
>>> }
>>> ]
>>>
>>> I also have a grok-patterns file there, but I don't think it's relevant.
>>>
>>> Any ideas are much appreciated!
>>>
>>> Thanks and best regards,
>>> Radu
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>
>>
>
Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Posted by Radu Gheorghe <ra...@sematext.com>.
Hi Gonzalo,
Thanks for your reply. Unfortunately, I've already went that route with no
success. I can only find out one jar holding that, located in Morphlines:
radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all> grep
-RHls javax.servlet.AsyncContext .
./target/lib/javax.servlet-3.0.0.v201112011016.jar
grepping in the Solr and Flume directories got me no result. Do you have
more ideas?
Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gh...@gmail.com>
wrote:
> Usually that means you are loading different version of the servlet API.
> You need to do a bit of classpath troubleshooting to find which jars
> contain javax.servlet.AsyncContext and keep just the one included in
> Flume/lib
>
> Regards,
> Gonzalo
>
> On 17 September 2015 at 11:31, Radu Gheorghe <ra...@sematext.com>
> wrote:
>
>> Hello flume users!
>>
>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did
>> that with 1.5 but now I seem to fail :(
>>
>> I got this error in the log (full exception below):
>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
>> information does not match signer information of other classes in the same
>> package
>>
>> Basically, what I did was to install Flume 1.6, install Solr 5.3, compile
>> Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to Flume's
>> classpath. Here's the full command line:
>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
>> -Djava.library.path= org.apache.flume.node.Application -n agent -f
>> conf/flume-conf.properties
>>
>> Here's my Java version, if it matters:
>> java version "1.7.0_79"
>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>>
>> From what I can understand out off the Internet (thanks, Internet!) this
>> may be caused by multiple jars having different versions of the same thing.
>> But I can only find it in Morphlines (via grep -RHls in all the
>> classpaths). Maybe I'm bumping into
>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to
>> verify this by compiling Kite 0.12.0 but it failed with:
>> [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.9:check
>> (default) on project kite-data: Too many files with unapproved license: 2
>> See RAT report....
>>
>> So then I compiled only morphlines and that worked, only to bump into the
>> same error.
>>
>> As promised, below is the full exception. For completeness, I'm also
>> including the main config and the morphline config:
>>
>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) -
>> Unable to start SinkRunner: {
>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585
>> counterGroup:{ name:null counters:{} } } - Exception follows.
>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
>> information does not match signer information of other classes in the same
>> package
>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at
>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
>> at
>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
>> at
>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
>> at
>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
>> at
>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
>> at
>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>> at
>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> > cat conf/flume-conf.properties
>> agent.sources = spoolDir
>> agent.channels = spillableMem
>> agent.sinks = solr
>>
>> agent.sources.spoolDir.type = spooldir
>> agent.sources.spoolDir.spoolDir = /opt/logs
>> agent.sources.spoolDir.batchSize = 1000
>> agent.sources.spoolDir.channels = spillableMem
>>
>> agent.channels.spillableMem.type = SPILLABLEMEMORY
>> agent.channels.spillableMem.memoryCapacity = 100000
>> agent.channels.spillableMem.overflowCapacity = 1000000
>> agent.channels.spillableMem.checkpointDir = /var/run/flume
>> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>>
>> agent.sinks.solr.type =
>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
>> agent.sinks.solr.channel = spillableMem
>> agent.sinks.solr.morphlineFile = conf/morphline.conf
>> agent.sinks.solr.morphlineId = 1
>> agent.sinks.solr.batchSize = 10000
>>
>> > cat conf/morphline.conf
>> SOLR_LOCATOR : {
>> collection : gettingstarted
>> solrUrl : "http://192.168.1.3:8983/solr/"
>> batchSize : 5000
>> }
>>
>> morphlines : [
>> { id : 1
>> commands : [
>> { readLine { charset : UTF-8 } }
>> {
>> grok {
>> dictionaryFiles : [conf/grok-patterns]
>> expressions : {
>> message : """%{COMBINEDAPACHELOG}"""
>> }
>> }
>> }
>> # { generateUUID {
>> # field : id
>> # }
>> # }
>> {
>> loadSolr {
>> solrLocator : ${SOLR_LOCATOR}
>> }
>> }
>> ]
>> }
>> ]
>>
>> I also have a grok-patterns file there, but I don't think it's relevant.
>>
>> Any ideas are much appreciated!
>>
>> Thanks and best regards,
>> Radu
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>
>
Re: Dependency issues while starting Flume 1.6 with MorphlineSolrSink
Posted by Gonzalo Herreros <gh...@gmail.com>.
Usually that means you are loading different version of the servlet API.
You need to do a bit of classpath troubleshooting to find which jars
contain javax.servlet.AsyncContext and keep just the one included in
Flume/lib
Regards,
Gonzalo
On 17 September 2015 at 11:31, Radu Gheorghe <ra...@sematext.com>
wrote:
> Hello flume users!
>
> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did
> that with 1.5 but now I seem to fail :(
>
> I got this error in the log (full exception below):
> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
> information does not match signer information of other classes in the same
> package
>
> Basically, what I did was to install Flume 1.6, install Solr 5.3, compile
> Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to Flume's
> classpath. Here's the full command line:
> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m
> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp
> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*'
> -Djava.library.path= org.apache.flume.node.Application -n agent -f
> conf/flume-conf.properties
>
> Here's my Java version, if it matters:
> java version "1.7.0_79"
> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64)
> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
>
> From what I can understand out off the Internet (thanks, Internet!) this
> may be caused by multiple jars having different versions of the same thing.
> But I can only find it in Morphlines (via grep -RHls in all the
> classpaths). Maybe I'm bumping into
> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to verify
> this by compiling Kite 0.12.0 but it failed with:
> [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.9:check
> (default) on project kite-data: Too many files with unapproved license: 2
> See RAT report....
>
> So then I compiled only morphlines and that worked, only to bump into the
> same error.
>
> As promised, below is the full exception. For completeness, I'm also
> including the main config and the morphline config:
>
> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8]
> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) -
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585 counterGroup:{
> name:null counters:{} } } - Exception follows.
> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s signer
> information does not match signer information of other classes in the same
> package
> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224)
> at
> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176)
> at
> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91)
> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43)
> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126)
> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55)
> at
> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101)
> at
> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97)
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> > cat conf/flume-conf.properties
> agent.sources = spoolDir
> agent.channels = spillableMem
> agent.sinks = solr
>
> agent.sources.spoolDir.type = spooldir
> agent.sources.spoolDir.spoolDir = /opt/logs
> agent.sources.spoolDir.batchSize = 1000
> agent.sources.spoolDir.channels = spillableMem
>
> agent.channels.spillableMem.type = SPILLABLEMEMORY
> agent.channels.spillableMem.memoryCapacity = 100000
> agent.channels.spillableMem.overflowCapacity = 1000000
> agent.channels.spillableMem.checkpointDir = /var/run/flume
> agent.channels.spillableMem.dataDirs = /var/run/flume-data
>
> agent.sinks.solr.type =
> org.apache.flume.sink.solr.morphline.MorphlineSolrSink
> agent.sinks.solr.channel = spillableMem
> agent.sinks.solr.morphlineFile = conf/morphline.conf
> agent.sinks.solr.morphlineId = 1
> agent.sinks.solr.batchSize = 10000
>
> > cat conf/morphline.conf
> SOLR_LOCATOR : {
> collection : gettingstarted
> solrUrl : "http://192.168.1.3:8983/solr/"
> batchSize : 5000
> }
>
> morphlines : [
> { id : 1
> commands : [
> { readLine { charset : UTF-8 } }
> {
> grok {
> dictionaryFiles : [conf/grok-patterns]
> expressions : {
> message : """%{COMBINEDAPACHELOG}"""
> }
> }
> }
> # { generateUUID {
> # field : id
> # }
> # }
> {
> loadSolr {
> solrLocator : ${SOLR_LOCATOR}
> }
> }
> ]
> }
> ]
>
> I also have a grok-patterns file there, but I don't think it's relevant.
>
> Any ideas are much appreciated!
>
> Thanks and best regards,
> Radu
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>