You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Albert Manyà <al...@eml.cc> on 2014/12/11 17:43:38 UTC
Exception using amazonaws library
Hi,
I've made a simple script in scala that after doing a spark sql query it
sends the result to AWS's cloudwatch.
I've tested both parts individually (the spark sql one and the
cloudwatch one) and they worked fine. The trouble comes when I execute
the script through spark-submit that gives me the following exception:
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V
at
com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:95)
at
com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:193)
at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120)
at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:104)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:171)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:152)
at
com.scmspain.synapse.SynapseMonitor$.sendMetrics(SynapseMonitor.scala:50)
at
com.scmspain.synapse.SynapseMonitor$.main(SynapseMonitor.scala:45)
at
com.scmspain.synapse.SynapseMonitor.main(SynapseMonitor.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
my class is com.scmspain.synapse.SynapseMonitor
I've build my script with sbt assembly, having the following
dependencies:
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.1.0" % "provided",
"org.apache.spark" %% "spark-sql" % "1.1.0" % "provided",
"com.amazonaws" % "aws-java-sdk-cloudwatch" % "1.9.10"
)
I've unzipped the generated jar assembly and searched for the
HttpConnectionParams.class and I've found it out under
org/apache/http/params and having the following signature for
setSoKeepalive:
public static void setSoKeepalive(HttpParams params, boolean
enableKeepalive)
At this point I'm stuck and didn't know where to keep looking... some
help would be greatly appreciated :)
Thank you very much!
--
Albert Manyà
albertmp@eml.cc
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Exception using amazonaws library
Posted by Albert Manyà <al...@eml.cc>.
Hi, Thanks for your reply.
I tried with the jar you pointed but It complains about missing
HttpPatch that appears on httpclient 4.2
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/http/client/methods/HttpPatch at
com.amazonaws.http.AmazonHttpClient.<clinit>(AmazonHttpClient.java:129)
at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120)
at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:104)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:171)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:152)
at
com.scmspain.synapse.SynapseMonitor$.sendMetrics(SynapseMonitor.scala:48)
at com.scmspain.synapse.SynapseMonitor$.main(SynapseMonitor.scala:28) at
com.scmspain.synapse.SynapseMonitor.main(SynapseMonitor.scala) ...
I do not understand who is compiled against such an old version of
httpclient, I see in the project dependencies that amazonaws 1.9.10
depends on httclient 4.3... It is spark who is compiled against an old
version of amazonaws?
Thanks.
--
Albert Manyà albertmp@eml.cc
On Fri, Dec 12, 2014, at 09:27 AM, Akhil Das wrote:
> Its a jar conflict (http-client[1] jar), You could download the
> appropriate version of that jar and put that in the classpath before
> your assembly jar and hopefully it will avoid the conflict.
>
> Thanks Best Regards
>
> On Thu, Dec 11, 2014 at 10:13 PM, Albert Manyà
> <al...@eml.cc> wrote:
>> Hi,
>>
>>
I've made a simple script in scala that after doing a spark sql query it
>>
sends the result to AWS's cloudwatch.
>>
>>
I've tested both parts individually (the spark sql one and the
>>
cloudwatch one) and they worked fine. The trouble comes when I execute
>>
the script through spark-submit that gives me the following exception:
>>
>>
Exception in thread "main" java.lang.NoSuchMethodError:
>>
org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V
>>
at
>>
com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:95)
>>
at
>>
com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:193)
>>
at
>>
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120)
>>
at
>>
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:104)
>>
at
>>
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:171)
>>
at
>>
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:152)
>>
at
>>
com.scmspain.synapse.SynapseMonitor$.sendMetrics(SynapseMonitor.scala:50)
>>
at
>>
com.scmspain.synapse.SynapseMonitor$.main(SynapseMonitor.scala:45)
>>
at
>>
com.scmspain.synapse.SynapseMonitor.main(SynapseMonitor.scala)
>>
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
at
>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
at java.lang.reflect.Method.invoke(Method.java:483)
>>
at
>>
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
>>
at
>>
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>>
my class is com.scmspain.synapse.SynapseMonitor
>>
>>
I've build my script with sbt assembly, having the following
>>
dependencies:
>>
>>
libraryDependencies ++= Seq(
>>
"org.apache.spark" %% "spark-core" % "1.1.0" % "provided",
>>
"org.apache.spark" %% "spark-sql" % "1.1.0" % "provided",
>>
"com.amazonaws" % "aws-java-sdk-cloudwatch" % "1.9.10"
>>
)
>>
>>
I've unzipped the generated jar assembly and searched for the
>>
HttpConnectionParams.class and I've found it out under
>>
org/apache/http/params and having the following signature for
>>
setSoKeepalive:
>>
>>
public static void setSoKeepalive(HttpParams params, boolean
>>
enableKeepalive)
>>
>>
At this point I'm stuck and didn't know where to keep looking... some
>>
help would be greatly appreciated :)
>>
>>
Thank you very much!
>>
>>
--
>>
Albert Manyà
>>
albertmp@eml.cc
>>
>>
---------------------------------------------------------------------
>>
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>
For additional commands, e-mail: user-help@spark.apache.org
Links:
1. http://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient/4.0-alpha4
Re: Exception using amazonaws library
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Its a jar conflict (http-client
<http://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient/4.0-alpha4>
jar), You could download the appropriate version of that jar and put that
in the classpath before your assembly jar and hopefully it will avoid the
conflict.
Thanks
Best Regards
On Thu, Dec 11, 2014 at 10:13 PM, Albert Manyà <al...@eml.cc> wrote:
>
> Hi,
>
> I've made a simple script in scala that after doing a spark sql query it
> sends the result to AWS's cloudwatch.
>
> I've tested both parts individually (the spark sql one and the
> cloudwatch one) and they worked fine. The trouble comes when I execute
> the script through spark-submit that gives me the following exception:
>
> Exception in thread "main" java.lang.NoSuchMethodError:
>
> org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V
> at
>
> com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:95)
> at
>
> com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:193)
> at
>
> com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120)
> at
>
> com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:104)
> at
>
> com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:171)
> at
>
> com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:152)
> at
>
> com.scmspain.synapse.SynapseMonitor$.sendMetrics(SynapseMonitor.scala:50)
> at
> com.scmspain.synapse.SynapseMonitor$.main(SynapseMonitor.scala:45)
> at
> com.scmspain.synapse.SynapseMonitor.main(SynapseMonitor.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
> at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> my class is com.scmspain.synapse.SynapseMonitor
>
> I've build my script with sbt assembly, having the following
> dependencies:
>
> libraryDependencies ++= Seq(
> "org.apache.spark" %% "spark-core" % "1.1.0" % "provided",
> "org.apache.spark" %% "spark-sql" % "1.1.0" % "provided",
> "com.amazonaws" % "aws-java-sdk-cloudwatch" % "1.9.10"
> )
>
> I've unzipped the generated jar assembly and searched for the
> HttpConnectionParams.class and I've found it out under
> org/apache/http/params and having the following signature for
> setSoKeepalive:
>
> public static void setSoKeepalive(HttpParams params, boolean
> enableKeepalive)
>
> At this point I'm stuck and didn't know where to keep looking... some
> help would be greatly appreciated :)
>
> Thank you very much!
>
> --
> Albert Manyà
> albertmp@eml.cc
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>