You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Neelesh <ne...@gmail.com> on 2015/03/29 09:10:34 UTC
Untangling dependency issues in spark streaming
Hi,
My streaming app uses org.apache.httpcomponent:httpclient:4.3.6, but
spark uses 4.2.6 , and I believe thats what's causing the following error.
I've tried setting
spark.executor.userClassPathFirst & spark.driver.userClassPathFirst to true
in the config, but that does not solve it either. Finally I had to resort
to relocating classes using maven shade plugin while building my apps uber
jar, using
<relocations>
<relocation>
<pattern>org.apache.http</pattern>
<shadedPattern>org.shaded.apache.http</shadedPattern>
</relocation>
</relocations>
Hope this is useful to others in the same situation. It would be
really great to deal with this the right way (like tomcat or any other
servlet container - classloader hierarchy etc).
Caused by: java.lang.NoSuchFieldError: INSTANCE
at
org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:52)
at
org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:56)
at
org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<clinit>(DefaultHttpRequestWriterFactory.java:46)
at
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:72)
at
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:84)
at
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<clinit>(ManagedHttpClientConnectionFactory.java:59)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)
and ...
Caused by: java.lang.NoClassDefFoundError: Could not initialize class
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)
Re: Untangling dependency issues in spark streaming
Posted by Ted Yu <yu...@gmail.com>.
For Gradle, there are:
https://github.com/musketyr/gradle-fatjar-plugin
https://github.com/johnrengelman/shadow
FYI
On Sun, Mar 29, 2015 at 4:29 PM, jay vyas <ja...@gmail.com>
wrote:
> thanks for posting this! Ive ran into similar issues before, and generally
> its a bad idea to swap the libraries out and "pray fot the best", so the
> shade functionality is probably the best feature.
>
> Unfortunately, im not sure how well SBT and Gradle support shading... how
> do folks using next gen build tools solve this problem?
>
>
>
> On Sun, Mar 29, 2015 at 3:10 AM, Neelesh <ne...@gmail.com> wrote:
>
>> Hi,
>> My streaming app uses org.apache.httpcomponent:httpclient:4.3.6, but
>> spark uses 4.2.6 , and I believe thats what's causing the following error.
>> I've tried setting
>> spark.executor.userClassPathFirst & spark.driver.userClassPathFirst to
>> true in the config, but that does not solve it either. Finally I had to
>> resort to relocating classes using maven shade plugin while building my
>> apps uber jar, using
>>
>> <relocations>
>> <relocation>
>> <pattern>org.apache.http</pattern>
>> <shadedPattern>org.shaded.apache.http</shadedPattern>
>> </relocation>
>> </relocations>
>>
>>
>> Hope this is useful to others in the same situation. It would be really great to deal with this the right way (like tomcat or any other servlet container - classloader hierarchy etc).
>>
>>
>> Caused by: java.lang.NoSuchFieldError: INSTANCE
>> at
>> org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:52)
>> at
>> org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:56)
>> at
>> org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<clinit>(DefaultHttpRequestWriterFactory.java:46)
>> at
>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:72)
>> at
>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:84)
>> at
>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<clinit>(ManagedHttpClientConnectionFactory.java:59)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)
>>
>> and ...
>> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
>> at
>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)
>>
>>
>
>
> --
> jay vyas
>
Re: Untangling dependency issues in spark streaming
Posted by jay vyas <ja...@gmail.com>.
thanks for posting this! Ive ran into similar issues before, and generally
its a bad idea to swap the libraries out and "pray fot the best", so the
shade functionality is probably the best feature.
Unfortunately, im not sure how well SBT and Gradle support shading... how
do folks using next gen build tools solve this problem?
On Sun, Mar 29, 2015 at 3:10 AM, Neelesh <ne...@gmail.com> wrote:
> Hi,
> My streaming app uses org.apache.httpcomponent:httpclient:4.3.6, but
> spark uses 4.2.6 , and I believe thats what's causing the following error.
> I've tried setting
> spark.executor.userClassPathFirst & spark.driver.userClassPathFirst to
> true in the config, but that does not solve it either. Finally I had to
> resort to relocating classes using maven shade plugin while building my
> apps uber jar, using
>
> <relocations>
> <relocation>
> <pattern>org.apache.http</pattern>
> <shadedPattern>org.shaded.apache.http</shadedPattern>
> </relocation>
> </relocations>
>
>
> Hope this is useful to others in the same situation. It would be really great to deal with this the right way (like tomcat or any other servlet container - classloader hierarchy etc).
>
>
> Caused by: java.lang.NoSuchFieldError: INSTANCE
> at
> org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:52)
> at
> org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:56)
> at
> org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<clinit>(DefaultHttpRequestWriterFactory.java:46)
> at
> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:72)
> at
> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:84)
> at
> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<clinit>(ManagedHttpClientConnectionFactory.java:59)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)
>
> and ...
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:494)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:138)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:114)
>
>
--
jay vyas