You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by satishl <sa...@gmail.com> on 2017/04/07 06:34:29 UTC
reading snappy eventlog files from hdfs using spark
Hi, I am planning to process spark app eventlogs with another spark app.
These event logs are saved with snappy compression (extension: .snappy).
When i read the file in a new spark app - i get a snappy library not found
error. I am confused as to how can spark write eventlog in snappy format
without an error, but reading fails with the above error.
Any help in unblocking myself to read snappy eventlog files from hdfs using
spark?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/reading-snappy-eventlog-files-from-hdfs-using-spark-tp28577.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: reading snappy eventlog files from hdfs using spark
Posted by Jörn Franke <jo...@gmail.com>.
How do you read them?
> On 7. Apr 2017, at 12:11, Jacek Laskowski <ja...@japila.pl> wrote:
>
> Hi,
>
> If your Spark app uses snappy in the code, define an appropriate library dependency to have it on classpath. Don't rely on transitive dependencies.
>
> Jacek
>
> On 7 Apr 2017 8:34 a.m., "satishl" <sa...@gmail.com> wrote:
> Hi, I am planning to process spark app eventlogs with another spark app.
> These event logs are saved with snappy compression (extension: .snappy).
> When i read the file in a new spark app - i get a snappy library not found
> error. I am confused as to how can spark write eventlog in snappy format
> without an error, but reading fails with the above error.
>
> Any help in unblocking myself to read snappy eventlog files from hdfs using
> spark?
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/reading-snappy-eventlog-files-from-hdfs-using-spark-tp28577.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: reading snappy eventlog files from hdfs using spark
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
If your Spark app uses snappy in the code, define an appropriate library
dependency to have it on classpath. Don't rely on transitive dependencies.
Jacek
On 7 Apr 2017 8:34 a.m., "satishl" <sa...@gmail.com> wrote:
Hi, I am planning to process spark app eventlogs with another spark app.
These event logs are saved with snappy compression (extension: .snappy).
When i read the file in a new spark app - i get a snappy library not found
error. I am confused as to how can spark write eventlog in snappy format
without an error, but reading fails with the above error.
Any help in unblocking myself to read snappy eventlog files from hdfs using
spark?
--
View this message in context: http://apache-spark-user-list.
1001560.n3.nabble.com/reading-snappy-eventlog-files-from-
hdfs-using-spark-tp28577.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org