Re: Text search in Spark on compressed bz2 files

[Mich Talebzadeh <mi...@cloudtechnologypartners.co.uk>]

 

On 16/02/2016 00:02, Mich Talebzadeh wrote: 

> Hi 
> 
> It does not seem that sc.textFile supports search on log files compressed with bzip2 
> 
> val logfile2 = sc.textFile("hdfs://rhes564:9000/test/REP_*.log.bz2") 
> 
> val df2 = logfile2.toDF("line")
> val errors2 = df2.filter(col("line").contains("E."))
> errors2.count() 
> 
> Nothing is returned. Is there as method call to read a compressed file? 
> 
> Thanks. 
> -- 
> 
> Dr Mich Talebzadeh
> 
> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> 
> http://talebzadehmich.wordpress.com
> 
> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Cloud Technology Partners Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Cloud Technology partners Ltd, its subsidiaries nor their employees accept any responsibility.

-- 

Dr Mich Talebzadeh

LinkedIn
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

http://talebzadehmich.wordpress.com

NOTE: The information in this email is proprietary and confidential.
This message is for the designated recipient only, if you are not the
intended recipient, you should destroy it immediately. Any information
in this message shall not be understood as given or endorsed by Cloud
Technology Partners Ltd, its subsidiaries or their employees, unless
expressly so stated. It is the responsibility of the recipient to ensure
that this email is virus free, therefore neither Cloud Technology
partners Ltd, its subsidiaries nor their employees accept any
responsibility.