You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by S Malligarjunan <sm...@yahoo.com.INVALID> on 2014/08/25 20:46:35 UTC
HiveContext ouput log file
Hello All,
I have executed the following udf sql in my spark hivecontext,
hiveContext.hql(select count(t1.col1) from t1 join t2 where myUDF(t1.id , t2.id) = true)
Where do i find the count output?
Thanks and Regards,
Sankar S.
Re: HiveContext ouput log file
Posted by S Malligarjunan <sm...@yahoo.com.INVALID>.
Hello Michel,
I get the following error if i execute the query with collect method
Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCod
Thanks and Regards,
Sankar S.
On Tuesday, 26 August 2014, 0:58, Michael Armbrust <mi...@databricks.com> wrote:
Just like with normal Spark Jobs, that command returns an RDD that contains the lineage for computing the answer but does not actually compute the answer. You'll need to run collect() on the RDD in order to get the result.
On Mon, Aug 25, 2014 at 11:46 AM, S Malligarjunan <sm...@yahoo.com.invalid> wrote:
Hello All,
>
>
>I have executed the following udf sql in my spark hivecontext,
>
>
>hiveContext.hql(select count(t1.col1) from t1 join t2 where myUDF(t1.id , t2.id) = true)
>Where do i find the count output?
>
>Thanks and Regards,
>Sankar S.
>
>
Re: HiveContext ouput log file
Posted by Michael Armbrust <mi...@databricks.com>.
Just like with normal Spark Jobs, that command returns an RDD that contains
the lineage for computing the answer but does not actually compute the
answer. You'll need to run collect() on the RDD in order to get the result.
On Mon, Aug 25, 2014 at 11:46 AM, S Malligarjunan <
smalligarjunan@yahoo.com.invalid> wrote:
> Hello All,
>
> I have executed the following udf sql in my spark hivecontext,
>
> hiveContext.hql(select count(t1.col1) from t1 join t2 where myUDF(t1.id ,
> t2.id) = true)
> Where do i find the count output?
>
> Thanks and Regards,
> Sankar S.
>
>