You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by 진상형 <js...@gmail.com> on 2016/01/25 13:00:09 UTC

declare function but not works (value ~ is not a member of ~)

I want to use custom function, but it doesn't work.

Here are codes and outs

/// codes
def unionByName(a: DataFrame, b:DataFrame): DataFrame = {
    val columns = a.dtypes.intersect(b.dtypes).map{case (c, _) => col(c)}
    a.select(columns: _*).unionAll(b.select(columns: _*))
}
raw1.select(columns: _*).unionByName(raw3.select(columns: _*)).count()

/// outputs
unionByName: (a: org.apache.spark.sql.DataFrame, b: org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame
<console>:141: error: value unionByName is not a member of org.apache.spark.sql.DataFrame
              raw1.select(columns: _*).unionByName(raw3.select(columns: _*)).count()


p.s. Is there any function to check types of RDD or else? (e.g. dtypes for checking types of column)

Re: declare function but not works (value ~ is not a member of ~)

Posted by 진상형 <js...@gmail.com>.
Oh, now I get it 

Thanks, it works fine.


> On Jan 25, 2016, at 9:07 PM, Chiwan Park <ch...@apache.org> wrote:
> 
> I think that it is not related to Zeppelin. Because your function `unionByName` is not method of `DataFrame`, you have to call it like following:
> 
> val unionized = unionByName(raw1.select(columns: _*), raw3.select(columns: _*))
> 
> Regards,
> Chiwan Park
> 
>> On Jan 25, 2016, at 9:00 PM, 진상형 <js...@gmail.com> wrote:
>> 
>> I want to use custom function, but it doesn't work.
>> 
>> Here are codes and outs
>> 
>> /// codes
>> def unionByName(a: DataFrame, b:DataFrame): DataFrame = {
>>    val columns = a.dtypes.intersect(b.dtypes).map{case (c, _) => col(c)}
>>    a.select(columns: _*).unionAll(b.select(columns: _*))
>> }
>> raw1.select(columns: _*).unionByName(raw3.select(columns: _*)).count()
>> 
>> /// outputs
>> unionByName: (a: org.apache.spark.sql.DataFrame, b: org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame
>> <console>:141: error: value unionByName is not a member of org.apache.spark.sql.DataFrame
>>              raw1.select(columns: _*).unionByName(raw3.select(columns: _*)).count()
>> 
>> 
>> 
>> p.s. Is there any function to check types of RDD or else? (e.g. dtypes for checking types of column)
>

Re: declare function but not works (value ~ is not a member of ~)

Posted by Chiwan Park <ch...@apache.org>.
I think that it is not related to Zeppelin. Because your function `unionByName` is not method of `DataFrame`, you have to call it like following:

val unionized = unionByName(raw1.select(columns: _*), raw3.select(columns: _*))

Regards,
Chiwan Park

> On Jan 25, 2016, at 9:00 PM, 진상형 <js...@gmail.com> wrote:
> 
> I want to use custom function, but it doesn't work.
> 
> Here are codes and outs
> 
> /// codes
> def unionByName(a: DataFrame, b:DataFrame): DataFrame = {
>     val columns = a.dtypes.intersect(b.dtypes).map{case (c, _) => col(c)}
>     a.select(columns: _*).unionAll(b.select(columns: _*))
> }
> raw1.select(columns: _*).unionByName(raw3.select(columns: _*)).count()
> 
> /// outputs
> unionByName: (a: org.apache.spark.sql.DataFrame, b: org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame
> <console>:141: error: value unionByName is not a member of org.apache.spark.sql.DataFrame
>               raw1.select(columns: _*).unionByName(raw3.select(columns: _*)).count()
> 
> 
> 
> p.s. Is there any function to check types of RDD or else? (e.g. dtypes for checking types of column)