You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ophir Cohen <op...@gmail.com> on 2015/03/17 14:34:28 UTC
HiveContext can't find registered function
Hi Guys,
I'm registering a function using:
sqlc.registerFunction("makeEstEntry",ReutersDataFunctions.makeEstEntry _)
Then I register the table and try to query the table using that function
and I get:
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved
attributes:
'makeEstEntry(numest#20,median#21,mean#22,stddev#23,high#24,low#25,currency_#26,units#27,'basic_null_diluted_d)
AS FY0#2837, tree:
Thanks!
Ophir
Re: HiveContext can't find registered function
Posted by Ophir Cohen <op...@gmail.com>.
Very helpful!
Thank you
On Mar 17, 2015 9:24 PM, "Yin Huai" <yh...@databricks.com> wrote:
> Initially, an attribute reference (column reference), like selecting a
> column from a table, is not resolved since we do not know if the reference
> is valid or not (if this column exists in the underlying table). In the
> query compilation process, we will first analyze this query and resolved
> those attribute references. A resolved attribute reference means that this
> reference is valid and we know where to get the column values from the
> input. Hope this is helpful.
>
> On Tue, Mar 17, 2015 at 2:19 PM, Ophir Cohen <op...@gmail.com> wrote:
>
>> Thanks you for the answer and one more question: what does it mean
>> 'resolved attribute'?
>> On Mar 17, 2015 8:14 PM, "Yin Huai" <yh...@databricks.com> wrote:
>>
>>> The number is an id we used internally to identify an resolved
>>> Attribute. Looks like basic_null_diluted_d was not resolved since there
>>> is no id associated with it.
>>>
>>> On Tue, Mar 17, 2015 at 2:08 PM, Ophir Cohen <op...@gmail.com> wrote:
>>>
>>>> Interesting, I thought the problem is with the method itself.
>>>> I will check it soon and update.
>>>> Can you elaborate what does it mean the # and the number? Is that a
>>>> reference to the field in the rdd?
>>>> Thank you,
>>>> Ophir
>>>> On Mar 17, 2015 7:06 PM, "Yin Huai" <yh...@databricks.com> wrote:
>>>>
>>>>> Seems "basic_null_diluted_d" was not resolved? Can you check if
>>>>> basic_null_diluted_d is in you table?
>>>>>
>>>>> On Tue, Mar 17, 2015 at 9:34 AM, Ophir Cohen <op...@gmail.com> wrote:
>>>>>
>>>>>> Hi Guys,
>>>>>> I'm registering a function using:
>>>>>> sqlc.registerFunction("makeEstEntry",ReutersDataFunctions.makeEstEntry
>>>>>> _)
>>>>>>
>>>>>> Then I register the table and try to query the table using that
>>>>>> function and I get:
>>>>>> org.apache.spark.sql.catalyst.errors.package$TreeNodeException:
>>>>>> Unresolved attributes:
>>>>>> 'makeEstEntry(numest#20,median#21,mean#22,stddev#23,high#24,low#25,currency_#26,units#27,'basic_null_diluted_d)
>>>>>> AS FY0#2837, tree:
>>>>>>
>>>>>> Thanks!
>>>>>> Ophir
>>>>>>
>>>>>
>>>>>
>>>
>
Re: HiveContext can't find registered function
Posted by Yin Huai <yh...@databricks.com>.
Initially, an attribute reference (column reference), like selecting a
column from a table, is not resolved since we do not know if the reference
is valid or not (if this column exists in the underlying table). In the
query compilation process, we will first analyze this query and resolved
those attribute references. A resolved attribute reference means that this
reference is valid and we know where to get the column values from the
input. Hope this is helpful.
On Tue, Mar 17, 2015 at 2:19 PM, Ophir Cohen <op...@gmail.com> wrote:
> Thanks you for the answer and one more question: what does it mean
> 'resolved attribute'?
> On Mar 17, 2015 8:14 PM, "Yin Huai" <yh...@databricks.com> wrote:
>
>> The number is an id we used internally to identify an resolved Attribute.
>> Looks like basic_null_diluted_d was not resolved since there is no id
>> associated with it.
>>
>> On Tue, Mar 17, 2015 at 2:08 PM, Ophir Cohen <op...@gmail.com> wrote:
>>
>>> Interesting, I thought the problem is with the method itself.
>>> I will check it soon and update.
>>> Can you elaborate what does it mean the # and the number? Is that a
>>> reference to the field in the rdd?
>>> Thank you,
>>> Ophir
>>> On Mar 17, 2015 7:06 PM, "Yin Huai" <yh...@databricks.com> wrote:
>>>
>>>> Seems "basic_null_diluted_d" was not resolved? Can you check if
>>>> basic_null_diluted_d is in you table?
>>>>
>>>> On Tue, Mar 17, 2015 at 9:34 AM, Ophir Cohen <op...@gmail.com> wrote:
>>>>
>>>>> Hi Guys,
>>>>> I'm registering a function using:
>>>>> sqlc.registerFunction("makeEstEntry",ReutersDataFunctions.makeEstEntry
>>>>> _)
>>>>>
>>>>> Then I register the table and try to query the table using that
>>>>> function and I get:
>>>>> org.apache.spark.sql.catalyst.errors.package$TreeNodeException:
>>>>> Unresolved attributes:
>>>>> 'makeEstEntry(numest#20,median#21,mean#22,stddev#23,high#24,low#25,currency_#26,units#27,'basic_null_diluted_d)
>>>>> AS FY0#2837, tree:
>>>>>
>>>>> Thanks!
>>>>> Ophir
>>>>>
>>>>
>>>>
>>
Re: HiveContext can't find registered function
Posted by Ophir Cohen <op...@gmail.com>.
Thanks you for the answer and one more question: what does it mean
'resolved attribute'?
On Mar 17, 2015 8:14 PM, "Yin Huai" <yh...@databricks.com> wrote:
> The number is an id we used internally to identify an resolved Attribute.
> Looks like basic_null_diluted_d was not resolved since there is no id
> associated with it.
>
> On Tue, Mar 17, 2015 at 2:08 PM, Ophir Cohen <op...@gmail.com> wrote:
>
>> Interesting, I thought the problem is with the method itself.
>> I will check it soon and update.
>> Can you elaborate what does it mean the # and the number? Is that a
>> reference to the field in the rdd?
>> Thank you,
>> Ophir
>> On Mar 17, 2015 7:06 PM, "Yin Huai" <yh...@databricks.com> wrote:
>>
>>> Seems "basic_null_diluted_d" was not resolved? Can you check if
>>> basic_null_diluted_d is in you table?
>>>
>>> On Tue, Mar 17, 2015 at 9:34 AM, Ophir Cohen <op...@gmail.com> wrote:
>>>
>>>> Hi Guys,
>>>> I'm registering a function using:
>>>> sqlc.registerFunction("makeEstEntry",ReutersDataFunctions.makeEstEntry
>>>> _)
>>>>
>>>> Then I register the table and try to query the table using that
>>>> function and I get:
>>>> org.apache.spark.sql.catalyst.errors.package$TreeNodeException:
>>>> Unresolved attributes:
>>>> 'makeEstEntry(numest#20,median#21,mean#22,stddev#23,high#24,low#25,currency_#26,units#27,'basic_null_diluted_d)
>>>> AS FY0#2837, tree:
>>>>
>>>> Thanks!
>>>> Ophir
>>>>
>>>
>>>
>
Re: HiveContext can't find registered function
Posted by Yin Huai <yh...@databricks.com>.
The number is an id we used internally to identify an resolved Attribute.
Looks like basic_null_diluted_d was not resolved since there is no id
associated with it.
On Tue, Mar 17, 2015 at 2:08 PM, Ophir Cohen <op...@gmail.com> wrote:
> Interesting, I thought the problem is with the method itself.
> I will check it soon and update.
> Can you elaborate what does it mean the # and the number? Is that a
> reference to the field in the rdd?
> Thank you,
> Ophir
> On Mar 17, 2015 7:06 PM, "Yin Huai" <yh...@databricks.com> wrote:
>
>> Seems "basic_null_diluted_d" was not resolved? Can you check if
>> basic_null_diluted_d is in you table?
>>
>> On Tue, Mar 17, 2015 at 9:34 AM, Ophir Cohen <op...@gmail.com> wrote:
>>
>>> Hi Guys,
>>> I'm registering a function using:
>>> sqlc.registerFunction("makeEstEntry",ReutersDataFunctions.makeEstEntry _)
>>>
>>> Then I register the table and try to query the table using that function
>>> and I get:
>>> org.apache.spark.sql.catalyst.errors.package$TreeNodeException:
>>> Unresolved attributes:
>>> 'makeEstEntry(numest#20,median#21,mean#22,stddev#23,high#24,low#25,currency_#26,units#27,'basic_null_diluted_d)
>>> AS FY0#2837, tree:
>>>
>>> Thanks!
>>> Ophir
>>>
>>
>>
Re: HiveContext can't find registered function
Posted by Ophir Cohen <op...@gmail.com>.
Interesting, I thought the problem is with the method itself.
I will check it soon and update.
Can you elaborate what does it mean the # and the number? Is that a
reference to the field in the rdd?
Thank you,
Ophir
On Mar 17, 2015 7:06 PM, "Yin Huai" <yh...@databricks.com> wrote:
> Seems "basic_null_diluted_d" was not resolved? Can you check if
> basic_null_diluted_d is in you table?
>
> On Tue, Mar 17, 2015 at 9:34 AM, Ophir Cohen <op...@gmail.com> wrote:
>
>> Hi Guys,
>> I'm registering a function using:
>> sqlc.registerFunction("makeEstEntry",ReutersDataFunctions.makeEstEntry _)
>>
>> Then I register the table and try to query the table using that function
>> and I get:
>> org.apache.spark.sql.catalyst.errors.package$TreeNodeException:
>> Unresolved attributes:
>> 'makeEstEntry(numest#20,median#21,mean#22,stddev#23,high#24,low#25,currency_#26,units#27,'basic_null_diluted_d)
>> AS FY0#2837, tree:
>>
>> Thanks!
>> Ophir
>>
>
>
Re: HiveContext can't find registered function
Posted by Yin Huai <yh...@databricks.com>.
Seems "basic_null_diluted_d" was not resolved? Can you check if
basic_null_diluted_d is in you table?
On Tue, Mar 17, 2015 at 9:34 AM, Ophir Cohen <op...@gmail.com> wrote:
> Hi Guys,
> I'm registering a function using:
> sqlc.registerFunction("makeEstEntry",ReutersDataFunctions.makeEstEntry _)
>
> Then I register the table and try to query the table using that function
> and I get:
> org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved
> attributes:
> 'makeEstEntry(numest#20,median#21,mean#22,stddev#23,high#24,low#25,currency_#26,units#27,'basic_null_diluted_d)
> AS FY0#2837, tree:
>
> Thanks!
> Ophir
>