You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Lars Francke <la...@gmail.com> on 2013/07/17 23:24:02 UTC
Problem with the windowing function ntile (Exceptions)
Hi,
I'm running a query like this:
CREATE TABLE foo
STORED AS ORC
AS
SELECT
id,
season,
amount,
ntile(10)
OVER (
PARTITION BY season
ORDER BY amount DESC
)
FROM bar;
On a small enough dataset that works fine but when switching to a
larger sample we're seeing exceptions like this:
"Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Reset on
PersistentByteBasedList not supported"
Looking at the code (without really understanding it) we tried setting:
SET hive.ptf.partition.persistence='org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList';
because that List supports reset but we are seeing a
ClassNotFoundException so we're doing that wrong.
Next try was setting hive.ptf.partition.persistence.memsize higher
which worked but first of all we don't really understand what all of
that stuff is doing and second of all we fear that it just might break
down again.
Any hints as to what that error really means and how to deal with it
would be greatly appreciated.
Thanks!
Lars
Re: Problem with the windowing function ntile (Exceptions)
Posted by Lars Francke <la...@gmail.com>.
Hi Navis,
that's awesome! Thank you for the quick response and the patch. I just
tested it against my query which used to fail and it just worked. So
you have at least my thumbs-up.
Thanks again for your help.
Cheers,
Lars
On Thu, Jul 25, 2013 at 10:01 AM, Navis류승우 <na...@nexr.com> wrote:
> I've booked this and attached patch for it.
>
> https://issues.apache.org/jira/browse/HIVE-4932
>
> Could you test with that? thanks.
>
> 2013/7/25 Lars Francke <la...@gmail.com>:
>> We're still being bitten by this problem without a workaround. Does
>> anyone have an idea?
>>
>> Thanks,
>> Lars
>>
>> On Wed, Jul 17, 2013 at 11:24 PM, Lars Francke <la...@gmail.com> wrote:
>>> Hi,
>>>
>>> I'm running a query like this:
>>>
>>> CREATE TABLE foo
>>> STORED AS ORC
>>> AS
>>> SELECT
>>> id,
>>> season,
>>> amount,
>>> ntile(10)
>>> OVER (
>>> PARTITION BY season
>>> ORDER BY amount DESC
>>> )
>>> FROM bar;
>>>
>>> On a small enough dataset that works fine but when switching to a
>>> larger sample we're seeing exceptions like this:
>>>
>>> "Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Reset on
>>> PersistentByteBasedList not supported"
>>>
>>> Looking at the code (without really understanding it) we tried setting:
>>> SET hive.ptf.partition.persistence='org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList';
>>>
>>> because that List supports reset but we are seeing a
>>> ClassNotFoundException so we're doing that wrong.
>>>
>>> Next try was setting hive.ptf.partition.persistence.memsize higher
>>> which worked but first of all we don't really understand what all of
>>> that stuff is doing and second of all we fear that it just might break
>>> down again.
>>>
>>> Any hints as to what that error really means and how to deal with it
>>> would be greatly appreciated.
>>>
>>> Thanks!
>>>
>>> Lars
Re: Problem with the windowing function ntile (Exceptions)
Posted by Navis류승우 <na...@nexr.com>.
I've booked this and attached patch for it.
https://issues.apache.org/jira/browse/HIVE-4932
Could you test with that? thanks.
2013/7/25 Lars Francke <la...@gmail.com>:
> We're still being bitten by this problem without a workaround. Does
> anyone have an idea?
>
> Thanks,
> Lars
>
> On Wed, Jul 17, 2013 at 11:24 PM, Lars Francke <la...@gmail.com> wrote:
>> Hi,
>>
>> I'm running a query like this:
>>
>> CREATE TABLE foo
>> STORED AS ORC
>> AS
>> SELECT
>> id,
>> season,
>> amount,
>> ntile(10)
>> OVER (
>> PARTITION BY season
>> ORDER BY amount DESC
>> )
>> FROM bar;
>>
>> On a small enough dataset that works fine but when switching to a
>> larger sample we're seeing exceptions like this:
>>
>> "Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Reset on
>> PersistentByteBasedList not supported"
>>
>> Looking at the code (without really understanding it) we tried setting:
>> SET hive.ptf.partition.persistence='org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList';
>>
>> because that List supports reset but we are seeing a
>> ClassNotFoundException so we're doing that wrong.
>>
>> Next try was setting hive.ptf.partition.persistence.memsize higher
>> which worked but first of all we don't really understand what all of
>> that stuff is doing and second of all we fear that it just might break
>> down again.
>>
>> Any hints as to what that error really means and how to deal with it
>> would be greatly appreciated.
>>
>> Thanks!
>>
>> Lars
Re: Problem with the windowing function ntile (Exceptions)
Posted by Lars Francke <la...@gmail.com>.
We're still being bitten by this problem without a workaround. Does
anyone have an idea?
Thanks,
Lars
On Wed, Jul 17, 2013 at 11:24 PM, Lars Francke <la...@gmail.com> wrote:
> Hi,
>
> I'm running a query like this:
>
> CREATE TABLE foo
> STORED AS ORC
> AS
> SELECT
> id,
> season,
> amount,
> ntile(10)
> OVER (
> PARTITION BY season
> ORDER BY amount DESC
> )
> FROM bar;
>
> On a small enough dataset that works fine but when switching to a
> larger sample we're seeing exceptions like this:
>
> "Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Reset on
> PersistentByteBasedList not supported"
>
> Looking at the code (without really understanding it) we tried setting:
> SET hive.ptf.partition.persistence='org.apache.hadoop.hive.ql.exec.PTFPersistence$PartitionedByteBasedList';
>
> because that List supports reset but we are seeing a
> ClassNotFoundException so we're doing that wrong.
>
> Next try was setting hive.ptf.partition.persistence.memsize higher
> which worked but first of all we don't really understand what all of
> that stuff is doing and second of all we fear that it just might break
> down again.
>
> Any hints as to what that error really means and how to deal with it
> would be greatly appreciated.
>
> Thanks!
>
> Lars