You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Du Li <li...@yahoo-inc.com.INVALID> on 2014/10/02 21:39:05 UTC
HiveContext: cache table not supported for partitioned table?
Hi,
In Spark 1.1 HiveContext, I ran a create partitioned table command followed by a cache table command and got a java.sql.SQLSyntaxErrorException: Table/View 'PARTITIONS' does not exist. But cache table worked fine if the table is not a partitioned table.
Can anybody confirm that cache of partitioned table is not supported yet in current version?
Thanks,
Du
Re: HiveContext: cache table not supported for partitioned table?
Posted by Du Li <li...@yahoo-inc.com.INVALID>.
Thanks for your explanation.
From: Cheng Lian <li...@gmail.com>>
Date: Thursday, October 2, 2014 at 8:01 PM
To: Du Li <li...@yahoo-inc.com.INVALID>>, "dev@spark.apache.org<ma...@spark.apache.org>" <de...@spark.apache.org>>
Cc: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: Re: HiveContext: cache table not supported for partitioned table?
Cache table works with partitioned table.
I guess you’re experimenting with a default local metastore and the metastore_db directory doesn’t exist at the first place. In this case, all metastore tables/views don’t exist at first and will throw the error message you saw when the PARTITIONS metastore table is accessed for the first time by Hive client. However, you should also see this line before this error:
14/10/03 10:51:30 ERROR ObjectStore: Direct SQL failed, falling back to ORM
And then the table is created on the fly. The cache operation is also performed normally. You can verify this by selecting it and check the Spark UI for cached RDDs. If you try to uncache the table and cache it again, you won’t see this error any more.
Normally, in production environment you won’t see this error because metastore database is usually setup ahead of time.
On 10/3/14 3:39 AM, Du Li wrote:
Hi,
In Spark 1.1 HiveContext, I ran a create partitioned table command followed by a cache table command and got a java.sql.SQLSyntaxErrorException: Table/View 'PARTITIONS' does not exist. But cache table worked fine if the table is not a partitioned table.
Can anybody confirm that cache of partitioned table is not supported yet in current version?
Thanks,
Du
Re: HiveContext: cache table not supported for partitioned table?
Posted by Du Li <li...@yahoo-inc.com.INVALID>.
Thanks for your explanation.
From: Cheng Lian <li...@gmail.com>>
Date: Thursday, October 2, 2014 at 8:01 PM
To: Du Li <li...@yahoo-inc.com.INVALID>>, "dev@spark.apache.org<ma...@spark.apache.org>" <de...@spark.apache.org>>
Cc: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: Re: HiveContext: cache table not supported for partitioned table?
Cache table works with partitioned table.
I guess you’re experimenting with a default local metastore and the metastore_db directory doesn’t exist at the first place. In this case, all metastore tables/views don’t exist at first and will throw the error message you saw when the PARTITIONS metastore table is accessed for the first time by Hive client. However, you should also see this line before this error:
14/10/03 10:51:30 ERROR ObjectStore: Direct SQL failed, falling back to ORM
And then the table is created on the fly. The cache operation is also performed normally. You can verify this by selecting it and check the Spark UI for cached RDDs. If you try to uncache the table and cache it again, you won’t see this error any more.
Normally, in production environment you won’t see this error because metastore database is usually setup ahead of time.
On 10/3/14 3:39 AM, Du Li wrote:
Hi,
In Spark 1.1 HiveContext, I ran a create partitioned table command followed by a cache table command and got a java.sql.SQLSyntaxErrorException: Table/View 'PARTITIONS' does not exist. But cache table worked fine if the table is not a partitioned table.
Can anybody confirm that cache of partitioned table is not supported yet in current version?
Thanks,
Du
Re: HiveContext: cache table not supported for partitioned table?
Posted by Cheng Lian <li...@gmail.com>.
Cache table works with partitioned table.
I guess you’re experimenting with a default local metastore and the
metastore_db directory doesn’t exist at the first place. In this case,
all metastore tables/views don’t exist at first and will throw the error
message you saw when the |PARTITIONS| metastore table is accessed for
the first time by Hive client. However, you should also see this line
before this error:
14/10/03 10:51:30 ERROR ObjectStore: Direct SQL failed, falling back
to ORM
And then the table is created on the fly. The cache operation is also
performed normally. You can verify this by selecting it and check the
Spark UI for cached RDDs. If you try to uncache the table and cache it
again, you won’t see this error any more.
Normally, in production environment you won’t see this error because
metastore database is usually setup ahead of time.
On 10/3/14 3:39 AM, Du Li wrote:
> Hi,
>
> In Spark 1.1 HiveContext, I ran a create partitioned table command
> followed by a cache table command and got
> a java.sql.SQLSyntaxErrorException: Table/View 'PARTITIONS' does not
> exist. But cache table worked fine if the table is not a partitioned
> table.
>
> Can anybody confirm that cache of partitioned table is not supported
> yet in current version?
>
> Thanks,
> Du
Re: HiveContext: cache table not supported for partitioned table?
Posted by Cheng Lian <li...@gmail.com>.
Cache table works with partitioned table.
I guess you’re experimenting with a default local metastore and the
metastore_db directory doesn’t exist at the first place. In this case,
all metastore tables/views don’t exist at first and will throw the error
message you saw when the |PARTITIONS| metastore table is accessed for
the first time by Hive client. However, you should also see this line
before this error:
14/10/03 10:51:30 ERROR ObjectStore: Direct SQL failed, falling back
to ORM
And then the table is created on the fly. The cache operation is also
performed normally. You can verify this by selecting it and check the
Spark UI for cached RDDs. If you try to uncache the table and cache it
again, you won’t see this error any more.
Normally, in production environment you won’t see this error because
metastore database is usually setup ahead of time.
On 10/3/14 3:39 AM, Du Li wrote:
> Hi,
>
> In Spark 1.1 HiveContext, I ran a create partitioned table command
> followed by a cache table command and got
> a java.sql.SQLSyntaxErrorException: Table/View 'PARTITIONS' does not
> exist. But cache table worked fine if the table is not a partitioned
> table.
>
> Can anybody confirm that cache of partitioned table is not supported
> yet in current version?
>
> Thanks,
> Du