You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2016/08/01 21:13:18 UTC
Hive transactional table with delta files, Spark cannot read and
sends error
Hi,
This is an ORC transactional table
Hive 2, Spark 1.6.1
hive> show create table payees;
OK
CREATE TABLE `payees`(
`transactiondescription` string,
`hits` int,
`hashtag` string)
CLUSTERED BY (
transactiondescription)
INTO 256 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'hdfs://rhes564:9000/user/hive/warehouse/accounts.db/payees'
TBLPROPERTIES (
'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
'numFiles'='887',
'numRows'='620',
'orc.compress'='ZLIB',
'rawDataSize'='0',
'totalSize'='844236',
'transactional'='true',
'transient_lastDdlTime'='1470085025')
With a number of delta files due to updates
drwxr-xr-x - hduser supergroup 0 2016-07-31 17:50
/user/hive/warehouse/accounts.db/payees/.hive-staging_hive_2016-07-31_17-50-32_170_8812909383478460781-1
drwxr-xr-x - hduser supergroup 0 2016-07-31 17:52
/user/hive/warehouse/accounts.db/payees/.hive-staging_hive_2016-07-31_17-52-07_293_1488464319826411209-1
drwxr-xr-x - hduser supergroup 0 2016-07-31 17:53
/user/hive/warehouse/accounts.db/payees/.hive-staging_hive_2016-07-31_17-53-50_895_2719298331883820815-1
drwxr-xr-x - hduser supergroup 0 2016-07-31 17:55
/user/hive/warehouse/accounts.db/payees/.hive-staging_hive_2016-07-31_17-55-05_534_3195777747078132526-1
drwxr-xr-x - hduser supergroup 0 2016-07-31 20:26
/user/hive/warehouse/accounts.db/payees/.hive-staging_hive_2016-07-31_20-25-47_734_1633377491601926952-1
drwxr-xr-x - hduser supergroup 0 2016-07-29 21:01
/user/hive/warehouse/accounts.db/payees/delta_0000099_0000099_0000
drwxr-xr-x - hduser supergroup 0 2016-07-29 21:02
/user/hive/warehouse/accounts.db/payees/delta_0000100_0000100_0000
drwxr-xr-x - hduser supergroup 0 2016-07-29 21:20
/user/hive/warehouse/accounts.db/payees/delta_0000101_0000101_0000
Spark fails reading this table. What options do I have here?
And interestingly Hive running on Spark engine and its works fine
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: Hive transactional table with delta files, Spark cannot read and
sends error
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> I am on Spark 1.6.1 and getting the following error
Ah, I realize that it's yet to be released officially.
Here's the demo from HadoopSummit -
<https://www.youtube.com/watch?v=wNYlHwsjhWU#t=26m>
I doubt this will ever be available for older spark releases but will be a
datasource package like spark-redshift.
Cheers,
Gopal
Re: Hive transactional table with delta files, Spark cannot read and
sends error
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Gopal.
I am on Spark 1.6.1 and getting the following error
scala> var conn = LlapContext.newInstance(sc, hs2_url);
<console>:28: error: not found: value LlapContext
var conn = LlapContext.newInstance(sc, hs2_url);
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 1 August 2016 at 22:53, Gopal Vijayaraghavan <go...@apache.org> wrote:
>
>
> > Spark fails reading this table. What options do I have here?
>
> Would your issue be the same as
> https://issues.apache.org/jira/browse/SPARK-13129?
>
>
> LLAPContext in Spark can read those tables with ACID semantics (as in
> delete/updates will work right).
>
> var conn = LlapContext.newInstance(sc, hs2_url);
> var df: DataFrame = conn.sql("select * from payees").persist();
>
> Please be aware that's entirely in auto-commit mode, so you will be
> getting lazy snapshot isolation (hence, persist is a good idea).
>
> Even though "payees" is a placeholder, but this approach is intended for
> tables like that which have multiple consumers, the practical reason to
> use this pathway would be to apply specific masking/filtering by accessing
> user (like hide amounts or just fit amounts into ranges, like 0-99, 99-999
> etc instead of actual values for compliance audits without creating
> complete copies).
>
> Cheers,
> Gopal
>
>
>
Re: Hive transactional table with delta files, Spark cannot read and
sends error
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> Spark fails reading this table. What options do I have here?
Would your issue be the same as
https://issues.apache.org/jira/browse/SPARK-13129?
LLAPContext in Spark can read those tables with ACID semantics (as in
delete/updates will work right).
var conn = LlapContext.newInstance(sc, hs2_url);
var df: DataFrame = conn.sql("select * from payees").persist();
Please be aware that's entirely in auto-commit mode, so you will be
getting lazy snapshot isolation (hence, persist is a good idea).
Even though "payees" is a placeholder, but this approach is intended for
tables like that which have multiple consumers, the practical reason to
use this pathway would be to apply specific masking/filtering by accessing
user (like hide amounts or just fit amounts into ranges, like 0-99, 99-999
etc instead of actual values for compliance audits without creating
complete copies).
Cheers,
Gopal