You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/12 14:06:41 UTC
[GitHub] [hudi] jiangbiao910 opened a new issue, #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
jiangbiao910 opened a new issue, #6089:
URL: https://github.com/apache/hudi/issues/6089
Hudi has recently released the latest version of Hudi, 0.11.1, which is pulled from Github and modified as follows for our Hadoop environment CDH6.3.2。
Upgrading to 0.11.1 resulting use sparksql：
`
create table if not exists zone_test.hudi_spark_table_0707
(
id string,
brand_id int,
brand_name string comment '品牌名称',
model_id int,
model_name string,
etl_update_time string,
dt string,
hh string
) using hudi
options (
type = 'cow',
primaryKey = 'brand_id,model_id',
preCombineField = 'etl_update_time'
)
partitioned by (dt,hh)
;
insert into zone_test.hudi_spark_table_0707 partition (dt, hh)
select id,
brand_id,
brand_name,
model_id,
model_name,
CAST(current_timestamp AS string) as etl_update_time,
'20220707' as dt,
'13' as hh
from zone_test.test_status_2_hi
;`
HS2 does not start after **hudi-hadoop-mr-bundle-0.11.1.jar** package is put into hive/lib directory and restart Hive, the log :
After the hbase-site. xml and hbase-default.xml file in the **hudi-hadoop-mr-bundle-0.11.1.jar** package is deleted , HS2 starts normally, hudi -spark-sql tasks are normal, but data query is abnormal.
what can I do next step? Thank you for your help。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6089:
URL: https://github.com/apache/hudi/issues/6089#issuecomment-1229348401
@jiangbiao910 : can you create a new github issue please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JxKim commented on issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
Posted by GitBox <gi...@apache.org>.
JxKim commented on issue #6089:
URL: https://github.com/apache/hudi/issues/6089#issuecomment-1182700767
1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangbiao910 commented on issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
Posted by GitBox <gi...@apache.org>.
jiangbiao910 commented on issue #6089:
URL: https://github.com/apache/hudi/issues/6089#issuecomment-1182668195
Thank you for reply，
I try modify the **\hudi\hudi-common\src\main\resources\hbase-site.xml** ，setting **hbase.defaults.for.version.skip** to **true** ，but it does‘t work well， HS2 does not start ，
Log Error as：
Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be for an older version of HBase (2.4.9), this version is 2.1.0-cdh6.3.2。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
URL: https://github.com/apache/hudi/issues/6089
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #6089:
URL: https://github.com/apache/hudi/issues/6089#issuecomment-1182553873
It looks like there is a conflict between the HBase hbase-default.xml from `hudi-hadoop-mr-bundle-0.11.1.jar` (HBase 2.4.9) and the CDH Hive HBase library (HBase 2.1.1-cdh6.3.2). Could you try setting `hbase.defaults.for.version.skip` to `true` in hbase-site.xml or hbase-defaults.xml?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #6089:
URL: https://github.com/apache/hudi/issues/6089#issuecomment-1184974827
> I try modify the \hudi\hudi-common\src\main\resources\hbase-site.xml ，setting hbase.defaults.for.version.skip to true
It looks work well。
@jiangbiao910 Sounds good. I'll put up a fix to change the default value.
> but I found that the MOR partition table cannot be queried directly through Hive unless you manually add partitions .
the MOR no partition table and COW(partition or no partition） can be queried directly through Hive .
is this a bug？ 0.11.0 and 0.11.1 the same result .
Let's follow up on this in a separate Github issue for traceability. Could you file a new Github issue on the Hive issue and provide reproducible steps? It could be a bug.
Feel free to close this one regarding the HBase issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #6089:
URL: https://github.com/apache/hudi/issues/6089#issuecomment-1412304963
The config change has landed in #6114. Please create a separate issue for partition syncing in Hive. You could also use `msck repair table <table_name>` as a workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6089:
URL: https://github.com/apache/hudi/issues/6089#issuecomment-1306130195
@yihua : is there any more pending work to be done here. I see we already have a FAQ on this https://hudi.apache.org/docs/faq#how-can-i-resolve-the-runtimeexception-saying-hbase-defaultxml-file-seems-to-be-for-an-older-version-of-hbase
can you follow up on this when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jiangbiao910 commented on issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
Posted by GitBox <gi...@apache.org>.
jiangbiao910 commented on issue #6089:
URL: https://github.com/apache/hudi/issues/6089#issuecomment-1182719121
> It looks like there is a conflict between the HBase hbase-default.xml from `hudi-hadoop-mr-bundle-0.11.1.jar` (HBase 2.4.9) and the CDH Hive HBase library (HBase 2.1.1-cdh6.3.2). Could you try setting `hbase.defaults.for.version.skip` to `true` in hbase-site.xml or hbase-defaults.xml?
Thank you for reply，
I try modify the \hudi\hudi-common\src\main\resources\hbase-site.xml ，setting **hbase.defaults.for.version.skip** to **true**
It looks work well。
but I found that the **MOR partition table** cannot be queried directly through Hive **unless you manually add partitions** .
the **MOR no partition table and COW(partition or no partition） can be queried directly through Hive .**
is this a bug？ 0.11.0 and 0.11.1 the same result .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #6089: [SUPPORT] Upgrading to 0.11.1 resulting use sparksql
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #6089:
URL: https://github.com/apache/hudi/issues/6089#issuecomment-1296302200
> I'll put up a fix to change the default value.
@yihua have you made the patch already? let's link the JIRA or PR here.
> but I found that the MOR partition table cannot be queried directly through Hive unless you manually add partitions .
> @jiangbiao910 : can you create a new github issue please.
@jiangbiao910 would you mind sharing some reproducible steps please? would like to look deeper and verify if it's bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org