You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/04/21 09:56:33 UTC
[GitHub] [incubator-doris] Jibing-Li opened a new pull request, #9151: Support hive with HA HDFS. Pass ha configuration through hive create table properties.
Jibing-Li opened a new pull request, #9151:
URL: https://github.com/apache/incubator-doris/pull/9151
Support hive with HA HDFS. Pass ha configuration through hive create table properties.
# Proposed changes
Issue Number: close #xxx
https://github.com/apache/incubator-doris/issues/9129
## Problem Summary:
Doris couldn't resolve the defaultFS of HDFS with HA configuration, so it could query hive table on HA HDFS. This is because there's no way to send the HA configs to hive external table.
Describe the overview of changes.
Pass the ha configs to hive external table through create table properties.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know) No
2. Has unit tests been added: (Yes/No/No Need) No Need
3. Has document been added or modified: (Yes/No/No Need) No
4. Does it need to update dependencies: (Yes/No) No
5. Are there any changes that cannot be rolled back: (Yes/No) No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #9151: Support hive with HA HDFS. Pass ha configuration through hive create table properties.
Posted by GitBox <gi...@apache.org>.
morningman merged PR #9151:
URL: https://github.com/apache/incubator-doris/pull/9151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] Jibing-Li commented on a diff in pull request #9151: Support hive with HA HDFS. Pass ha configuration through hive create table properties.
Posted by GitBox <gi...@apache.org>.
Jibing-Li commented on code in PR #9151:
URL: https://github.com/apache/incubator-doris/pull/9151#discussion_r859334673
##########
docs/en/extending-doris/hive-of-doris.md:
##########
@@ -83,6 +103,10 @@ PROPERTIES (
- `hive.metastore.uris`: Hive Metastore service address
- `database`: the name of the database to which Hive is mounted
- `table`: the name of the table to which Hive is mounted
+ - `dfs.nameservices`：the logical name for this new nameservice. See hdfs-site.xml
+ - `dfs.ha.namenodes.[nameservice ID]：unique identifiers for each NameNode in the nameservice. See hdfs-site.xml
Review Comment:
Good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a diff in pull request #9151: Support hive with HA HDFS. Pass ha configuration through hive create table properties.
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #9151:
URL: https://github.com/apache/incubator-doris/pull/9151#discussion_r858270515
##########
fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java:
##########
@@ -522,7 +528,11 @@ private void processFileGroup(
rangeDesc.setNumAsString(context.fileGroup.isNumAsString());
rangeDesc.setReadJsonByLine(context.fileGroup.isReadJsonByLine());
}
- rangeDesc.setHdfsParams(tHdfsParams);
+ if (rangeDesc.hdfs_params != null && rangeDesc.hdfs_params.getFsName() == null)
+ rangeDesc.hdfs_params.setFsName(fsName);
+ else if (rangeDesc.hdfs_params == null)
Review Comment:
Use {} with if else
##########
fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java:
##########
@@ -499,7 +501,11 @@ private void processFileGroup(
} else {
TBrokerRangeDesc rangeDesc = createBrokerRangeDesc(curFileOffset, fileStatus, formatType,
leftBytes, columnsFromPath, numberOfColumnsFromFile, brokerDesc, header_type);
- rangeDesc.setHdfsParams(tHdfsParams);
+ if (rangeDesc.hdfs_params != null && rangeDesc.hdfs_params.getFsName() == null)
+ rangeDesc.hdfs_params.setFsName(fsName);
+ else if (rangeDesc.hdfs_params == null)
Review Comment:
use {} with if else block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9151: Support hive with HA HDFS. Pass ha configuration through hive create table properties.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9151:
URL: https://github.com/apache/incubator-doris/pull/9151#issuecomment-1118719790
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] Jibing-Li commented on a diff in pull request #9151: Support hive with HA HDFS. Pass ha configuration through hive create table properties.
Posted by GitBox <gi...@apache.org>.
Jibing-Li commented on code in PR #9151:
URL: https://github.com/apache/incubator-doris/pull/9151#discussion_r859333798
##########
fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java:
##########
@@ -522,7 +529,12 @@ private void processFileGroup(
rangeDesc.setNumAsString(context.fileGroup.isNumAsString());
rangeDesc.setReadJsonByLine(context.fileGroup.isReadJsonByLine());
}
- rangeDesc.setHdfsParams(tHdfsParams);
+ if (rangeDesc.hdfs_params != null && rangeDesc.hdfs_params.getFsName() == null) {
Review Comment:
The origin code calls rangeDesc.setHdfsParams twice in the if and else block. I don't think we need to add a function for the duplicate code, because it is only used in these two places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] stalary commented on a diff in pull request #9151: Support hive with HA HDFS. Pass ha configuration through hive create table properties.
Posted by GitBox <gi...@apache.org>.
stalary commented on code in PR #9151:
URL: https://github.com/apache/incubator-doris/pull/9151#discussion_r858804841
##########
docs/en/extending-doris/hive-of-doris.md:
##########
@@ -83,6 +103,10 @@ PROPERTIES (
- `hive.metastore.uris`: Hive Metastore service address
- `database`: the name of the database to which Hive is mounted
- `table`: the name of the table to which Hive is mounted
+ - `dfs.nameservices`：the logical name for this new nameservice. See hdfs-site.xml
+ - `dfs.ha.namenodes.[nameservice ID]：unique identifiers for each NameNode in the nameservice. See hdfs-site.xml
Review Comment:
dfs.ha.namenodes.[nameservice ID] need end with `
##########
docs/en/extending-doris/hive-of-doris.md:
##########
@@ -83,6 +103,10 @@ PROPERTIES (
- `hive.metastore.uris`: Hive Metastore service address
- `database`: the name of the database to which Hive is mounted
- `table`: the name of the table to which Hive is mounted
+ - `dfs.nameservices`：the logical name for this new nameservice. See hdfs-site.xml
+ - `dfs.ha.namenodes.[nameservice ID]：unique identifiers for each NameNode in the nameservice. See hdfs-site.xml
+ - `dfs.namenode.rpc-address.[nameservice ID].[name node ID]`：the fully-qualified RPC address for each NameNode to listen on. See hdfs-site.xml
+ - `dfs.client.failover.proxy.provider.[nameservice ID] `：the Java class that HDFS clients use to contact the Active NameNode, usually it is org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
Review Comment:
[nameservice ID] ` remove blank
##########
fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java:
##########
@@ -522,7 +529,12 @@ private void processFileGroup(
rangeDesc.setNumAsString(context.fileGroup.isNumAsString());
rangeDesc.setReadJsonByLine(context.fileGroup.isReadJsonByLine());
}
- rangeDesc.setHdfsParams(tHdfsParams);
+ if (rangeDesc.hdfs_params != null && rangeDesc.hdfs_params.getFsName() == null) {
Review Comment:
duplicate with 504
##########
docs/zh-CN/extending-doris/hive-of-doris.md:
##########
@@ -83,6 +103,10 @@ PROPERTIES (
- `hive.metastore.uris`：Hive Metastore 服务地址
- `database`：挂载 Hive 对应的数据库名
- `table`：挂载 Hive 对应的表名
+ - `dfs.nameservices`：name service名称，与hdfs-site.xml保持一致
+ - `dfs.ha.namenodes.[nameservice ID]：namenode的id列表,与hdfs-site.xml保持一致
Review Comment:
Same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9151: Support hive with HA HDFS. Pass ha configuration through hive create table properties.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9151:
URL: https://github.com/apache/incubator-doris/pull/9151#issuecomment-1118719837
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org