You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/07/08 09:01:30 UTC
[GitHub] [doris] icedrugs89 opened a new issue, #10696: [Bug] BrokerLoad导入任务出现type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = Broker list path exception. path=hdfs:xxx
icedrugs89 opened a new issue, #10696:
URL: https://github.com/apache/doris/issues/10696
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Version
1.0.0
### What's Wrong?
1、创建BrokerLoad任务，使用kerberos认证访问hdfs的hive文件导入数据
2、hive文件路径中分区和下一级目录使用通配符*，访问所有分区所有文件
3、任务提交后隔40多秒出现如下的错误
<img width="1438" alt="image" src="https://user-images.githubusercontent.com/100941547/177956278-7a2cfd76-8df0-409a-8ccc-caa5d1d561dd.png">
### What You Expected?
BrokerLoad任务提交后，show load 看到具体的任务状态为ETL:100%; LOAD:99%，以及JobDetails结果中的ScannedRows行数一直在增长
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei closed issue #10696: [Bug] BrokerLoad导入任务出现type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = Broker list path exception. path=hdfs:xxx
Posted by GitBox <gi...@apache.org>.
yiguolei closed issue #10696: [Bug] BrokerLoad导入任务出现type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = Broker list path exception. path=hdfs:xxx
URL: https://github.com/apache/doris/issues/10696
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] shixiutao commented on issue #10696: [Bug] BrokerLoad导入任务出现type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = Broker list path exception. path=hdfs:xxx
Posted by "shixiutao (via GitHub)" <gi...@apache.org>.
shixiutao commented on issue #10696:
URL: https://github.com/apache/doris/issues/10696#issuecomment-1607057582
It is recommended to send the complete command for **broker load** and the **hdfs** path. I have encountered similar issues before. At that time, I reported an error due to the inconsistent format extracted from the **partition** and **COLUMNS FROM PATH** of the **hive** table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] kpfly commented on issue #10696: [Bug] BrokerLoad导入任务出现type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = Broker list path exception. path=hdfs:xxx
Posted by GitBox <gi...@apache.org>.
kpfly commented on issue #10696:
URL: https://github.com/apache/doris/issues/10696#issuecomment-1274249010
Good Job，跨地域了么？
icedrugs89 ***@***.***> 于2022年7月8日周五 17:19写道：
> 1、指定具体的年月日分区是可以导入的，怀疑是表的分区目录过大，进行如下验证：
> 1)broker机器安装hdfs客户端环境，使用通配符*访问hive对应的所有分区文件，耗时大约在56秒
> 2)查看对应的thrift中的socket代码，执行获取File的status状态可能由于hive分区数太大耗时太久超时
>
> 3)查看thrift-0.9.3/lib/java/src/org/apache/thrift/transport/TSocket.java中的SocketTimeout默认读写超时时间太短小于50秒
> [image: image]
> <https://user-images.githubusercontent.com/100941547/177959022-6b10291d-4846-4b94-80b9-44c534da1b8a.png>
> 4)由社区的技术团队修改FE的参数fe/fe-core/src/main/java/org/apache/doris/common/ClientPool.java增加static
> int brokerTimeoutMs = 10000;
> 5)生成测试版本， 重启FE后再次验证，问题修复
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/doris/issues/10696#issuecomment-1178755340>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AXXX5T4F6WGIFFGSINISQVTVS7XAZANCNFSM53AIZ7QQ>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] icedrugs89 commented on issue #10696: [Bug] BrokerLoad导入任务出现type:ETL_RUN_FAIL; msg:errCode = 2, detailMessage = Broker list path exception. path=hdfs:xxx
Posted by GitBox <gi...@apache.org>.
icedrugs89 commented on issue #10696:
URL: https://github.com/apache/doris/issues/10696#issuecomment-1178755340
1、指定具体的年月日分区是可以导入的，怀疑是表的分区目录过大，进行如下验证：
1)broker机器安装hdfs客户端环境，使用通配符*访问hive对应的所有分区文件，耗时大约在56秒
2)查看对应的thrift中的socket代码，执行获取File的status状态可能由于hive分区数太大耗时太久超时
3)查看thrift-0.9.3/lib/java/src/org/apache/thrift/transport/TSocket.java中的SocketTimeout默认读写超时时间太短小于50秒
4)由社区的技术团队修改FE的参数fe/fe-core/src/main/java/org/apache/doris/common/ClientPool.java增加static int brokerTimeoutMs = 10000;
5)生成测试版本， 重启FE后再次验证，问题修复
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org