You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/11/23 14:54:20 UTC
[GitHub] [incubator-doris] doudianer opened a new issue #7209: [Bug]
doudianer opened a new issue #7209:
URL: https://github.com/apache/incubator-doris/issues/7209
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Version
Doris：0.14.13
### What's Wrong?
The amount of data written by spark is 1000000000, occupying 10GB of storage. The specified number of partitions is 20, and an error is found when writing. The specified number of partitions is 5, and it is found that the write speed is 10000 pieces / min.
Spark写入数据量在10亿，占用存储10GB。指定分区数为20，发现写入报错。指定分区数为5，发现写入速度在10000 条/分钟
The code is as follows：
代码如下：
`dataFrame.write.format("doris")
.option("doris.table.identifier", "test.table_test")
.option("doris.fenodes", "IP:PORT")
.option("user", "user")
.option("password", "password")`
The error information is as follows：
报错信息如下：
<img width="1390" alt="infoflow 2021-11-23 21-36-11" src="https://user-images.githubusercontent.com/33255987/143044501-43553cb3-3047-4a25-a87c-179ab05af81c.png">
### What You Expected?
The write speed is close to the broker load
写入速度接近broker load
### How to Reproduce?
无
### Anything Else?
无
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on issue #7209: [Bug] Spark Doris Connector：add batch req success but status isn't ok
Posted by GitBox <gi...@apache.org>.
hf200012 commented on issue #7209:
URL: https://github.com/apache/incubator-doris/issues/7209#issuecomment-979836160
Your import frequency is too high. OLAP data is generally received in batches. You can try to read and import 10,000 or more in each batch. If you are migrating data from legacy systems such as spark and Hive, such as your 20G For data, you can try spark load or broker load
Your import frequency is too high. OLAP data is generally received in batches. You can try to read and import 10,000 or more in each batch. If you are migrating data from legacy systems such as spark and Hive, such as your 20G For data, you can try spark load or broker load
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #7209: [Bug]
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #7209:
URL: https://github.com/apache/incubator-doris/issues/7209#issuecomment-976695694
-235 means load frequency is too high. But Spark-connector does not have a config to modify the interval of loading batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] doudianer commented on issue #7209: [Bug] Spark Doris Connector：add batch req success but status isn't ok
Posted by GitBox <gi...@apache.org>.
doudianer commented on issue #7209:
URL: https://github.com/apache/incubator-doris/issues/7209#issuecomment-981052992
Thank you. I found doris.batch.size: batch size for reading and writing. The size I set is 5000000, which is very good in my business scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] doudianer commented on issue #7209: [Bug] Spark Doris Connector：add batch req success but status isn't ok
Posted by GitBox <gi...@apache.org>.
doudianer commented on issue #7209:
URL: https://github.com/apache/incubator-doris/issues/7209#issuecomment-977473219
Spark Doris connector does not open this configuration. Do I need to adjust it based on the source code？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] doudianer closed issue #7209: [Bug] Spark Doris Connector：add batch req success but status isn't ok
Posted by GitBox <gi...@apache.org>.
doudianer closed issue #7209:
URL: https://github.com/apache/incubator-doris/issues/7209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org