You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/12/25 12:37:29 UTC
[GitHub] [incubator-doris] morningman commented on a change in pull request #7485: Modify the maximum number of records written by Spark/Flink connector each time. The default value is 1000.
morningman commented on a change in pull request #7485:
URL: https://github.com/apache/incubator-doris/pull/7485#discussion_r775137129
##########
File path: extension/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisExecutionOptions.java
##########
@@ -28,7 +28,7 @@
public class DorisExecutionOptions implements Serializable {
private static final long serialVersionUID = 1L;
- public static final Integer DEFAULT_BATCH_SIZE = 1000;
+ public static final Integer DEFAULT_BATCH_SIZE = 10000;
public static final Integer DEFAULT_MAX_RETRY_TIMES = 3;
Review comment:
I think the `DEFAULT_MAX_RETRY_TIMES` can be set to 1 to avoid too many unnecessary retries.
If `DEFAULT_MAX_RETRY_TIMES` is 1, it will only retry once.
##########
File path: extension/spark-doris-connector/src/main/java/org/apache/doris/spark/cfg/ConfigurationOptions.java
##########
@@ -66,11 +66,9 @@
String DORIS_WRITE_FIELDS = "doris.write.fields";
- String SINK_BATCH_SIZE = "sink.batch.size";
String DORIS_SINK_BATCH_SIZE = "doris.sink.batch.size";
- int SINK_BATCH_SIZE_DEFAULT = 1024;
+ int SINK_BATCH_SIZE_DEFAULT = 10000;
- String SINK_MAX_RETRIES = "sink.max-retries";
String DORIS_SINK_MAX_RETRIES = "doris.sink.max-retries";
int SINK_MAX_RETRIES_DEFAULT = 3;
Review comment:
Set `SINK_MAX_RETRIES_DEFAULT` to 1, same as flink connector.
And we also need to modify `DorisStreamLoadSink.scala`, because when `SINK_MAX_RETRIES_DEFAULT` is 1, there is no retry at all. We should keep it consistent with flink connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org