You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "BePPPower (via GitHub)" <gi...@apache.org> on 2023/04/03 02:20:29 UTC
[GitHub] [doris] BePPPower opened a new pull request, #18325: [Enhencement](Export) support export
BePPPower opened a new pull request, #18325:
URL: https://github.com/apache/doris/pull/18325
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1514772821
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1510405150
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1514273842
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #18325: [Enhencement](Export) support export
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on code in PR #18325:
URL: https://github.com/apache/doris/pull/18325#discussion_r1160587627
##########
fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java:
##########
@@ -179,6 +145,27 @@ protected void exec() {
}
}
+ private AutoCloseConnectContext buildConnectContext() {
+ ConnectContext connectContext = new ConnectContext();
+ SessionVariable sessionVariable = connectContext.getSessionVariable();
+ sessionVariable.internalSession = true;
+ sessionVariable.setMaxExecMemByte(StatisticConstants.STATISTICS_MAX_MEM_PER_QUERY_IN_BYTES);
+ sessionVariable.setEnableInsertStrict(true);
+ sessionVariable.parallelExecInstanceNum = StatisticConstants.STATISTIC_PARALLEL_EXEC_INSTANCE_NUM;
+ sessionVariable.enableProfile = false;
+ connectContext.setEnv(Env.getCurrentEnv());
+ connectContext.setDatabase(job.getTableName().getDb());
+ // TODO(ftw): 这里的权限可能要改一下
+ connectContext.setQualifiedUser(UserIdentity.ROOT.getQualifiedUser());
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #18325: [Enhencement](Export) support export
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #18325:
URL: https://github.com/apache/doris/pull/18325#discussion_r1165447270
##########
fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java:
##########
@@ -714,6 +500,52 @@ public synchronized boolean isFinalState() {
return this.state == ExportJob.JobState.CANCELLED || this.state == ExportJob.JobState.FINISHED;
}
+ public Status makeSnapshots() {
Review Comment:
```suggestion
private Status makeSnapshots() {
```
##########
fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java:
##########
@@ -86,6 +95,53 @@ private void writeUnlock() {
lock.writeLock().unlock();
}
+ @Override
+ public synchronized void start() {
+ super.start();
+ exportingExecutor.start();
+ }
+
+ @Override
+ protected void runAfterCatalogReady() {
+ List<ExportJob> pendingJobs = getExportJobs(JobState.PENDING);
+ List<ExportJob> newInQueueJobs = Lists.newArrayList();
+ for (ExportJob job : pendingJobs) {
+ if (handlePendingJobs(job)) {
+ newInQueueJobs.add(job);
+ }
+ }
+ LOG.debug("new IN_QUEUE export job num: {}", newInQueueJobs.size());
+ for (ExportJob job : newInQueueJobs) {
+ try {
+ MasterTask task = new ExportExportingTask(job);
+ if (exportingExecutor.submit(task)) {
+ LOG.info("success to submit IN_QUEUE export job. job: {}", job);
+ } else {
+ LOG.info("fail to submit IN_QUEUE job to executor. job: {}", job);
+
+ }
+ } catch (Exception e) {
+ LOG.warn("run export exporting job error", e);
Review Comment:
```suggestion
LOG.warn("run export exporting job {}.", job, e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1510405142
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18325: [Enhencement](Export) support export
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1508121210
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #18325: [Enhencement](Export) support export
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on code in PR #18325:
URL: https://github.com/apache/doris/pull/18325#discussion_r1165161420
##########
fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java:
##########
@@ -86,6 +96,44 @@ private void writeUnlock() {
lock.writeLock().unlock();
}
+ @Override
+ protected void runAfterCatalogReady() {
+ List<ExportJob> jobs = getExportJobs(JobState.PENDING);
+ // Because exportJob may be replayed from log
+ // we also need handle EXPORTING state exportJob.
+ jobs.addAll(getExportJobs(JobState.EXPORTING));
+ int runningJobNumLimit = Config.export_running_job_num_limit;
+ if (runningJobNumLimit > 0 && !jobs.isEmpty()) {
Review Comment:
Exporting state means this exportJob is exporting. If we change the job's state to `EXPORTING` here, the job may not be assigned to the thread in the executor pool. So I think the change of job's state should in ExportExportingTask.exec()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on pull request #18325: [Enhencement](Export) support export
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1508213996
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #18325:
URL: https://github.com/apache/doris/pull/18325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1510405769
run beut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1514204165
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1510420284
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.76 seconds
stream load tsv: 424 seconds loaded 74807831229 Bytes, about 168 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230416160001_clickbench_pr_129767.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1515787053
run p1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1514756390
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #18325: [Enhencement](Export) support export
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #18325:
URL: https://github.com/apache/doris/pull/18325#discussion_r1164813455
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -547,6 +546,7 @@ private Env(boolean isCheckpointCatalog) {
this.routineLoadManager = new RoutineLoadManager();
this.sqlBlockRuleMgr = new SqlBlockRuleMgr();
this.exportMgr = new ExportMgr();
+ this.exportMgr.start();
Review Comment:
Move this to `startMasterOnlyDaemonThreads()`
##########
fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java:
##########
@@ -67,7 +72,12 @@ public class ExportMgr {
private Map<Long, ExportJob> idToJob = Maps.newHashMap(); // exportJobId to exportJob
private Map<String, Long> labelToJobId = Maps.newHashMap();
+ private MasterTaskExecutor exportingExecutor;
+
public ExportMgr() {
+ int poolSize = Config.export_running_job_num_limit == 0 ? 5 : Config.export_running_job_num_limit;
+ exportingExecutor = new MasterTaskExecutor("export-exporting-job", poolSize, true);
+ exportingExecutor.start();
Review Comment:
Override the `start()` of `MasterDaemon` and move this `exportingExecutor.start()` to `start()`
##########
fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java:
##########
@@ -86,6 +96,44 @@ private void writeUnlock() {
lock.writeLock().unlock();
}
+ @Override
+ protected void runAfterCatalogReady() {
+ List<ExportJob> jobs = getExportJobs(JobState.PENDING);
+ // Because exportJob may be replayed from log
+ // we also need handle EXPORTING state exportJob.
+ jobs.addAll(getExportJobs(JobState.EXPORTING));
Review Comment:
I think we don't need to handle `exporting` state jobs here.
All jobs replayed from log should be reset to PENDING.
##########
fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java:
##########
@@ -86,6 +96,44 @@ private void writeUnlock() {
lock.writeLock().unlock();
}
+ @Override
+ protected void runAfterCatalogReady() {
+ List<ExportJob> jobs = getExportJobs(JobState.PENDING);
+ // Because exportJob may be replayed from log
+ // we also need handle EXPORTING state exportJob.
+ jobs.addAll(getExportJobs(JobState.EXPORTING));
+ int runningJobNumLimit = Config.export_running_job_num_limit;
+ if (runningJobNumLimit > 0 && !jobs.isEmpty()) {
Review Comment:
No need to calc the remaining job num.
The number of thread in executor pool is limit to 5, and the submitted job will be queued when all threads are taken.
So here, we can simply do:
1. change the job's state to EXPORTING, so that it won't be scheduled again.
2. submit the job to the executor.
##########
fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java:
##########
@@ -281,67 +199,78 @@ private void registerProfile() {
ProfileManager.getInstance().pushProfile(profile);
}
- private Status moveTmpFiles() {
- FsBroker broker = null;
- try {
- String localIP = FrontendOptions.getLocalHostAddress();
- broker = Env.getCurrentEnv().getBrokerMgr().getBroker(job.getBrokerDesc().getName(), localIP);
- } catch (AnalysisException e) {
- String failMsg = "get broker failed. export job: " + job.getId() + ". msg: " + e.getMessage();
- LOG.warn(failMsg);
- return new Status(TStatusCode.CANCELLED, failMsg);
+ private void handlePendingState() {
+ long dbId = job.getDbId();
+ Database db = Env.getCurrentInternalCatalog().getDbNullable(dbId);
+ if (db == null) {
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, "database does not exist");
+ return;
}
- TNetworkAddress address = new TNetworkAddress(broker.ip, broker.port);
- TPaloBrokerService.Client client = null;
- try {
- client = ClientPool.brokerPool.borrowObject(address);
- } catch (Exception e) {
- try {
- client = ClientPool.brokerPool.borrowObject(address);
- } catch (Exception e1) {
- String failMsg = "create connection to broker(" + address + ") failed";
- LOG.warn(failMsg);
- return new Status(TStatusCode.CANCELLED, failMsg);
- }
+
+ if (job.isReplayed()) {
+ // If the job is created from replay thread, all plan info will be lost.
+ // so the job has to be cancelled.
+ String failMsg = "FE restarted or Master changed during exporting. Job must be cancelled.";
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, failMsg);
+ return;
+ }
+
+ // make snapshots
+ Status snapshotStatus = makeSnapshots();
+ if (!snapshotStatus.ok()) {
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, snapshotStatus.getErrorMsg());
+ return;
+ }
+
+ if (job.updateState(ExportJob.JobState.EXPORTING)) {
Review Comment:
No need to persist `EXPORTING` state.
The state should already be changed before submitting to the executor thread pool
##########
fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java:
##########
@@ -281,67 +199,78 @@ private void registerProfile() {
ProfileManager.getInstance().pushProfile(profile);
}
- private Status moveTmpFiles() {
- FsBroker broker = null;
- try {
- String localIP = FrontendOptions.getLocalHostAddress();
- broker = Env.getCurrentEnv().getBrokerMgr().getBroker(job.getBrokerDesc().getName(), localIP);
- } catch (AnalysisException e) {
- String failMsg = "get broker failed. export job: " + job.getId() + ". msg: " + e.getMessage();
- LOG.warn(failMsg);
- return new Status(TStatusCode.CANCELLED, failMsg);
+ private void handlePendingState() {
+ long dbId = job.getDbId();
+ Database db = Env.getCurrentInternalCatalog().getDbNullable(dbId);
+ if (db == null) {
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, "database does not exist");
+ return;
}
- TNetworkAddress address = new TNetworkAddress(broker.ip, broker.port);
- TPaloBrokerService.Client client = null;
- try {
- client = ClientPool.brokerPool.borrowObject(address);
- } catch (Exception e) {
- try {
- client = ClientPool.brokerPool.borrowObject(address);
- } catch (Exception e1) {
- String failMsg = "create connection to broker(" + address + ") failed";
- LOG.warn(failMsg);
- return new Status(TStatusCode.CANCELLED, failMsg);
- }
+
+ if (job.isReplayed()) {
+ // If the job is created from replay thread, all plan info will be lost.
+ // so the job has to be cancelled.
+ String failMsg = "FE restarted or Master changed during exporting. Job must be cancelled.";
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, failMsg);
+ return;
+ }
+
+ // make snapshots
+ Status snapshotStatus = makeSnapshots();
+ if (!snapshotStatus.ok()) {
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, snapshotStatus.getErrorMsg());
+ return;
+ }
+
+ if (job.updateState(ExportJob.JobState.EXPORTING)) {
+ LOG.info("Exchange pending status to exporting status success. job: {}", job);
+ return;
}
- boolean failed = false;
- Set<String> exportedFiles = job.getExportedFiles();
- List<String> newFiles = Lists.newArrayList();
- String exportPath = job.getExportPath();
- for (String exportedFile : exportedFiles) {
- // move exportPath/__doris_tmp/file to exportPath/file
- String file = exportedFile.substring(exportedFile.lastIndexOf("/") + 1);
- String destPath = exportPath + "/" + file;
- LOG.debug("rename {} to {}, export job: {}", exportedFile, destPath, job.getId());
- String failMsg = "";
- try {
- TBrokerRenamePathRequest request = new TBrokerRenamePathRequest(
- TBrokerVersion.VERSION_ONE, exportedFile, destPath, job.getBrokerDesc().getProperties());
- TBrokerOperationStatus tBrokerOperationStatus = null;
- tBrokerOperationStatus = client.renamePath(request);
- if (tBrokerOperationStatus.getStatusCode() != TBrokerOperationStatusCode.OK) {
- failed = true;
- failMsg = "Broker renamePath failed. srcPath=" + exportedFile + ", destPath=" + destPath
- + ", broker=" + address + ", msg=" + tBrokerOperationStatus.getMessage();
- return new Status(TStatusCode.CANCELLED, failMsg);
- } else {
- newFiles.add(destPath);
+ }
+
+ private Status makeSnapshots() {
Review Comment:
Currently, we can comment out this `makeSnapshots()` method because it is not used.
It can be reopened after we implement exporting tablet one by one
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java:
##########
@@ -308,13 +340,17 @@ private void checkProperties(Map<String, String> properties) throws UserExceptio
properties.put(TABLET_NUMBER_PER_TASK_PROP, String.valueOf(Config.export_tablet_num_per_task));
}
+ // max_file_size
+ this.maxFileSize = properties.getOrDefault(OutFileClause.PROP_MAX_FILE_SIZE, "");
+
if (properties.containsKey(LABEL)) {
FeNameFormat.checkLabel(properties.get(LABEL));
} else {
// generate a random label
- String label = "export_" + UUID.randomUUID().toString();
+ String label = "export_" + UUID.randomUUID();
properties.put(LABEL, label);
}
+ lable = properties.get(LABEL);
Review Comment:
```suggestion
label = properties.get(LABEL);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1514212641
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1514255580
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1515641193
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1508527160
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #18325: [Enhencement](Export) support export
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on code in PR #18325:
URL: https://github.com/apache/doris/pull/18325#discussion_r1165441995
##########
fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java:
##########
@@ -281,67 +199,78 @@ private void registerProfile() {
ProfileManager.getInstance().pushProfile(profile);
}
- private Status moveTmpFiles() {
- FsBroker broker = null;
- try {
- String localIP = FrontendOptions.getLocalHostAddress();
- broker = Env.getCurrentEnv().getBrokerMgr().getBroker(job.getBrokerDesc().getName(), localIP);
- } catch (AnalysisException e) {
- String failMsg = "get broker failed. export job: " + job.getId() + ". msg: " + e.getMessage();
- LOG.warn(failMsg);
- return new Status(TStatusCode.CANCELLED, failMsg);
+ private void handlePendingState() {
+ long dbId = job.getDbId();
+ Database db = Env.getCurrentInternalCatalog().getDbNullable(dbId);
+ if (db == null) {
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, "database does not exist");
+ return;
}
- TNetworkAddress address = new TNetworkAddress(broker.ip, broker.port);
- TPaloBrokerService.Client client = null;
- try {
- client = ClientPool.brokerPool.borrowObject(address);
- } catch (Exception e) {
- try {
- client = ClientPool.brokerPool.borrowObject(address);
- } catch (Exception e1) {
- String failMsg = "create connection to broker(" + address + ") failed";
- LOG.warn(failMsg);
- return new Status(TStatusCode.CANCELLED, failMsg);
- }
+
+ if (job.isReplayed()) {
+ // If the job is created from replay thread, all plan info will be lost.
+ // so the job has to be cancelled.
+ String failMsg = "FE restarted or Master changed during exporting. Job must be cancelled.";
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, failMsg);
+ return;
+ }
+
+ // make snapshots
+ Status snapshotStatus = makeSnapshots();
+ if (!snapshotStatus.ok()) {
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, snapshotStatus.getErrorMsg());
+ return;
+ }
+
+ if (job.updateState(ExportJob.JobState.EXPORTING)) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1510405446
run clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #18325: [Enhencement](Export) support export
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #18325:
URL: https://github.com/apache/doris/pull/18325#discussion_r1159920065
##########
fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java:
##########
@@ -179,6 +145,27 @@ protected void exec() {
}
}
+ private AutoCloseConnectContext buildConnectContext() {
+ ConnectContext connectContext = new ConnectContext();
+ SessionVariable sessionVariable = connectContext.getSessionVariable();
+ sessionVariable.internalSession = true;
+ sessionVariable.setMaxExecMemByte(StatisticConstants.STATISTICS_MAX_MEM_PER_QUERY_IN_BYTES);
+ sessionVariable.setEnableInsertStrict(true);
+ sessionVariable.parallelExecInstanceNum = StatisticConstants.STATISTIC_PARALLEL_EXEC_INSTANCE_NUM;
+ sessionVariable.enableProfile = false;
+ connectContext.setEnv(Env.getCurrentEnv());
+ connectContext.setDatabase(job.getTableName().getDb());
+ // TODO(ftw): 这里的权限可能要改一下
+ connectContext.setQualifiedUser(UserIdentity.ROOT.getQualifiedUser());
Review Comment:
Need to save user info in export job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18325: [Enhencement](Export) support export
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1508217077
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #18325: [Enhencement](Export) support export
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on code in PR #18325:
URL: https://github.com/apache/doris/pull/18325#discussion_r1165096930
##########
fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java:
##########
@@ -281,67 +199,78 @@ private void registerProfile() {
ProfileManager.getInstance().pushProfile(profile);
}
- private Status moveTmpFiles() {
- FsBroker broker = null;
- try {
- String localIP = FrontendOptions.getLocalHostAddress();
- broker = Env.getCurrentEnv().getBrokerMgr().getBroker(job.getBrokerDesc().getName(), localIP);
- } catch (AnalysisException e) {
- String failMsg = "get broker failed. export job: " + job.getId() + ". msg: " + e.getMessage();
- LOG.warn(failMsg);
- return new Status(TStatusCode.CANCELLED, failMsg);
+ private void handlePendingState() {
+ long dbId = job.getDbId();
+ Database db = Env.getCurrentInternalCatalog().getDbNullable(dbId);
+ if (db == null) {
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, "database does not exist");
+ return;
}
- TNetworkAddress address = new TNetworkAddress(broker.ip, broker.port);
- TPaloBrokerService.Client client = null;
- try {
- client = ClientPool.brokerPool.borrowObject(address);
- } catch (Exception e) {
- try {
- client = ClientPool.brokerPool.borrowObject(address);
- } catch (Exception e1) {
- String failMsg = "create connection to broker(" + address + ") failed";
- LOG.warn(failMsg);
- return new Status(TStatusCode.CANCELLED, failMsg);
- }
+
+ if (job.isReplayed()) {
+ // If the job is created from replay thread, all plan info will be lost.
+ // so the job has to be cancelled.
+ String failMsg = "FE restarted or Master changed during exporting. Job must be cancelled.";
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, failMsg);
+ return;
+ }
+
+ // make snapshots
+ Status snapshotStatus = makeSnapshots();
+ if (!snapshotStatus.ok()) {
+ job.cancel(ExportFailMsg.CancelType.RUN_FAIL, snapshotStatus.getErrorMsg());
+ return;
+ }
+
+ if (job.updateState(ExportJob.JobState.EXPORTING)) {
+ LOG.info("Exchange pending status to exporting status success. job: {}", job);
+ return;
}
- boolean failed = false;
- Set<String> exportedFiles = job.getExportedFiles();
- List<String> newFiles = Lists.newArrayList();
- String exportPath = job.getExportPath();
- for (String exportedFile : exportedFiles) {
- // move exportPath/__doris_tmp/file to exportPath/file
- String file = exportedFile.substring(exportedFile.lastIndexOf("/") + 1);
- String destPath = exportPath + "/" + file;
- LOG.debug("rename {} to {}, export job: {}", exportedFile, destPath, job.getId());
- String failMsg = "";
- try {
- TBrokerRenamePathRequest request = new TBrokerRenamePathRequest(
- TBrokerVersion.VERSION_ONE, exportedFile, destPath, job.getBrokerDesc().getProperties());
- TBrokerOperationStatus tBrokerOperationStatus = null;
- tBrokerOperationStatus = client.renamePath(request);
- if (tBrokerOperationStatus.getStatusCode() != TBrokerOperationStatusCode.OK) {
- failed = true;
- failMsg = "Broker renamePath failed. srcPath=" + exportedFile + ", destPath=" + destPath
- + ", broker=" + address + ", msg=" + tBrokerOperationStatus.getMessage();
- return new Status(TStatusCode.CANCELLED, failMsg);
- } else {
- newFiles.add(destPath);
+ }
+
+ private Status makeSnapshots() {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1514229930
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BePPPower commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "BePPPower (via GitHub)" <gi...@apache.org>.
BePPPower commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1514416798
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #18325: [Enhencement](Export) support export with outfile syntax
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #18325:
URL: https://github.com/apache/doris/pull/18325#issuecomment-1510421168
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org