You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/05/27 12:14:18 UTC
[GitHub] [incubator-doris] EmmyMiao87 opened a new pull request #3705: Serialize origin stmt in Rollup Job and MV Meta
EmmyMiao87 opened a new pull request #3705:
URL: https://github.com/apache/incubator-doris/pull/3705
In materialized view 2.0, the define expr is serialized in column.
The method is that doris serialzie the origin stmt of Create Materialzied View Stmt in RollupJobV2 and MVMeta.
The define expr will be extract from the origin stmt after meta is deserialized.
The define expr is necessary for bitmap and hll materialized view.
For example:
MV meta: __doris_mv_bitmap_k1, bitmap_union, to_bitmap(k1)
Origin stmt: select bitmap_union(to_bitmap(k1)) from table
Deserialize meta: __doris_mv_bitmap_k1, bitmap_union, null
After extract: the define expr `to_bitmap(k1)` from origin stmt should be extracted.
__doris_mv_bitmap_v1, bitmap_union, to_bitmap(k1) (which comes from the origin stmt)
#3344
Change-Id: Ic2da093188d8985f5e97be5bd094e5d60d82c9a7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] HangyuanLiu merged pull request #3705: Serialize origin stmt in Rollup Job and MV Meta
Posted by GitBox <gi...@apache.org>.
HangyuanLiu merged pull request #3705:
URL: https://github.com/apache/incubator-doris/pull/3705
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #3705: Serialize origin stmt in Rollup Job and MV Meta
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on a change in pull request #3705:
URL: https://github.com/apache/incubator-doris/pull/3705#discussion_r431549428
##########
File path: fe/src/main/java/org/apache/doris/alter/RollupJobV2.java
##########
@@ -770,4 +714,85 @@ public void setJobState(JobState jobState) {
this.jobState = jobState;
}
+ private void setColumnsDefineExpr(List<MVColumnItem> items) {
+ for (MVColumnItem item : items) {
+ for (Column column : rollupSchema) {
+ if (column.getName().equals(item.getName())) {
+ column.setDefineExpr(item.getDefineExpr());
+ break;
+ }
+ }
+ }
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ String json = GsonUtils.GSON.toJson(this, AlterJobV2.class);
+ Text.writeString(out, json);
+ }
+
+ public static RollupJobV2 read(DataInput in) throws IOException {
Review comment:
This is also used by the Meta <85. If the Meta is more then 86, the `read` of AlterJobV2 wil be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] HangyuanLiu commented on a change in pull request #3705: Serialize origin stmt in Rollup Job and MV Meta
Posted by GitBox <gi...@apache.org>.
HangyuanLiu commented on a change in pull request #3705:
URL: https://github.com/apache/incubator-doris/pull/3705#discussion_r431551350
##########
File path: fe/src/main/java/org/apache/doris/analysis/StatementBase.java
##########
@@ -153,7 +156,17 @@ public String getClusterName() {
public void setClusterName(String clusterName) {
this.clusterName = clusterName;
- }
+ }
+
+ public void setOrigStmt(OriginStatement origStmt) {
+ this.origStmt = origStmt;
+ }
+
+ public OriginStatement getOrigStmt() {
+ Preconditions.checkState(origStmt != null);
Review comment:
NULL should be judged in set, it may be too late to judge in get function
##########
File path: fe/src/main/java/org/apache/doris/catalog/MaterializedIndexMeta.java
##########
@@ -46,9 +55,11 @@
private TStorageType storageType;
@SerializedName(value = "keysType")
private KeysType keysType;
+ @SerializedName(value = "defineStmt")
+ private OriginStatement defineStmt;
Review comment:
The name defineStmt can easily cause ambiguity with DefineExpr. But this statement may not only parse the semantics of defineExpr in the future。MaterializedViewStmt is a better name ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3705: Serialize origin stmt in Rollup Job and MV Meta
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3705:
URL: https://github.com/apache/incubator-doris/pull/3705#discussion_r431540575
##########
File path: fe/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
##########
@@ -1081,14 +969,8 @@ private void readJobFinishedData(DataInput in) throws IOException {
@Override
public void write(DataOutput out) throws IOException {
- super.write(out);
-
- out.writeBoolean(isMetaPruned);
- if (isMetaPruned) {
- writeJobFinishedData(out);
- } else {
- writeJobNotFinishData(out);
- }
+ String json = GsonUtils.GSON.toJson(this, AlterJobV2.class);
Review comment:
write don't match with readFields
##########
File path: fe/src/main/java/org/apache/doris/alter/RollupJobV2.java
##########
@@ -770,4 +714,85 @@ public void setJobState(JobState jobState) {
this.jobState = jobState;
}
+ private void setColumnsDefineExpr(List<MVColumnItem> items) {
+ for (MVColumnItem item : items) {
+ for (Column column : rollupSchema) {
+ if (column.getName().equals(item.getName())) {
+ column.setDefineExpr(item.getDefineExpr());
+ break;
+ }
+ }
+ }
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ String json = GsonUtils.GSON.toJson(this, AlterJobV2.class);
+ Text.writeString(out, json);
+ }
+
+ public static RollupJobV2 read(DataInput in) throws IOException {
Review comment:
This is not match with what `write` writes. We should make them consistent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 closed pull request #3705: Serialize origin stmt in Rollup Job and MV Meta
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 closed pull request #3705:
URL: https://github.com/apache/incubator-doris/pull/3705
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #3705: Serialize origin stmt in Rollup Job and MV Meta
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on a change in pull request #3705:
URL: https://github.com/apache/incubator-doris/pull/3705#discussion_r431651862
##########
File path: fe/src/main/java/org/apache/doris/catalog/MaterializedIndexMeta.java
##########
@@ -46,9 +55,11 @@
private TStorageType storageType;
@SerializedName(value = "keysType")
private KeysType keysType;
+ @SerializedName(value = "defineStmt")
+ private OriginStatement defineStmt;
Review comment:
The define stmt means that this is the definition of materialized views.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #3705: Serialize origin stmt in Rollup Job and MV Meta
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on a change in pull request #3705:
URL: https://github.com/apache/incubator-doris/pull/3705#discussion_r431549970
##########
File path: fe/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
##########
@@ -1081,14 +969,8 @@ private void readJobFinishedData(DataInput in) throws IOException {
@Override
public void write(DataOutput out) throws IOException {
- super.write(out);
-
- out.writeBoolean(isMetaPruned);
- if (isMetaPruned) {
- writeJobFinishedData(out);
- } else {
- writeJobNotFinishData(out);
- }
+ String json = GsonUtils.GSON.toJson(this, AlterJobV2.class);
Review comment:
I have conducted a compatibility test. The `read` of AlterJobV2 match the `write` of RollupJobV2 and SchemaChangeV2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3705: Serialize origin stmt in Rollup Job and MV Meta
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3705:
URL: https://github.com/apache/incubator-doris/pull/3705#discussion_r431628375
##########
File path: fe/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java
##########
@@ -100,9 +102,8 @@ public KeysType getMVKeysType() {
@Override
public void analyze(Analyzer analyzer) throws UserException {
- // TODO(ml): remove it
if (!Config.enable_materialized_view) {
- throw new AnalysisException("The materialized view is coming soon");
+ throw new AnalysisException("The materialized view is disable");
Review comment:
```suggestion
throw new AnalysisException("The materialized view is disabled");
```
##########
File path: fe/src/main/java/org/apache/doris/alter/AlterJobV2.java
##########
@@ -220,16 +233,20 @@ protected boolean checkTableStable(Database db) throws AlterCancelException {
public abstract void replay(AlterJobV2 replayedJob);
public static AlterJobV2 read(DataInput in) throws IOException {
- JobType type = JobType.valueOf(Text.readString(in));
- switch (type) {
- case ROLLUP:
- return RollupJobV2.read(in);
- case SCHEMA_CHANGE:
- return SchemaChangeJobV2.read(in);
- default:
- Preconditions.checkState(false);
- return null;
+ if (Catalog.getCurrentCatalogJournalVersion() < FeMetaVersion.VERSION_86) {
+ JobType type = JobType.valueOf(Text.readString(in));
+ switch (type) {
+ case ROLLUP:
+ return RollupJobV2.read(in);
+ case SCHEMA_CHANGE:
+ return SchemaChangeJobV2.read(in);
+ default:
+ Preconditions.checkState(false);
+ return null;
+ }
}
Review comment:
Add `else` to make logic more explict
##########
File path: fe/src/main/java/org/apache/doris/qe/DdlExecutor.java
##########
@@ -132,7 +131,7 @@ public static void execute(Catalog catalog, DdlStmt ddlStmt, OriginStatement ori
if (loadStmt.getVersion().equals(Load.VERSION) || jobType == EtlJobType.HADOOP) {
catalog.getLoadManager().createLoadJobV1FromStmt(loadStmt, jobType, System.currentTimeMillis());
} else {
- catalog.getLoadManager().createLoadJobFromStmt(loadStmt, origStmt);
+ catalog.getLoadManager().createLoadJobFromStmt(loadStmt, ddlStmt.getOrigStmt());
Review comment:
Parameter `ddlStmt.getOrigStmt()` is no longer needed, it can be got from `loadStmt`
##########
File path: fe/src/main/java/org/apache/doris/catalog/Column.java
##########
@@ -411,28 +412,8 @@ public boolean equals(Object obj) {
@Override
public void write(DataOutput out) throws IOException {
- Text.writeString(out, name);
- ColumnType.write(out, type);
- if (null == aggregationType) {
- out.writeBoolean(false);
- } else {
- out.writeBoolean(true);
- Text.writeString(out, aggregationType.name());
- out.writeBoolean(isAggregationTypeImplicit);
- }
-
- out.writeBoolean(isKey);
- out.writeBoolean(isAllowNull);
-
- if (defaultValue == null) {
- out.writeBoolean(false);
- } else {
- out.writeBoolean(true);
- Text.writeString(out, defaultValue);
- }
- stats.write(out);
-
- Text.writeString(out, comment);
+ String json = GsonUtils.GSON.toJson(this);
+ Text.writeString(out, json);
}
public void readFields(DataInput in) throws IOException {
Review comment:
```suggestion
private void readFields(DataInput in) throws IOException {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #3705: Serialize origin stmt in Rollup Job and MV Meta
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on a change in pull request #3705:
URL: https://github.com/apache/incubator-doris/pull/3705#discussion_r431651256
##########
File path: fe/src/main/java/org/apache/doris/alter/AlterJobV2.java
##########
@@ -220,16 +233,20 @@ protected boolean checkTableStable(Database db) throws AlterCancelException {
public abstract void replay(AlterJobV2 replayedJob);
public static AlterJobV2 read(DataInput in) throws IOException {
- JobType type = JobType.valueOf(Text.readString(in));
- switch (type) {
- case ROLLUP:
- return RollupJobV2.read(in);
- case SCHEMA_CHANGE:
- return SchemaChangeJobV2.read(in);
- default:
- Preconditions.checkState(false);
- return null;
+ if (Catalog.getCurrentCatalogJournalVersion() < FeMetaVersion.VERSION_86) {
+ JobType type = JobType.valueOf(Text.readString(in));
+ switch (type) {
+ case ROLLUP:
+ return RollupJobV2.read(in);
+ case SCHEMA_CHANGE:
+ return SchemaChangeJobV2.read(in);
+ default:
+ Preconditions.checkState(false);
+ return null;
+ }
}
Review comment:
Added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org