You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/11/21 08:07:16 UTC
[GitHub] [inlong] chestnut-c opened a new pull request, #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
chestnut-c opened a new pull request, #6585:
URL: https://github.com/apache/inlong/pull/6585
### Prepare a Pull Request
Add read phase metric and table level metric for MySQL
- Title Example: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
*(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)*
- Fixes #6584
### Motivation
Add read phase metric and table level metric for MySQL
### Modifications
Add read phase metric and table level metric for MySQL
### Verifying this change
*(Please pick either of the following options)*
- [ ] This change is a trivial rework/code cleanup without any test coverage.
- [ ] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1029977267
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/metric/SourceMetricData.java:
##########
@@ -222,18 +389,119 @@ public void outputMetrics(long rowCountSize, long rowDataSize) {
}
}
+ public void outputMetrics(long rowCountSize, long rowDataSize, boolean isSnapshotRecord) {
+
+ outputMetrics(rowCountSize, rowDataSize);
+
+ if (readPhase == null) {
+ return;
+ }
+ long count = this.readPhase.getCount();
+ if (isSnapshotRecord && count != SNAPSHOT_PHASE) {
+ this.readPhase.dec(count);
+ this.readPhase.inc(SNAPSHOT_PHASE);
+ } else if (!isSnapshotRecord && count != INCREASE_PHASE) {
+ this.readPhase.dec(count);
+ this.readPhase.inc(INCREASE_PHASE);
+ }
+ }
+
@Override
public String toString() {
return "SourceMetricData{"
+ "metricGroup=" + metricGroup
+ ", labels=" + labels
+ + ", metricType=" + metricType
+ + ", readPhase=" + (readPhase != null ? readPhase.getCount() : null)
+ ", numRecordsIn=" + numRecordsIn.getCount()
+ ", numBytesIn=" + numBytesIn.getCount()
+ ", numRecordsInForMeter=" + numRecordsInForMeter.getCount()
+ ", numBytesInForMeter=" + numBytesInForMeter.getCount()
+ ", numRecordsInPerSecond=" + numRecordsInPerSecond.getRate()
+ ", numBytesInPerSecond=" + numBytesInPerSecond.getRate()
+ ", auditOperator=" + auditOperator
+ + ", subSourceMetricMap=" + subSourceMetricMap
+ '}';
}
+
+ /**
+ * Source Record Schema Info
+ */
+ public static class SourceRecordSchemaInfo {
+
+ @NonNull
+ private String databaseName;
Review Comment:
ok，I will add the form of kafka topic, although it is not used at present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on PR #6585:
URL: https://github.com/apache/inlong/pull/6585#issuecomment-1325168980
> > > Please explain in detail the requirements background and design ideas？
> >
> >
> > Hi , The background is that when the entire database is migrated, the business needs to count the indicators of table granularity. Dynamically generate table granularity MetricGroup according to the schema of data records; internally maintain MetricGroup container to realize table granularity reporting indicators.
>
> You can add this to the describe of pr.
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c closed pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c closed pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
URL: https://github.com/apache/inlong/pull/6585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1030431087
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -73,11 +76,48 @@ public final class Constants {
* Node id used in inlong metric
*/
public static final String NODE_ID = "nodeId";
+ /**
+ * Database Name used in inlong metric
+ */
+ public static final String DATABASE_NAME = "database";
Review Comment:
> These constants are not considered general concepts, it is recommended to consider a more general design？
Specific tags are listed for flexible business use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1031489597
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/metric/SourceMetricData.java:
##########
@@ -189,11 +237,126 @@ public Map<String, String> getLabels() {
return labels;
}
+ /**
+ * register sub metrics group from metric state
+ *
+ * @param metricState MetricState
+ */
+ public void registerSubMetricsGroup(MetricState metricState) {
+ if (metricState == null || metricState.getSubMetricStateMap() == null
+ || metricState.getSubMetricStateMap().isEmpty()) {
+ return;
+ }
+
+ Map<String, MetricState> subMetricStateMap = metricState.getSubMetricStateMap();
+ for (Entry<String, MetricState> subMetricStateEntry : subMetricStateMap.entrySet()) {
+ String schemaIdentify = subMetricStateEntry.getKey();
+ SourceRecordSchemaInfo sourceRecordSchemaInfo = new SourceRecordSchemaInfo(schemaIdentify);
+ final MetricState subMetricState = subMetricStateEntry.getValue();
+ SourceMetricData subSourceMetricData = buildSubSourceMetricData(sourceRecordSchemaInfo,
+ subMetricState, this);
+ subSourceMetricMap.put(subMetricStateEntry.getKey(), subSourceMetricData);
+ }
+ LOGGER.info("register subMetricsGroup from metricState,sub metric map size:{}", subSourceMetricMap.size());
+ }
+
+ /**
+ * build sub source metric data
+ *
+ * @param recordSchemaInfo source record schema info
+ * @param subMetricState sub metric state
+ * @param sourceMetricData source metric data
+ * @return sub source metric data
+ */
+ private SourceMetricData buildSubSourceMetricData(SourceRecordSchemaInfo recordSchemaInfo,
+ MetricState subMetricState, SourceMetricData sourceMetricData) {
+ if (sourceMetricData == null || recordSchemaInfo == null) {
+ return null;
+ }
+
+ // build sub metricGroup labels
+ String metricGroupLabels = this.labels.entrySet().stream().map(entry -> entry.getKey() + "=" + entry.getValue())
+ .collect(Collectors.joining(DELIMITER));
+
+ StringBuilder labelStringBuilder = new StringBuilder(metricGroupLabels);
+ String topicName = recordSchemaInfo.getTopicName();
+ if (StringUtils.isNotBlank(topicName)) {
+ // judging only the topic
+ labelStringBuilder.append(DELIMITER).append(Constants.TOPIC_NAME).append("=").append(topicName);
+ } else {
+ // judge The case of database.schema.table or database.table
+ String databaseName = recordSchemaInfo.getDatabaseName();
+ labelStringBuilder.append(DELIMITER).append(Constants.DATABASE_NAME).append("=").append(databaseName);
+ String schemaName = recordSchemaInfo.getSchemaName();
+ if (StringUtils.isNotBlank(schemaName)) {
+ labelStringBuilder.append(DELIMITER).append(Constants.SCHEMA_NAME).append("=").append(schemaName);
+ }
+ String tableName = recordSchemaInfo.getTableName();
+ labelStringBuilder.append(DELIMITER).append(Constants.TABLE_NAME).append("=").append(tableName);
+ }
+
+ // build option labels
+ MetricOption metricOption = MetricOption.builder()
+ .withInitRecords(subMetricState != null ? subMetricState.getMetricValue(NUM_RECORDS_IN) : 0L)
+ .withInitBytes(subMetricState != null ? subMetricState.getMetricValue(NUM_BYTES_IN) : 0L)
+ .withReadPhase(subMetricState != null ? subMetricState.getMetricValue(READ_PHASE) : 0L)
+ .withInlongLabels(labelStringBuilder.toString())
+ .withRegisterMetric(RegisteredMetric.NORMAL)
+ .build();
+ return new SourceMetricData(metricOption, sourceMetricData.getMetricGroup());
+ }
+
+ /**
+ * build record schema identify
+ *
+ * @param recordSchemaInfo source record schema info
+ * @return record schema identify
+ */
+ public String buildSchemaIdentify(SourceRecordSchemaInfo recordSchemaInfo) {
+ String database = recordSchemaInfo.getDatabaseName();
+ String topicName = recordSchemaInfo.getTopicName();
+ // Judging only the topic
+ if (StringUtils.isNotBlank(topicName)) {
+ return topicName;
+ }
+ // judge The case of database.schema.table or database.table
+ String table = recordSchemaInfo.getTableName();
+ String schema = recordSchemaInfo.getSchemaName();
+ StringBuilder identifyBuilder = new StringBuilder();
+ identifyBuilder.append(database).append(Constants.SEMICOLON);
+ if (StringUtils.isNotBlank(schema)) {
+ identifyBuilder.append(schema).append(Constants.SEMICOLON);
+ }
+ identifyBuilder.append(table);
+ return identifyBuilder.toString();
+ }
+
public void outputMetricsWithEstimate(Object o) {
long size = o.toString().getBytes(StandardCharsets.UTF_8).length;
outputMetrics(1, size);
}
+ public void outputMetricsWithEstimate(SourceRecordSchemaInfo recordSchemaInfo, Object o) {
Review Comment:
> Object data
OK ,i will change the name to data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] thesumery commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1030355672
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/metric/SourceMetricData.java:
##########
@@ -222,18 +389,119 @@ public void outputMetrics(long rowCountSize, long rowDataSize) {
}
}
+ public void outputMetrics(long rowCountSize, long rowDataSize, boolean isSnapshotRecord) {
+
+ outputMetrics(rowCountSize, rowDataSize);
+
+ if (readPhase == null) {
+ return;
+ }
+ long count = this.readPhase.getCount();
+ if (isSnapshotRecord && count != SNAPSHOT_PHASE) {
+ this.readPhase.dec(count);
+ this.readPhase.inc(SNAPSHOT_PHASE);
+ } else if (!isSnapshotRecord && count != INCREASE_PHASE) {
+ this.readPhase.dec(count);
+ this.readPhase.inc(INCREASE_PHASE);
+ }
+ }
+
@Override
public String toString() {
return "SourceMetricData{"
+ "metricGroup=" + metricGroup
+ ", labels=" + labels
+ + ", metricType=" + metricType
+ + ", readPhase=" + (readPhase != null ? readPhase.getCount() : null)
+ ", numRecordsIn=" + numRecordsIn.getCount()
+ ", numBytesIn=" + numBytesIn.getCount()
+ ", numRecordsInForMeter=" + numRecordsInForMeter.getCount()
+ ", numBytesInForMeter=" + numBytesInForMeter.getCount()
+ ", numRecordsInPerSecond=" + numRecordsInPerSecond.getRate()
+ ", numBytesInPerSecond=" + numBytesInPerSecond.getRate()
+ ", auditOperator=" + auditOperator
+ + ", subSourceMetricMap=" + subSourceMetricMap
+ '}';
}
+
+ /**
+ * Source Record Schema Info
+ */
+ public static class SourceRecordSchemaInfo {
+
+ @NonNull
+ private String databaseName;
Review Comment:
i mean use a `fullTableName` instead of using `databaseName`、`tableName`、`topic`，those field could be determinded by connectors themselves
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] yunqingmoswu commented on pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on PR #6585:
URL: https://github.com/apache/inlong/pull/6585#issuecomment-1324837596
Please explain in detail the requirements background and design ideas？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1031096745
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -73,11 +79,48 @@ public final class Constants {
* Node id used in inlong metric
*/
public static final String NODE_ID = "nodeId";
+ /**
+ * Database Name used in inlong metric
+ */
+ public static final String DATABASE_NAME = "database";
+ /**
+ * Table Name used in inlong metric
+ */
+ public static final String TABLE_NAME = "table";
+ /**
+ * Schema Name used in inlong metric
+ */
+ public static final String SCHEMA_NAME = "schema";
+ /**
+ * Topic Name used in inlong metric
+ */
+ public static final String TOPIC_NAME = "topic";
+
/**
* It is used for inlong.metric
*/
public static final String DELIMITER = "&";
+ /**
+ * It is used for metric data to build schema identify
+ */
+ public static final String SEMICOLON = ".";
+
+ /**
+ * It is used for metric data to spilt schema identify
+ */
+ public static final String SPILT_SEMICOLON = "\\.";
+
+ /**
+ * This indicator represents the snapshot phase during the read phase
+ */
+ public static final Long SNAPSHOT_PHASE = 1L;
Review Comment:
suggest use enum for snapshot or incremental
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c closed pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c closed pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
URL: https://github.com/apache/inlong/pull/6585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1028374393
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -73,11 +76,44 @@ public final class Constants {
* Node id used in inlong metric
*/
public static final String NODE_ID = "nodeId";
+ /**
+ * Database Name used in inlong metric
+ */
+ public static final String DATABASE_NAME = "database";
+ /**
+ * Table Name used in inlong metric
+ */
+ public static final String TABLE_NAME = "table";
+ /**
+ * Schema Name used in inlong metric
+ */
+ public static final String SCHEMA_NAME = "schema";
+
/**
* It is used for inlong.metric
*/
public static final String DELIMITER = "&";
+ /**
+ * It is used for inlong.metric
+ */
+ public static final String SEMICOLON = ".";
+
+ /**
+ * It is used for inlong.metric
+ */
+ public static final String SPILT_SEMICOLON = "\\.";
+
+ /**
+ * It is used for inlong.metric
+ */
+ public static final Long SNAPSHOT_PHASE = 1L;
+
+ /**
+ * It is used for inlong.metric
+ */
+ public static final Long INCREASE_PHASE = 2L;
+
Review Comment:
okay i will fix it ,thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1029273924
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/MetricType.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base;
+
+/**
+ * connector metric type enum
+ */
+public enum MetricType {
+
+ /**
+ * node level metric
+ */
+ NODE,
Review Comment:
> node level why not add streamId groupId nodeId label?
The node level has added streamId groupId nodeId tags in the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on PR #6585:
URL: https://github.com/apache/inlong/pull/6585#issuecomment-1324867109
> Please explain in detail the requirements background and design ideas？
hi , The background is that when the entire database is migrated, the business needs to count the indicators of table granularity. Dynamically generate table granularity MetricGroup according to the schema of data records; internally maintain MetricGroup container to realize table granularity reporting indicators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1031190852
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/metric/SourceMetricData.java:
##########
@@ -189,11 +237,126 @@ public Map<String, String> getLabels() {
return labels;
}
+ /**
+ * register sub metrics group from metric state
+ *
+ * @param metricState MetricState
+ */
+ public void registerSubMetricsGroup(MetricState metricState) {
+ if (metricState == null || metricState.getSubMetricStateMap() == null
+ || metricState.getSubMetricStateMap().isEmpty()) {
+ return;
+ }
+
+ Map<String, MetricState> subMetricStateMap = metricState.getSubMetricStateMap();
+ for (Entry<String, MetricState> subMetricStateEntry : subMetricStateMap.entrySet()) {
+ String schemaIdentify = subMetricStateEntry.getKey();
+ SourceRecordSchemaInfo sourceRecordSchemaInfo = new SourceRecordSchemaInfo(schemaIdentify);
+ final MetricState subMetricState = subMetricStateEntry.getValue();
+ SourceMetricData subSourceMetricData = buildSubSourceMetricData(sourceRecordSchemaInfo,
+ subMetricState, this);
+ subSourceMetricMap.put(subMetricStateEntry.getKey(), subSourceMetricData);
+ }
+ LOGGER.info("register subMetricsGroup from metricState,sub metric map size:{}", subSourceMetricMap.size());
+ }
+
+ /**
+ * build sub source metric data
+ *
+ * @param recordSchemaInfo source record schema info
+ * @param subMetricState sub metric state
+ * @param sourceMetricData source metric data
+ * @return sub source metric data
+ */
+ private SourceMetricData buildSubSourceMetricData(SourceRecordSchemaInfo recordSchemaInfo,
+ MetricState subMetricState, SourceMetricData sourceMetricData) {
+ if (sourceMetricData == null || recordSchemaInfo == null) {
+ return null;
+ }
+
+ // build sub metricGroup labels
+ String metricGroupLabels = this.labels.entrySet().stream().map(entry -> entry.getKey() + "=" + entry.getValue())
+ .collect(Collectors.joining(DELIMITER));
+
+ StringBuilder labelStringBuilder = new StringBuilder(metricGroupLabels);
+ String topicName = recordSchemaInfo.getTopicName();
+ if (StringUtils.isNotBlank(topicName)) {
+ // judging only the topic
+ labelStringBuilder.append(DELIMITER).append(Constants.TOPIC_NAME).append("=").append(topicName);
+ } else {
+ // judge The case of database.schema.table or database.table
+ String databaseName = recordSchemaInfo.getDatabaseName();
+ labelStringBuilder.append(DELIMITER).append(Constants.DATABASE_NAME).append("=").append(databaseName);
+ String schemaName = recordSchemaInfo.getSchemaName();
+ if (StringUtils.isNotBlank(schemaName)) {
+ labelStringBuilder.append(DELIMITER).append(Constants.SCHEMA_NAME).append("=").append(schemaName);
+ }
+ String tableName = recordSchemaInfo.getTableName();
+ labelStringBuilder.append(DELIMITER).append(Constants.TABLE_NAME).append("=").append(tableName);
+ }
+
+ // build option labels
+ MetricOption metricOption = MetricOption.builder()
+ .withInitRecords(subMetricState != null ? subMetricState.getMetricValue(NUM_RECORDS_IN) : 0L)
+ .withInitBytes(subMetricState != null ? subMetricState.getMetricValue(NUM_BYTES_IN) : 0L)
+ .withReadPhase(subMetricState != null ? subMetricState.getMetricValue(READ_PHASE) : 0L)
+ .withInlongLabels(labelStringBuilder.toString())
+ .withRegisterMetric(RegisteredMetric.NORMAL)
+ .build();
+ return new SourceMetricData(metricOption, sourceMetricData.getMetricGroup());
+ }
+
+ /**
+ * build record schema identify
+ *
+ * @param recordSchemaInfo source record schema info
+ * @return record schema identify
+ */
+ public String buildSchemaIdentify(SourceRecordSchemaInfo recordSchemaInfo) {
+ String database = recordSchemaInfo.getDatabaseName();
+ String topicName = recordSchemaInfo.getTopicName();
+ // Judging only the topic
+ if (StringUtils.isNotBlank(topicName)) {
+ return topicName;
+ }
+ // judge The case of database.schema.table or database.table
+ String table = recordSchemaInfo.getTableName();
+ String schema = recordSchemaInfo.getSchemaName();
+ StringBuilder identifyBuilder = new StringBuilder();
+ identifyBuilder.append(database).append(Constants.SEMICOLON);
+ if (StringUtils.isNotBlank(schema)) {
+ identifyBuilder.append(schema).append(Constants.SEMICOLON);
+ }
+ identifyBuilder.append(table);
+ return identifyBuilder.toString();
+ }
+
public void outputMetricsWithEstimate(Object o) {
long size = o.toString().getBytes(StandardCharsets.UTF_8).length;
outputMetrics(1, size);
}
+ public void outputMetricsWithEstimate(SourceRecordSchemaInfo recordSchemaInfo, Object o) {
Review Comment:
Object data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] thesumery commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1029283146
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/metric/SourceMetricData.java:
##########
@@ -222,18 +389,119 @@ public void outputMetrics(long rowCountSize, long rowDataSize) {
}
}
+ public void outputMetrics(long rowCountSize, long rowDataSize, boolean isSnapshotRecord) {
+
+ outputMetrics(rowCountSize, rowDataSize);
+
+ if (readPhase == null) {
+ return;
+ }
+ long count = this.readPhase.getCount();
+ if (isSnapshotRecord && count != SNAPSHOT_PHASE) {
+ this.readPhase.dec(count);
+ this.readPhase.inc(SNAPSHOT_PHASE);
+ } else if (!isSnapshotRecord && count != INCREASE_PHASE) {
+ this.readPhase.dec(count);
+ this.readPhase.inc(INCREASE_PHASE);
+ }
+ }
+
@Override
public String toString() {
return "SourceMetricData{"
+ "metricGroup=" + metricGroup
+ ", labels=" + labels
+ + ", metricType=" + metricType
+ + ", readPhase=" + (readPhase != null ? readPhase.getCount() : null)
+ ", numRecordsIn=" + numRecordsIn.getCount()
+ ", numBytesIn=" + numBytesIn.getCount()
+ ", numRecordsInForMeter=" + numRecordsInForMeter.getCount()
+ ", numBytesInForMeter=" + numBytesInForMeter.getCount()
+ ", numRecordsInPerSecond=" + numRecordsInPerSecond.getRate()
+ ", numBytesInPerSecond=" + numBytesInPerSecond.getRate()
+ ", auditOperator=" + auditOperator
+ + ", subSourceMetricMap=" + subSourceMetricMap
+ '}';
}
+
+ /**
+ * Source Record Schema Info
+ */
+ public static class SourceRecordSchemaInfo {
+
+ @NonNull
+ private String databaseName;
Review Comment:
Could it put a `fullTableName` here and transfer of naming rights to API Users. For example, mysql is '${db}.${table}', oracle is '${db}.${schema}.${table}', kafka is '${topic}'
databaseName、tableName、schemaName is not the model of table level metric and sourceMetricData does not need to know them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1030403641
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -73,11 +76,48 @@ public final class Constants {
* Node id used in inlong metric
*/
public static final String NODE_ID = "nodeId";
+ /**
+ * Database Name used in inlong metric
+ */
+ public static final String DATABASE_NAME = "database";
Review Comment:
These constants are not considered general concepts, it is recommended to consider a more general design？
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -57,6 +57,9 @@ public final class Constants {
public static final String NUM_BYTES_IN_PER_SECOND = "numBytesInPerSecond";
public static final String NUM_RECORDS_IN_PER_SECOND = "numRecordsInPerSecond";
+
+ public static final String READ_PHASE = "readPhase";
Review Comment:
Please add some comment for it,simply looking at this variable does not know the specific meaning?
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/MetricType.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base;
+
+/**
+ * connector metric type enum
+ */
+public enum MetricType {
Review Comment:
It seems unnecessary to define the metric type, just extend it on the option 'inlong.metric.label'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1030435173
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/MetricType.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base;
+
+/**
+ * connector metric type enum
+ */
+public enum MetricType {
Review Comment:
> It seems unnecessary to define the metric type, just extend it on the option 'inlong.metric.label'.
This enumeration is to distinguish Node indicators or more fine-grained indicators. Fine-grained indicators are currently expanded on the main node indicators. The purpose of adding enumeration is to facilitate maintenance, if not needed also ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1030534863
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -57,6 +57,9 @@ public final class Constants {
public static final String NUM_BYTES_IN_PER_SECOND = "numBytesInPerSecond";
public static final String NUM_RECORDS_IN_PER_SECOND = "numRecordsInPerSecond";
+
+ public static final String READ_PHASE = "readPhase";
Review Comment:
> Please add some comment for it,simply looking at this variable does not know the specific meaning?
ok. i will some comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on PR #6585:
URL: https://github.com/apache/inlong/pull/6585#issuecomment-1325197148
> > > Please explain in detail the requirements background and design ideas？
> >
> >
> > Hi , The background is that when the entire database is migrated, the business needs to count the indicators of table granularity. Dynamically generate table granularity MetricGroup according to the schema of data records; internally maintain MetricGroup container to realize table granularity reporting indicators.
>
> You can add this to the describe of pr.
ok i will add it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1030428437
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -57,6 +57,9 @@ public final class Constants {
public static final String NUM_BYTES_IN_PER_SECOND = "numBytesInPerSecond";
public static final String NUM_RECORDS_IN_PER_SECOND = "numRecordsInPerSecond";
+
+ public static final String READ_PHASE = "readPhase";
Review Comment:
> Please add some comment for it,simply looking at this variable does not know the specific meaning?
The whole database migration stage distinguishes between snapshot and incremental data, business indicators need to know the current stage。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1029977365
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/metric/SourceMetricData.java:
##########
@@ -222,18 +389,119 @@ public void outputMetrics(long rowCountSize, long rowDataSize) {
}
}
+ public void outputMetrics(long rowCountSize, long rowDataSize, boolean isSnapshotRecord) {
+
+ outputMetrics(rowCountSize, rowDataSize);
+
+ if (readPhase == null) {
+ return;
+ }
+ long count = this.readPhase.getCount();
+ if (isSnapshotRecord && count != SNAPSHOT_PHASE) {
+ this.readPhase.dec(count);
+ this.readPhase.inc(SNAPSHOT_PHASE);
+ } else if (!isSnapshotRecord && count != INCREASE_PHASE) {
+ this.readPhase.dec(count);
+ this.readPhase.inc(INCREASE_PHASE);
+ }
+ }
+
@Override
public String toString() {
return "SourceMetricData{"
+ "metricGroup=" + metricGroup
+ ", labels=" + labels
+ + ", metricType=" + metricType
+ + ", readPhase=" + (readPhase != null ? readPhase.getCount() : null)
+ ", numRecordsIn=" + numRecordsIn.getCount()
+ ", numBytesIn=" + numBytesIn.getCount()
+ ", numRecordsInForMeter=" + numRecordsInForMeter.getCount()
+ ", numBytesInForMeter=" + numBytesInForMeter.getCount()
+ ", numRecordsInPerSecond=" + numRecordsInPerSecond.getRate()
+ ", numBytesInPerSecond=" + numBytesInPerSecond.getRate()
+ ", auditOperator=" + auditOperator
+ + ", subSourceMetricMap=" + subSourceMetricMap
+ '}';
}
+
+ /**
+ * Source Record Schema Info
+ */
+ public static class SourceRecordSchemaInfo {
+
+ @NonNull
+ private String databaseName;
Review Comment:
> Could it put a `fullTableName` here and transfer of naming rights to API Users. For example, mysql is '${db}.${table}', oracle is '${db}.${schema}.${table}', kafka is '${topic}' databaseName、tableName、schemaName is not the model of table level metric and sourceMetricData does not need to know them.
I will add the form of kafka topic, although it is not used at present,but In order to facilitate the query and use of business aspects, the original method is still maintained, and only the form of kafka topic is added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1029977365
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/metric/SourceMetricData.java:
##########
@@ -222,18 +389,119 @@ public void outputMetrics(long rowCountSize, long rowDataSize) {
}
}
+ public void outputMetrics(long rowCountSize, long rowDataSize, boolean isSnapshotRecord) {
+
+ outputMetrics(rowCountSize, rowDataSize);
+
+ if (readPhase == null) {
+ return;
+ }
+ long count = this.readPhase.getCount();
+ if (isSnapshotRecord && count != SNAPSHOT_PHASE) {
+ this.readPhase.dec(count);
+ this.readPhase.inc(SNAPSHOT_PHASE);
+ } else if (!isSnapshotRecord && count != INCREASE_PHASE) {
+ this.readPhase.dec(count);
+ this.readPhase.inc(INCREASE_PHASE);
+ }
+ }
+
@Override
public String toString() {
return "SourceMetricData{"
+ "metricGroup=" + metricGroup
+ ", labels=" + labels
+ + ", metricType=" + metricType
+ + ", readPhase=" + (readPhase != null ? readPhase.getCount() : null)
+ ", numRecordsIn=" + numRecordsIn.getCount()
+ ", numBytesIn=" + numBytesIn.getCount()
+ ", numRecordsInForMeter=" + numRecordsInForMeter.getCount()
+ ", numBytesInForMeter=" + numBytesInForMeter.getCount()
+ ", numRecordsInPerSecond=" + numRecordsInPerSecond.getRate()
+ ", numBytesInPerSecond=" + numBytesInPerSecond.getRate()
+ ", auditOperator=" + auditOperator
+ + ", subSourceMetricMap=" + subSourceMetricMap
+ '}';
}
+
+ /**
+ * Source Record Schema Info
+ */
+ public static class SourceRecordSchemaInfo {
+
+ @NonNull
+ private String databaseName;
Review Comment:
> Could it put a `fullTableName` here and transfer of naming rights to API Users. For example, mysql is '${db}.${table}', oracle is '${db}.${schema}.${table}', kafka is '${topic}' databaseName、tableName、schemaName is not the model of table level metric and sourceMetricData does not need to know them.
I will add the form of kafka topic, although it is not used at present
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1031490303
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -73,11 +79,48 @@ public final class Constants {
* Node id used in inlong metric
*/
public static final String NODE_ID = "nodeId";
+ /**
+ * Database Name used in inlong metric
+ */
+ public static final String DATABASE_NAME = "database";
+ /**
+ * Table Name used in inlong metric
+ */
+ public static final String TABLE_NAME = "table";
+ /**
+ * Schema Name used in inlong metric
+ */
+ public static final String SCHEMA_NAME = "schema";
+ /**
+ * Topic Name used in inlong metric
+ */
+ public static final String TOPIC_NAME = "topic";
+
/**
* It is used for inlong.metric
*/
public static final String DELIMITER = "&";
+ /**
+ * It is used for metric data to build schema identify
+ */
+ public static final String SEMICOLON = ".";
+
+ /**
+ * It is used for metric data to spilt schema identify
+ */
+ public static final String SPILT_SEMICOLON = "\\.";
+
+ /**
+ * This indicator represents the snapshot phase during the read phase
+ */
+ public static final Long SNAPSHOT_PHASE = 1L;
Review Comment:
> suggest use enum for snapshot or incremental
good idea ,i will do it .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] yunqingmoswu commented on pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on PR #6585:
URL: https://github.com/apache/inlong/pull/6585#issuecomment-1328491853
> > It is recommended to extend the option 'inlong.metric.labels' instead of adding metric type. For example you can format as 'groupId=xxx&streamId=xxx&nodeId=xxx&database=${database}&table=${table}', in this format the '${xxx}' represents a variable, and the specific supported variables are determined by the connector.
>
> > 'inlong.metric.labels'
>
> If you use what you said to do it, it feels a bit redundant. First of all, you can’t directly get the label at the node level, unless you add 'inlong.metric.tableLabels'. It is better to hand over the specific sub-label to the specific connector.
Normally, it should be like this. The reporting of indicators based on nodeId is closed, so it is a closed loop inside sort, but this function has broken this agreement and is open. Then, general rules should be defined, and each connector can implement it according to the situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1029273924
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/MetricType.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base;
+
+/**
+ * connector metric type enum
+ */
+public enum MetricType {
+
+ /**
+ * node level metric
+ */
+ NODE,
Review Comment:
> node level why not add streamId groupId nodeId label?
The node level has added streamId groupId nodeId tags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] thesumery commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
thesumery commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1029263544
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/MetricType.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base;
+
+/**
+ * connector metric type enum
+ */
+public enum MetricType {
+
+ /**
+ * node level metric
+ */
+ NODE,
Review Comment:
node level why not add streamId groupId nodeId label?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1030435173
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/MetricType.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base;
+
+/**
+ * connector metric type enum
+ */
+public enum MetricType {
Review Comment:
> It seems unnecessary to define the metric type, just extend it on the option 'inlong.metric.label'.
This enumeration is to distinguish Node indicators or more fine-grained indicators. Fine-grained indicators are currently expanded on the main node indicators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] chestnut-c commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
chestnut-c commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1030476454
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/MetricType.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.base;
+
+/**
+ * connector metric type enum
+ */
+public enum MetricType {
Review Comment:
> It seems unnecessary to define the metric type, just extend it on the option 'inlong.metric.label'.
i will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] yunqingmoswu commented on pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on PR #6585:
URL: https://github.com/apache/inlong/pull/6585#issuecomment-1325006799
> > Please explain in detail the requirements background and design ideas？
>
> Hi , The background is that when the entire database is migrated, the business needs to count the indicators of table granularity. Dynamically generate table granularity MetricGroup according to the schema of data records; internally maintain MetricGroup container to realize table granularity reporting indicators.
You can add this to the describe of pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #6585: [INLONG-6584][Sort] Add read phase metric and table level metric for MySQL
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #6585:
URL: https://github.com/apache/inlong/pull/6585#discussion_r1028157102
##########
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java:
##########
@@ -73,11 +76,44 @@ public final class Constants {
* Node id used in inlong metric
*/
public static final String NODE_ID = "nodeId";
+ /**
+ * Database Name used in inlong metric
+ */
+ public static final String DATABASE_NAME = "database";
+ /**
+ * Table Name used in inlong metric
+ */
+ public static final String TABLE_NAME = "table";
+ /**
+ * Schema Name used in inlong metric
+ */
+ public static final String SCHEMA_NAME = "schema";
+
/**
* It is used for inlong.metric
*/
public static final String DELIMITER = "&";
+ /**
+ * It is used for inlong.metric
+ */
+ public static final String SEMICOLON = ".";
+
+ /**
+ * It is used for inlong.metric
+ */
+ public static final String SPILT_SEMICOLON = "\\.";
+
+ /**
+ * It is used for inlong.metric
+ */
+ public static final Long SNAPSHOT_PHASE = 1L;
+
+ /**
+ * It is used for inlong.metric
+ */
+ public static final Long INCREASE_PHASE = 2L;
+
Review Comment:
`It is used for inlong.metric` should be modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org