You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "fuweng11 (via GitHub)" <gi...@apache.org> on 2023/03/06 03:07:26 UTC
[GitHub] [inlong] fuweng11 opened a new pull request, #7526: [INLONG-7525][Manager] Support to save additional info for the ClickHouse field
fuweng11 opened a new pull request, #7526:
URL: https://github.com/apache/inlong/pull/7526
### Prepare a Pull Request
- Fixes #7525
### Motivation
1.Support to save additional info for the ClickHouse field.
2.Support to set the ttl of the table.
### Modifications
1.Support to save additional info for the ClickHouse field.
2.Support to set the ttl of the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #7526: [INLONG-7525][Manager] Support to save additional info for the ClickHouse field
Posted by "fuweng11 (via GitHub)" <gi...@apache.org>.
fuweng11 commented on code in PR #7526:
URL: https://github.com/apache/inlong/pull/7526#discussion_r1125867325
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/ck/ClickHouseSinkOperator.java:
##########
@@ -103,4 +109,65 @@ public StreamSink getFromEntity(@NotNull StreamSinkEntity entity) {
return sink;
}
+ @Override
+ public void saveFieldOpt(SinkRequest request) {
+ List<SinkField> fieldList = request.getSinkFieldList();
+ LOGGER.debug("begin to save es sink fields={}", fieldList);
+ if (CollectionUtils.isEmpty(fieldList)) {
+ return;
+ }
+ int size = fieldList.size();
+ List<StreamSinkFieldEntity> entityList = new ArrayList<>(size);
+ String groupId = request.getInlongGroupId();
+ String streamId = request.getInlongStreamId();
+ String sinkType = request.getSinkType();
+ Integer sinkId = request.getId();
+ for (SinkField fieldInfo : fieldList) {
+ this.checkFieldInfo(fieldInfo);
+ StreamSinkFieldEntity fieldEntity = CommonBeanUtils.copyProperties(fieldInfo, StreamSinkFieldEntity::new);
+ if (StringUtils.isEmpty(fieldEntity.getFieldComment())) {
+ fieldEntity.setFieldComment(fieldEntity.getFieldName());
+ }
+ try {
+ ClickHouseColumnInfo dto = ClickHouseColumnInfo.getFromRequest(fieldInfo);
+ fieldEntity.setExtParams(objectMapper.writeValueAsString(dto));
+ } catch (Exception e) {
+ throw new BusinessException(ErrorCodeEnum.SINK_SAVE_FAILED,
+ String.format("serialize extParams of ClickHouse FieldInfo failure: %s", e.getMessage()));
+ }
+ fieldEntity.setInlongGroupId(groupId);
+ fieldEntity.setInlongStreamId(streamId);
+ fieldEntity.setSinkType(sinkType);
+ fieldEntity.setSinkId(sinkId);
+ fieldEntity.setIsDeleted(InlongConstants.UN_DELETED);
+ entityList.add(fieldEntity);
+ }
+
+ sinkFieldMapper.insertAll(entityList);
+ LOGGER.debug("success to save es sink fields");
+ }
+
+ @Override
+ public List<SinkField> getSinkFields(Integer sinkId) {
+ List<StreamSinkFieldEntity> sinkFieldEntities = sinkFieldMapper.selectBySinkId(sinkId);
+ List<SinkField> fieldList = new ArrayList<>();
+ if (CollectionUtils.isEmpty(sinkFieldEntities)) {
+ return fieldList;
+ }
+ sinkFieldEntities.forEach(field -> {
+ SinkField sinkField = new SinkField();
+ if (StringUtils.isNotBlank(field.getExtParams())) {
+ ClickHouseColumnInfo clickHouseColumnInfo = ClickHouseColumnInfo.getFromJson(
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #7526: [INLONG-7525][Manager] Support to save additional info for the ClickHouse field
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow commented on code in PR #7526:
URL: https://github.com/apache/inlong/pull/7526#discussion_r1127308390
##########
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/node/ck/ClickHouseDataNodeDTO.java:
##########
@@ -53,4 +56,15 @@ public static ClickHouseDataNodeDTO getFromJson(@NotNull String extParams) {
}
}
+ /**
+ * Convert ip:post to jdbcurl.
+ */
+ public static String convertToJdbcurl(String url) {
Review Comment:
```suggestion
public static String convertToJdbcUrl(String url) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] fuweng11 commented on a diff in pull request #7526: [INLONG-7525][Manager] Support to save additional info for the ClickHouse field
Posted by "fuweng11 (via GitHub)" <gi...@apache.org>.
fuweng11 commented on code in PR #7526:
URL: https://github.com/apache/inlong/pull/7526#discussion_r1127310155
##########
inlong-manager/manager-pojo/src/main/java/org/apache/inlong/manager/pojo/node/ck/ClickHouseDataNodeDTO.java:
##########
@@ -53,4 +56,15 @@ public static ClickHouseDataNodeDTO getFromJson(@NotNull String extParams) {
}
}
+ /**
+ * Convert ip:post to jdbcurl.
+ */
+ public static String convertToJdbcurl(String url) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang commented on pull request #7526: [INLONG-7525][Manager] Support to save additional info for the ClickHouse field
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang commented on PR #7526:
URL: https://github.com/apache/inlong/pull/7526#issuecomment-1455922365
@fuweng11 please add the related UTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] vernedeng commented on a diff in pull request #7526: [INLONG-7525][Manager] Support to save additional info for the ClickHouse field
Posted by "vernedeng (via GitHub)" <gi...@apache.org>.
vernedeng commented on code in PR #7526:
URL: https://github.com/apache/inlong/pull/7526#discussion_r1125850051
##########
inlong-manager/manager-service/src/main/java/org/apache/inlong/manager/service/sink/ck/ClickHouseSinkOperator.java:
##########
@@ -103,4 +109,65 @@ public StreamSink getFromEntity(@NotNull StreamSinkEntity entity) {
return sink;
}
+ @Override
+ public void saveFieldOpt(SinkRequest request) {
+ List<SinkField> fieldList = request.getSinkFieldList();
+ LOGGER.debug("begin to save es sink fields={}", fieldList);
+ if (CollectionUtils.isEmpty(fieldList)) {
+ return;
+ }
+ int size = fieldList.size();
+ List<StreamSinkFieldEntity> entityList = new ArrayList<>(size);
+ String groupId = request.getInlongGroupId();
+ String streamId = request.getInlongStreamId();
+ String sinkType = request.getSinkType();
+ Integer sinkId = request.getId();
+ for (SinkField fieldInfo : fieldList) {
+ this.checkFieldInfo(fieldInfo);
+ StreamSinkFieldEntity fieldEntity = CommonBeanUtils.copyProperties(fieldInfo, StreamSinkFieldEntity::new);
+ if (StringUtils.isEmpty(fieldEntity.getFieldComment())) {
+ fieldEntity.setFieldComment(fieldEntity.getFieldName());
+ }
+ try {
+ ClickHouseColumnInfo dto = ClickHouseColumnInfo.getFromRequest(fieldInfo);
+ fieldEntity.setExtParams(objectMapper.writeValueAsString(dto));
+ } catch (Exception e) {
+ throw new BusinessException(ErrorCodeEnum.SINK_SAVE_FAILED,
+ String.format("serialize extParams of ClickHouse FieldInfo failure: %s", e.getMessage()));
+ }
+ fieldEntity.setInlongGroupId(groupId);
+ fieldEntity.setInlongStreamId(streamId);
+ fieldEntity.setSinkType(sinkType);
+ fieldEntity.setSinkId(sinkId);
+ fieldEntity.setIsDeleted(InlongConstants.UN_DELETED);
+ entityList.add(fieldEntity);
+ }
+
+ sinkFieldMapper.insertAll(entityList);
+ LOGGER.debug("success to save es sink fields");
+ }
+
+ @Override
+ public List<SinkField> getSinkFields(Integer sinkId) {
+ List<StreamSinkFieldEntity> sinkFieldEntities = sinkFieldMapper.selectBySinkId(sinkId);
+ List<SinkField> fieldList = new ArrayList<>();
+ if (CollectionUtils.isEmpty(sinkFieldEntities)) {
+ return fieldList;
+ }
+ sinkFieldEntities.forEach(field -> {
+ SinkField sinkField = new SinkField();
+ if (StringUtils.isNotBlank(field.getExtParams())) {
+ ClickHouseColumnInfo clickHouseColumnInfo = ClickHouseColumnInfo.getFromJson(
Review Comment:
I think **_ClickHouseFieldInfo_** is a better class name, since it inherit from SinkFields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow merged pull request #7526: [INLONG-7525][Manager] Support to save additional info for the ClickHouse field
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow merged PR #7526:
URL: https://github.com/apache/inlong/pull/7526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org