You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "LiJie20190102 (via GitHub)" <gi...@apache.org> on 2023/11/25 03:57:07 UTC
[PR] [INLONG-9333][Sort] Related problem with attribute exceptions when creating 'hudiSink' [inlong]
LiJie20190102 opened a new pull request, #9334:
URL: https://github.com/apache/inlong/pull/9334
Related problem with attribute exceptions when creating 'hudiSink'
### Prepare a Pull Request
*(Change the title refer to the following example)*
- [INLONG-9333][Sort] Related problem with attribute exceptions when creating 'hudiSink'
*(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)*
- Fixes #9333
### Motivation
When I tried to create a 'hudiSink', some properties did not prompt me to enter, but an error occurred
### Modifications
If some parameters are necessary, the task creation will prompt that these parameters must be assigned values
### Verifying this change
*(Please pick either of the following options)*
- [ ] This change is a trivial rework/code cleanup without any test coverage.
- [ √] This change is already covered by existing tests, such as:
1. Execute this method：org.apache.inlong.manager.client.File2HudiExample#testCreateGroupForHudi, an exception will occur in manager all log. Please refer to #9333 for detailed error logs;
2. After adding the code I added, the task can be successfully created
- [ ] This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-9333][Sort] Related problem with attribute exceptions when creating 'hudiSink' [inlong]
Posted by "LiJie20190102 (via GitHub)" <gi...@apache.org>.
LiJie20190102 commented on code in PR #9334:
URL: https://github.com/apache/inlong/pull/9334#discussion_r1405567037
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/HudiExtractNode.java:
##########
@@ -142,15 +143,17 @@ public Map<String, String> tableOptions() {
// If the extend attributes starts with .ddl,
// it will be passed to the ddl statement of the table
- extList.forEach(ext -> {
- String keyName = ext.get(EXTEND_ATTR_KEY_NAME);
- if (StringUtils.isNoneBlank(keyName) &&
- keyName.startsWith(DDL_ATTR_PREFIX)) {
- String ddlKeyName = keyName.substring(DDL_ATTR_PREFIX.length());
- String ddlValue = ext.get(EXTEND_ATTR_VALUE_NAME);
- options.put(ddlKeyName, ddlValue);
- }
- });
+ if (CollectionUtils.isNotEmpty(extList)) {
Review Comment:
I also think the same way. Initially, I prohibited 'primaryKey' from being empty, but I saw that there was already a 'PR' (#9323） solution to avoid setting 'primaryKey', so I changed it back. At the same time, the official website did not say that this configuration is mandatory. Finally, I believe it should not be empty, as normal use requires the use of primary keys. What do others think?
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/HudiExtractNode.java:
##########
@@ -142,15 +143,17 @@ public Map<String, String> tableOptions() {
// If the extend attributes starts with .ddl,
// it will be passed to the ddl statement of the table
- extList.forEach(ext -> {
- String keyName = ext.get(EXTEND_ATTR_KEY_NAME);
- if (StringUtils.isNoneBlank(keyName) &&
- keyName.startsWith(DDL_ATTR_PREFIX)) {
- String ddlKeyName = keyName.substring(DDL_ATTR_PREFIX.length());
- String ddlValue = ext.get(EXTEND_ATTR_VALUE_NAME);
- options.put(ddlKeyName, ddlValue);
- }
- });
+ if (CollectionUtils.isNotEmpty(extList)) {
Review Comment:
I also think the same way. Initially, I prohibited 'primaryKey' from being empty, but I saw that there was already a 'PR' (#9323） solution to avoid setting 'primaryKey', so I changed it back. At the same time, the official website did not say that this configuration is mandatory. Finally, I believe it should not be empty, as normal use requires the use of primary keys. What do others think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-9333][Sort] Related problem with attribute exceptions when creating 'hudiSink' [inlong]
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #9334:
URL: https://github.com/apache/inlong/pull/9334#discussion_r1405771175
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/HudiExtractNode.java:
##########
@@ -142,15 +143,17 @@ public Map<String, String> tableOptions() {
// If the extend attributes starts with .ddl,
// it will be passed to the ddl statement of the table
- extList.forEach(ext -> {
- String keyName = ext.get(EXTEND_ATTR_KEY_NAME);
- if (StringUtils.isNoneBlank(keyName) &&
- keyName.startsWith(DDL_ATTR_PREFIX)) {
- String ddlKeyName = keyName.substring(DDL_ATTR_PREFIX.length());
- String ddlValue = ext.get(EXTEND_ATTR_VALUE_NAME);
- options.put(ddlKeyName, ddlValue);
- }
- });
+ if (CollectionUtils.isNotEmpty(extList)) {
Review Comment:
1、the extList should add a check for npe, well done
2、as for primary key,
In hudi connector , the npe throws because we set string "null" for primary key but this "null" is not contained in hudi schema; that's the situation this pr #9323 trying to solve
Actually hudi can run without pk (in this case the table won't support upsert) and the hoodie.datasource.write.recordkey.field is optional;
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/HudiExtractNode.java:
##########
@@ -142,15 +143,17 @@ public Map<String, String> tableOptions() {
// If the extend attributes starts with .ddl,
// it will be passed to the ddl statement of the table
- extList.forEach(ext -> {
- String keyName = ext.get(EXTEND_ATTR_KEY_NAME);
- if (StringUtils.isNoneBlank(keyName) &&
- keyName.startsWith(DDL_ATTR_PREFIX)) {
- String ddlKeyName = keyName.substring(DDL_ATTR_PREFIX.length());
- String ddlValue = ext.get(EXTEND_ATTR_VALUE_NAME);
- options.put(ddlKeyName, ddlValue);
- }
- });
+ if (CollectionUtils.isNotEmpty(extList)) {
Review Comment:
1、the extList should add a check for npe, well done
2、as for primary key,
In hudi connector , the npe throws because we set string "null" for primary key but this "null" is not contained in hudi schema; that's the situation this pr #9323 trying to solve
Actually hudi can run without pk (in this case the table won't support upsert) and the hoodie.datasource.write.recordkey.field is optional;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-9333][Sort] Related problem with attribute exceptions when creating 'hudiSink' [inlong]
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #9334:
URL: https://github.com/apache/inlong/pull/9334#discussion_r1405771175
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/HudiExtractNode.java:
##########
@@ -142,15 +143,17 @@ public Map<String, String> tableOptions() {
// If the extend attributes starts with .ddl,
// it will be passed to the ddl statement of the table
- extList.forEach(ext -> {
- String keyName = ext.get(EXTEND_ATTR_KEY_NAME);
- if (StringUtils.isNoneBlank(keyName) &&
- keyName.startsWith(DDL_ATTR_PREFIX)) {
- String ddlKeyName = keyName.substring(DDL_ATTR_PREFIX.length());
- String ddlValue = ext.get(EXTEND_ATTR_VALUE_NAME);
- options.put(ddlKeyName, ddlValue);
- }
- });
+ if (CollectionUtils.isNotEmpty(extList)) {
Review Comment:
1、the extList should add a check for npe, well done
2、as for primary key,
In hudi connector , the npe throws because we set string "null" for primary key but this "null" is not contained in hudi schema; that's the situation this pr #9323 trying to solve
Actually hudi can run without pk (in this case hudi will generate a primary key itself) so in hudi official code, the hoodie.datasource.write.recordkey.field is optional;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-9333][Sort] Related problem with attribute exceptions when creating 'hudiSink' [inlong]
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang merged PR #9334:
URL: https://github.com/apache/inlong/pull/9334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-9333][Sort] Related problem with attribute exceptions when creating 'hudiSink' [inlong]
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #9334:
URL: https://github.com/apache/inlong/pull/9334#discussion_r1405771175
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/HudiExtractNode.java:
##########
@@ -142,15 +143,17 @@ public Map<String, String> tableOptions() {
// If the extend attributes starts with .ddl,
// it will be passed to the ddl statement of the table
- extList.forEach(ext -> {
- String keyName = ext.get(EXTEND_ATTR_KEY_NAME);
- if (StringUtils.isNoneBlank(keyName) &&
- keyName.startsWith(DDL_ATTR_PREFIX)) {
- String ddlKeyName = keyName.substring(DDL_ATTR_PREFIX.length());
- String ddlValue = ext.get(EXTEND_ATTR_VALUE_NAME);
- options.put(ddlKeyName, ddlValue);
- }
- });
+ if (CollectionUtils.isNotEmpty(extList)) {
Review Comment:
1、the extList should add a check for npe, well done
2、as for primary key,
In hudi connector , the npe throws because we set string "null" for primary key but this "null" is not contained in hudi schema; that's the situation this pr #9323 trying to solve
Actually hudi can run without setting a specific pk (in this case hudi will generate a primary key itself) so in hudi official code, the hoodie.datasource.write.recordkey.field is optional;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-9333][Sort] Related problem with attribute exceptions when creating 'hudiSink' [inlong]
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #9334:
URL: https://github.com/apache/inlong/pull/9334#discussion_r1405776513
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/HudiExtractNode.java:
##########
@@ -142,15 +143,17 @@ public Map<String, String> tableOptions() {
// If the extend attributes starts with .ddl,
// it will be passed to the ddl statement of the table
- extList.forEach(ext -> {
- String keyName = ext.get(EXTEND_ATTR_KEY_NAME);
- if (StringUtils.isNoneBlank(keyName) &&
- keyName.startsWith(DDL_ATTR_PREFIX)) {
- String ddlKeyName = keyName.substring(DDL_ATTR_PREFIX.length());
- String ddlValue = ext.get(EXTEND_ATTR_VALUE_NAME);
- options.put(ddlKeyName, ddlValue);
- }
- });
+ if (CollectionUtils.isNotEmpty(extList)) {
Review Comment:
see also in https://hudi.apache.org/docs/key_generation/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-9333][Sort] Related problem with attribute exceptions when creating 'hudiSink' [inlong]
Posted by "LiJie20190102 (via GitHub)" <gi...@apache.org>.
LiJie20190102 commented on code in PR #9334:
URL: https://github.com/apache/inlong/pull/9334#discussion_r1405781386
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/HudiExtractNode.java:
##########
@@ -142,15 +143,17 @@ public Map<String, String> tableOptions() {
// If the extend attributes starts with .ddl,
// it will be passed to the ddl statement of the table
- extList.forEach(ext -> {
- String keyName = ext.get(EXTEND_ATTR_KEY_NAME);
- if (StringUtils.isNoneBlank(keyName) &&
- keyName.startsWith(DDL_ATTR_PREFIX)) {
- String ddlKeyName = keyName.substring(DDL_ATTR_PREFIX.length());
- String ddlValue = ext.get(EXTEND_ATTR_VALUE_NAME);
- options.put(ddlKeyName, ddlValue);
- }
- });
+ if (CollectionUtils.isNotEmpty(extList)) {
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-9333][Sort] Related problem with attribute exceptions when creating 'hudiSink' [inlong]
Posted by "featzhang (via GitHub)" <gi...@apache.org>.
featzhang commented on code in PR #9334:
URL: https://github.com/apache/inlong/pull/9334#discussion_r1405562647
##########
inlong-sort/sort-common/src/main/java/org/apache/inlong/sort/protocol/node/extract/HudiExtractNode.java:
##########
@@ -142,15 +143,17 @@ public Map<String, String> tableOptions() {
// If the extend attributes starts with .ddl,
// it will be passed to the ddl statement of the table
- extList.forEach(ext -> {
- String keyName = ext.get(EXTEND_ATTR_KEY_NAME);
- if (StringUtils.isNoneBlank(keyName) &&
- keyName.startsWith(DDL_ATTR_PREFIX)) {
- String ddlKeyName = keyName.substring(DDL_ATTR_PREFIX.length());
- String ddlValue = ext.get(EXTEND_ATTR_VALUE_NAME);
- options.put(ddlKeyName, ddlValue);
- }
- });
+ if (CollectionUtils.isNotEmpty(extList)) {
Review Comment:
The primary key of hudi cannot be empty. Can an error be forced to be thrown?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org