You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/09/23 08:01:33 UTC
[GitHub] [inlong] GanfengTan opened a new pull request, #6008: [INLONG-5988][Agent] Add the Partition Key settings for file source
GanfengTan opened a new pull request, #6008:
URL: https://github.com/apache/inlong/pull/6008
Add the Partition Key settings for file source.
Synchronization mode the parameter needs to be set.
- Fixes #5988
### Motivation
No message
### Modifications
No message
### Verifying this change
*(Please pick either of the following options)*
- [x] This change is a trivial rework/code cleanup without any test coverage.
- [ ] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] pocozh commented on a diff in pull request #6008: [INLONG-5988][Agent] Add the Partition Key settings for file source
Posted by GitBox <gi...@apache.org>.
pocozh commented on code in PR #6008:
URL: https://github.com/apache/inlong/pull/6008#discussion_r978377497
##########
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/FileReaderOperator.java:
##########
@@ -90,7 +93,9 @@ public Message read() {
AuditUtils.add(AuditUtils.AUDIT_ID_AGENT_READ_SUCCESS,
inlongGroupId, inlongStreamId, System.currentTimeMillis());
readerMetric.pluginReadCount.incrementAndGet();
- return new DefaultMessage(message.getBytes(StandardCharsets.UTF_8));
+ Map<String, String> header = new HashMap<>();
+ header.put(PROXY_KEY_DATA, DigestUtils.md5Hex(inlongGroupId));
+ return new DefaultMessage(message.getBytes(StandardCharsets.UTF_8), header);
Review Comment:
Is it better to use inlongGroupId and inlongStreamId?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] GanfengTan commented on a diff in pull request #6008: [INLONG-5988][Agent] Add the Partition Key settings for file source
Posted by GitBox <gi...@apache.org>.
GanfengTan commented on code in PR #6008:
URL: https://github.com/apache/inlong/pull/6008#discussion_r979246966
##########
inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/file/FileReaderOperator.java:
##########
@@ -90,7 +93,9 @@ public Message read() {
AuditUtils.add(AuditUtils.AUDIT_ID_AGENT_READ_SUCCESS,
inlongGroupId, inlongStreamId, System.currentTimeMillis());
readerMetric.pluginReadCount.incrementAndGet();
- return new DefaultMessage(message.getBytes(StandardCharsets.UTF_8));
+ Map<String, String> header = new HashMap<>();
+ header.put(PROXY_KEY_DATA, DigestUtils.md5Hex(inlongGroupId));
+ return new DefaultMessage(message.getBytes(StandardCharsets.UTF_8), header);
Review Comment:
@pocozh 3qs your advice, I think inlongGroupId more better becuse the Group ID of the same task does not change. moreover added that users can configure Partition keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #6008: [INLONG-5988][Agent] Add the Partition Key settings for file source
Posted by GitBox <gi...@apache.org>.
dockerzhang merged PR #6008:
URL: https://github.com/apache/inlong/pull/6008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org