You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/17 03:57:49 UTC
[GitHub] [pulsar] casuallc opened a new pull request #13373: Support rename the file has been processed.
casuallc opened a new pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373
### Motivation
Support rename the file has been processed.
#13301
### Modifications
Add property 'processedFileSuffix'.
If set, do not delete but only rename file that has been processed.
This config only work when 'keepFile' property is false.
### Verifying this change
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (no)
- The public API: (no)
- The schema: (no)
- The default values of configurations: (no)
- The wire protocol: (no)
- The rest endpoints: (no)
- The admin cli options: (no)
- Anything that affects deployment: (no)
### Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?
- [√] `no-need-doc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Haaroon commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
Haaroon commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-1004780004
Any updates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] casuallc commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
casuallc commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-1000203411
@codelipenghui ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-997538055
@casuallc When submitting a PR, please provide doc related info [in the PR description](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation) by ticking the box or labeling a PR directly, so that Bot will recognize the info and then label the PR correctly, or else Bot can not recognize the info and then label the PR with the `doc-info-missing` label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-1000140987
@casuallc It's a great feature for the connector, would you please send your proposal to the dev mail list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] casuallc commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
casuallc commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-1019075049
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Haaroon commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
Haaroon commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-997765037
Thanks for fixing the issue! I really appreciate your work. Renaming the file is a good idea as it keeps the operation stateless, just incase the file reader crashes and has to be restarted for some reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-996417074
@casuallc:Thanks for your contribution. For this PR, do we need to update docs?
(The [PR template contains info about doc](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation), which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet removed a comment on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
Anonymitaet removed a comment on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-997538055
@casuallc When submitting a PR, please provide doc related info [in the PR description](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation) by ticking the box or labeling a PR directly, so that Bot will recognize the info and then label the PR correctly, or else Bot can not recognize the info and then label the PR with the `doc-info-missing` label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-997538640
@casuallc I've labelled your PR w/ `doc`, please refer to the instructions of doc labels: https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0/edit#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] casuallc commented on a change in pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
casuallc commented on a change in pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#discussion_r778479137
##########
File path: pulsar-io/file/src/main/java/org/apache/pulsar/io/file/FileSourceConfig.java
##########
@@ -112,6 +112,12 @@
*/
private Integer numWorkers = 1;
+ /**
+ * If set, do not delete but only rename file that has been processed.
+ * This config only work when 'keepFile' property is false.
+ */
Review comment:
@lhotari Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-1018188191
@casuallc Please check the checkstyle issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Haaroon commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
Haaroon commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-1021321376
thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] casuallc commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
casuallc commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-998491078
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Haaroon commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
Haaroon commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-1021321376
thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#discussion_r778072767
##########
File path: pulsar-io/file/src/main/java/org/apache/pulsar/io/file/FileSourceConfig.java
##########
@@ -112,6 +112,12 @@
*/
private Integer numWorkers = 1;
+ /**
+ * If set, do not delete but only rename file that has been processed.
+ * This config only work when 'keepFile' property is false.
+ */
Review comment:
Please add validation for this rule in the validate method of this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#discussion_r784847157
##########
File path: pulsar-io/file/src/main/java/org/apache/pulsar/io/file/FileSourceConfig.java
##########
@@ -112,6 +112,12 @@
*/
private Integer numWorkers = 1;
+ /**
+ * If set, do not delete but only rename file that has been processed.
+ * This config only work when 'keepFile' property is false.
+ */
Review comment:
@casuallc there are some checkstyle issues:
```
Error: src/main/java/org/apache/pulsar/io/file/FileSourceConfig.java:[174] (sizes) LineLength: Line is longer than 120 characters (found 129).
Error: src/main/java/org/apache/pulsar/io/file/FileListingThread.java:[188] (sizes) LineLength: Line is longer than 120 characters (found 129).
Error: Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:check (checkstyle) on project pulsar-io-file: You have 2 Checkstyle violations. -> [Help 1]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-1013123245
@dlg99 please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] casuallc commented on pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
casuallc commented on pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#issuecomment-1017203259
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on a change in pull request #13373: Support rename the file has been processed.
Posted by GitBox <gi...@apache.org>.
lhotari commented on a change in pull request #13373:
URL: https://github.com/apache/pulsar/pull/13373#discussion_r778072767
##########
File path: pulsar-io/file/src/main/java/org/apache/pulsar/io/file/FileSourceConfig.java
##########
@@ -112,6 +112,12 @@
*/
private Integer numWorkers = 1;
+ /**
+ * If set, do not delete but only rename file that has been processed.
+ * This config only work when 'keepFile' property is false.
+ */
Review comment:
Please add validation for this rule (keepFile must be false if the value is set) in the validate method of this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org