You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/08/06 12:56:41 UTC
[GitHub] [pulsar] BewareMyPower opened a new pull request #7771: [pulsar-io-hdfs2] Add config to create subdirectory from current time
BewareMyPower opened a new pull request #7771:
URL: https://github.com/apache/pulsar/pull/7771
### Motivation
Adding a subdirectory associated with current time willmake it easier to process HDFS files in batch.
For example, user can create multiple running sink instances with `yyyy-MM-dd-hh` pattern. Then stop all instances at next hour. Eventually, files of the subdirectory will contain all messages consumed during this hour.
### Modifications
- Add a `subdirectoryPattern` field to `HdfsSinkConfig`
- Update some simple tests for `HdfsSinkConfig`
- Update the doc of HDFS2 sink
### Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie merged pull request #7771: [pulsar-io-hdfs2] Add config to create subdirectory from current time
Posted by GitBox <gi...@apache.org>.
sijie merged pull request #7771:
URL: https://github.com/apache/pulsar/pull/7771
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #7771: [pulsar-io-hdfs2] Add config to create subdirectory from current time
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on pull request #7771:
URL: https://github.com/apache/pulsar/pull/7771#issuecomment-671061945
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #7771: [pulsar-io-hdfs2] Add config to create subdirectory from current time
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on pull request #7771:
URL: https://github.com/apache/pulsar/pull/7771#issuecomment-670621918
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #7771: [pulsar-io-hdfs2] Add config to create subdirectory from current time
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on pull request #7771:
URL: https://github.com/apache/pulsar/pull/7771#issuecomment-670943890
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org