You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by wenlong88 <gi...@git.apache.org> on 2016/08/11 13:36:09 UTC
[GitHub] flink pull request #2356: [FLINK-4378]Enable RollingSink to custom HDFS clie...
GitHub user wenlong88 opened a pull request:
https://github.com/apache/flink/pull/2356
[FLINK-4378]Enable RollingSink to custom HDFS client configuration
Adding a new interface to rolling sink
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wenlong88/flink master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2356.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2356
----
commit 9b4dcff49aa8f1685ef3e38634f24b180ffcc3d5
Author: wenlong.lwl <we...@alibaba-inc.com>
Date: 2016-08-11T13:26:01Z
enable rolling sink to custom hdfs client configuration
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
sorry,@aljoscha, I have miss the last message in august, I will update the pr quite soon
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2356
@wenlong88 thanks for the update! What's the reason for setting the `fs.%s.impl.disable.cache` parameter explicitly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
@aljoscha i have rebase the newest master branch, and forward port the changes in rollingsink to bucketsink
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2356
@wenlong88 Are there any updates on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2356
Ah I see. I didn't see that before, sorry. I'll merge this once it passes on Travis.
Thanks for your work! \U0001f603
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
I have commented above, it is to make sure that the customized configuration work
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2356: [FLINK-4378]Enable RollingSink to custom HDFS clie...
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 closed the pull request at:
https://github.com/apache/flink/pull/2356
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2356
The changes look good! \U0001f44d Thanks for the work. We recently (today) merged an updated version of `RollingSink` called `BucketingSink`. Would you be willing to forward port your changes to that as well. If you don't want to I can also do it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
all of the test in travis have passed. thanks for merging this. ^_^
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
it is ok for me
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2356
I merged it. \U0001f44d Could you please close this PR?
I changed `setHDFSConfig()` to `setFSConfig` to reflect the fact that a `FileSystem` is not necessarily used only for HDFS, I hope that's ok.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2356: [FLINK-4378]Enable RollingSink to custom HDFS client conf...
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2356
the change is a little bigger than expected. because {{FileSystem}} will be cached once created ,to make sure the user defined configuration work, {{fs.%s.impl.disable.cache}} need to be set true and I cache the file system to avoid creating the filesystem duplicated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---