You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streams.apache.org by steveblackmon <gi...@git.apache.org> on 2014/09/16 20:45:40 UTC
[GitHub] incubator-streams pull request: added support for file: and hdfs: ...
GitHub user steveblackmon opened a pull request:
https://github.com/apache/incubator-streams/pull/85
added support for file: and hdfs: schemes
reader and writer are not internally consistent - removing timestamp read from wrong field for testing
"scheme": "file" works :)
will submit another PR for STREAMS-167 which will allow timestamps and metadata reading from a previously written hdfs stream.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-streams STREAMS-173
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-streams/pull/85.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #85
----
commit 6016a7740ae52b14ed15647efa530c365d789e8d
Author: sblackmon <sb...@apache.org>
Date: 2014-09-14T02:56:42Z
added support for file: and hdfs: schemes
reader and writer are not internally consistent - removing timestamp read from wrong field for testing
"scheme": "file" works :)
will submit another PR for STREAMS-167 which will allow timestamps and metadata reading from a previously written hdfs stream.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: added support for file: and hdfs: ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-streams/pull/85
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: added support for file: and hdfs: ...
Posted by renato2099 <gi...@git.apache.org>.
Github user renato2099 commented on the pull request:
https://github.com/apache/incubator-streams/pull/85#issuecomment-56731929
+1 to what Matt said, maybe creating separate JIRA issues so anybody could create the tests on the future?
Regarding the tests for the PR, why don't we use something like http://mrunit.apache.org/? That should be fairly straightforward.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: added support for file: and hdfs: ...
Posted by steveblackmon <gi...@git.apache.org>.
Github user steveblackmon commented on the pull request:
https://github.com/apache/incubator-streams/pull/85#issuecomment-56537869
before PR: reader can't process a file written by writer, only webhdfs: scheme supported
after PR: reader can process a file written by writer, hdfs: and file: schemes supported
pretty clear win. i'm not prepared to prioritize figuring out how to run hdfs and webhdfs in a maven test harness at this time however.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: added support for file: and hdfs: ...
Posted by rbnks <gi...@git.apache.org>.
Github user rbnks commented on the pull request:
https://github.com/apache/incubator-streams/pull/85#issuecomment-56196753
Comment there are not any unit tests created or modified in this pull request, please explain.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: added support for file: and hdfs: ...
Posted by mfranklin <gi...@git.apache.org>.
Github user mfranklin commented on the pull request:
https://github.com/apache/incubator-streams/pull/85#issuecomment-56711278
IMO, the community should be organizing code to isolate difficult to test dependencies and creating issues in the tracker to do proper integration testing across external systems.
In the case of this pull request, the getURI method should be testable without having to spin up any external dependencies on HDFS. If we can cover those types of methods with good tests and ensure HDFS is isolated as much as possible, it would meet the minimum Ryan proposed on the list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-streams pull request: added support for file: and hdfs: ...
Posted by mfranklin <gi...@git.apache.org>.
Github user mfranklin commented on the pull request:
https://github.com/apache/incubator-streams/pull/85#issuecomment-58191416
:+1: I see the added tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---