You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by fhueske <gi...@git.apache.org> on 2015/10/09 17:38:43 UTC

[GitHub] flink pull request: [FLINK-2842] [documentation] Remove Flink S3Fi...

GitHub user fhueske opened a pull request:

    https://github.com/apache/flink/pull/1245

    [FLINK-2842] [documentation] Remove Flink S3FileSystem

    Extend documentation to use Hadoop S3FileSystem.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/fhueske/flink rmS3

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/1245.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1245
    
----
commit b754d5b05dd6fab45d0a665e04eb48aee02362f6
Author: Fabian Hueske <fh...@apache.org>
Date:   2015-10-09T15:35:00Z

    [FLINK-2842] [documentation] Remove Flink S3FileSystem, extend documentation to use Hadoop S3FileSystem.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2842] [documentation] Remove Flink S3Fi...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:

    https://github.com/apache/flink/pull/1245#issuecomment-149216856
  
    OK, will check if the dependency can be removed and merge it afterwards.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2842] [documentation] Remove Flink S3Fi...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/1245


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2842] [documentation] Remove Flink S3Fi...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:

    https://github.com/apache/flink/pull/1245#issuecomment-146934980
  
    Thanks for the pointer. I removed the AWS dependency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2842] [documentation] Remove Flink S3Fi...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/1245#issuecomment-146912958
  
    Looks good.
    
    Can you also remove the AWS tools dependency in the flink-runtime POM?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2842] [documentation] Remove Flink S3Fi...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:

    https://github.com/apache/flink/pull/1245#issuecomment-146914047
  
    sure, will do


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2842] [documentation] Remove Flink S3Fi...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:

    https://github.com/apache/flink/pull/1245#issuecomment-147009826
  
    Great that we finally get rid of this AWS dependency.
    It seems that the dependency actually influenced somehow other dependencies from Hadoop.
    
    I have a test in the YARN tests which verifies that the web interface is working. It seems that the change has broken that test now, so most likely the PR breaks the web interface.
    
    ```
      YARNSessionFIFOITCase.testTaskManagerFailure:229 expected:<[{"taskmanagers": 1, "slots": 1}]> but was:<[<html>
    <head>
    <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
    <title>Error 500 org.apache.http.client.utils.URLEncodedUtils.parse(Ljava/lang/String;Ljava/nio/charset/Charset;)Ljava/util/List;</title>
    </head>
    <body><h2>HTTP ERROR 500</h2>
    <p>Problem accessing /proxy/application_1444429019714_0007/jobsInfo. Reason:
    <pre>    org.apache.http.client.utils.URLEncodedUtils.parse(Ljava/lang/String;Ljava/nio/charset/Charset;)Ljava/util/List;</pre></p><h3>Caused by:</h3><pre>java.lang.NoSuchMethodError: org.apache.http.client.utils.URLEncodedUtils.parse(Ljava/lang/String;Ljava/nio/charset/Charset;)Ljava/util/List;
    	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.getUserName(PseudoAuthenticationHandler.java:124)
    	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:160)
    	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:361)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    	at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1223)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
    	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
    	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
    	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    	at org.mortbay.jetty.Server.handle(Server.java:326)
    	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
    	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
    	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
    	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
    	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
    </pre>
    <hr /><i><small>Powered by Jetty://</small></i><br/>  
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2842] [documentation] Remove Flink S3Fi...

Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the pull request:

    https://github.com/apache/flink/pull/1245#issuecomment-149203613
  
    The added dependency might not be necessary anymore for the new web interface (Jetty dependency was removed). Could you remove the dependency, rebase to the latest master, and push again to this pull request?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2842] [documentation] Remove Flink S3Fi...

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:

    https://github.com/apache/flink/pull/1245#issuecomment-149142412
  
    I checked the transitive dependencies of `aws-java-sdk` and found that it contains `org.apache.httpcomponents:httpclient:jar:4.2`. Adding this dependency to `flink-runtime` fixed the breaking YARN tests.
    
    Should be good to merge now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---