[jira] [Created] (SAMZA-171) Http Yarn package path failing

["Ethan Setnik (JIRA)" <ji...@apache.org>]

Ethan Setnik created SAMZA-171:
----------------------------------

             Summary: Http Yarn package path failing
                 Key: SAMZA-171
                 URL: https://issues.apache.org/jira/browse/SAMZA-171
             Project: Samza
          Issue Type: Bug
            Reporter: Ethan Setnik


When specifying an http package path for yarn jobs the jobs fail with the error:

14/03/05 16:28:40 WARN security.UserGroupInformation: PriviledgedActionException as:samza (auth:SIMPLE) cause:java.io.IOException: No FileSystem for scheme: http
14/03/05 16:28:40 INFO localizer.ResourceLocalizationService: DEBUG: FAILED { http://s3.amazonaws.com/samza_packages/wikipedia-job-package.tar.gz, 0, ARCHIVE, null }, No FileSystem for scheme: http
14/03/05 16:28:40 INFO localizer.LocalizedResource: Resource http://s3.amazonaws.com/samza_packages/wikipedia-job-package.tar.gz transitioned from DOWNLOADING to FAILED
14/03/05 16:28:40 INFO container.Container: Container container_1394035672475_0003_02_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED
14/03/05 16:28:40 INFO localizer.LocalResourcesTrackerImpl: Container container_1394035672475_0003_02_000001 sent RELEASE event on a resource request { http://s3.amazonaws.com/samza_packages/wikipedia-job-package.tar.gz, 0, ARCHIVE, null } not present in cache.

yarn.package.path=http://s3.amazonaws.com/samza_packages/wikipedia-job-package.tar.gz

It looks like some work has already been done to support this feature by configuring the "fs.http.imp".  I also noticed that this configuration was updated in SAMZA-63.

hConfig.set("fs.http.impl", classOf[HttpFileSystem].getName)

However my understanding is that the job package itself contains the necessary HttpFileSystem class to load http packages, and YARN does not support this configuration out of the box so i'm at a loss as to how to load a remote package over http.




--
This message was sent by Atlassian JIRA
(v6.2#6252)