Issue with loading dependencies and jars

[Shane Johnson <sh...@liftiq.com>]

Hi team,

I have been working with Mars to get a little more insight to an error I am
seeing. After narrowing down the issue, I am trying to find a way to force
the order of loading the jars. I am a novice in Scala and am looking for a
way to continue leveraging PredictionIO on Heroku but have run into a
roadblock as I believe ordering the jars would be a change to PredictionIO
itself. I would greatly appreciate your expertise and guidance.

Error:

Exception

 in thread "main" java.lang.NoSuchMethodError:
com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V

2018-03-02T02:49:42.245083+00:00 app[release.7253]: 	at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)

2018-03-02T02:49:42.245086+00:00 app[release.7253]: 	at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

2018-03-02T02:49:42.245087+00:00 app[release.7253]: 	at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

2018-03-02T02:49:42.245088+00:00 app[release.7253]: 	at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

2018-03-02T02:49:42.245090+00:00 app[release.7253]: 	at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)


Context:

   - I am using the predictionIO buildpack on heroku and using version 12.
   - I am using S3 to Persist the model objects as I am using the
   PAlgorithm vs L2LAlgorithm.
   - I see an intermittent error when the Algorithm.scala file tries to
   connect to S3. Sometimes it works and sometime is does not.
   - When it fails I get the message above - I think I have narrowed this
   down to the ordering of loading the jars.

*I have done a number of tests and can see that the problem always occurs
with a certain order of jars. This issue appears to be intermittent and
could be solved by forcing the order of the jars. I am assuming this could
be a straightforward fix, *
*I believe it is in the compute-classpath.sh. Perhaps it's something I can
even do on my end as an end user but I don't believe that to be the case.*

*Please let me know if this is something that can be changed and committed
or if I can contribute in someway. Again, I am fairly new to Scala and big
data infrastructure so I am intimidated by the thought of making the change
myself but we are at a roadblock:)*

*When aws-java-sdk.jar is loaded before (hadoop-aws.jar
and pio-data-s3-assembly-0.12.0-incubating.jar) it is always successful. If
it is loaded after it fails with the message above.*

*Success Examples:*

--jars
  file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
  file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,

file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar,
  ...

--jars
  file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,

file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar,
  file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
  ...

*Failure Examples:*

--jars
  file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,

file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar,
  file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
  ...

--jars

file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar,
  file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
  file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
  ...


Thank you again for your help!

*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>