You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@predictionio.apache.org by Shane Johnson <sh...@gmail.com> on 2018/03/05 14:57:50 UTC

Issue with loading dependencies and jars

Hi team,

I have been working with Mars to get a little more insight to an error I am
seeing. After narrowing down the issue, I am trying to find a way to force
the order of loading the jars. I am a novice in Scala and am looking for a
way to continue leveraging PredictionIO on Heroku but have run into a
roadblock as I believe ordering the jars would be a change to PredictionIO
itself. I would greatly appreciate your expertise and guidance.

Error:

Exception

 in thread "main" java.lang.NoSuchMethodError:
com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V

2018-03-02T02:49:42.245083+00:00 app[release.7253]: 	at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)

2018-03-02T02:49:42.245086+00:00 app[release.7253]: 	at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

2018-03-02T02:49:42.245087+00:00 app[release.7253]: 	at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

2018-03-02T02:49:42.245088+00:00 app[release.7253]: 	at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

2018-03-02T02:49:42.245090+00:00 app[release.7253]: 	at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)


Context:

   - I am using the predictionIO buildpack on heroku and using version 12.
   - I am using S3 to Persist the model objects as I am using the
   PAlgorithm vs L2LAlgorithm.
   - I see an intermittent error when the Algorithm.scala file tries to
   connect to S3. Sometimes it works and sometime is does not.
   - When it fails I get the message above - I think I have narrowed this
   down to the ordering of loading the jars.

*I have done a number of tests and can see that the problem always occurs
with a certain order of jars. This issue appears to be intermittent and
could be solved by forcing the order of the jars. I am assuming this could
be a straightforward fix, *
*I believe it is in the compute-classpath.sh. Perhaps it's something I can
even do on my end as an end user but I don't believe that to be the case.*

*Please let me know if this is something that can be changed and committed
or if I can contribute in someway. Again, I am fairly new to Scala and big
data infrastructure so I am intimidated by the thought of making the change
myself but we are at a roadblock:)*

*When aws-java-sdk.jar is loaded before
(hadoop-aws.jar and pio-data-s3-assembly-0.12.0-incubating.jar) it is
always successful. If it is loaded after it fails with the message above.*

*Success Examples:*

--jars
  file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
  file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
  file:/app/PredictionIO-dist/lib/spark/pio-data-s3-
assembly-0.12.0-incubating.jar,
  ...

--jars
  file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
  file:/app/PredictionIO-dist/lib/spark/pio-data-s3-
assembly-0.12.0-incubating.jar,
  file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
  ...

*Failure Examples:*

--jars
  file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
  file:/app/PredictionIO-dist/lib/spark/pio-data-s3-
assembly-0.12.0-incubating.jar,
  file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
  ...

--jars
  file:/app/PredictionIO-dist/lib/spark/pio-data-s3-
assembly-0.12.0-incubating.jar,
  file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
  file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
  ...


Thank you again for your help!

*Shane Johnson | 801.360.3350*
LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook
<https://www.facebook.com/shane.johnson.71653>

Re: Issue with loading dependencies and jars

Posted by Donald Szeto <do...@apache.org>.

Hi Shane,

Although not highly recommended, would you mind trying to set
MYSQL_JDBC_DRIVER in your conf/pio-env.sh to point to the aws-java-sdk.jar
and try running again? In code, third party JARs for MySQL and PostgreSQL
will always be appended at the very front for the "spark-submit --jars"
argument.

If that works for you, we should file a new feature request for this.

Regards,
Donald

On Sun, Mar 11, 2018 at 11:03 AM, Mars Hall <ma...@salesforce.com>
wrote:

> On Sat, Mar 10, 2018 at 7:49 PM, Shane Johnson <sh...@liftiq.com> wrote:
>
>> Mars, I was reviewing the code that you are referencing, the "jars for
>> Spark" function, this morning and trying to see how it ties in. This code
>> that is outside the custom binary distribution correct, I could not find it
>> in the distribution that is being used by the buildpack?
>>
>
> The PredictionIO binary distribution contains only the "compiled" code,
> its assembly jars. You'll need to dig into PredictionIO's source code (on
> Github) to find that function.
>
>
>
>> Do you think the ordering of jars may need to happen in the Common.scala
>> file instead of the compute-classpath.sh?
>>
>
> Yes, to effectively revise Common.scala, changes must be made to the
> source code (e.g. checked-out from Github develop branch) and then built
> into new custom binary distribution via `make-distribution.sh` script:
>
>   http://predictionio.apache.org/community/contribute-code/
> #getting-started
>
>
> --
> *Mars Hall
> 415-818-7039 <(415)%20818-7039>
> Customer Facing Architect
> Salesforce Platform / Heroku
> San Francisco, California
>

Re: Issue with loading dependencies and jars

Posted by Mars Hall <ma...@salesforce.com>.

On Sat, Mar 10, 2018 at 7:49 PM, Shane Johnson <sh...@liftiq.com> wrote:

> Mars, I was reviewing the code that you are referencing, the "jars for
> Spark" function, this morning and trying to see how it ties in. This code
> that is outside the custom binary distribution correct, I could not find it
> in the distribution that is being used by the buildpack?
>

The PredictionIO binary distribution contains only the "compiled" code, its
assembly jars. You'll need to dig into PredictionIO's source code (on
Github) to find that function.

> Do you think the ordering of jars may need to happen in the Common.scala
> file instead of the compute-classpath.sh?
>

Yes, to effectively revise Common.scala, changes must be made to the source
code (e.g. checked-out from Github develop branch) and then built into new
custom binary distribution via `make-distribution.sh` script:

  http://predictionio.apache.org/community/contribute-code/#getting-started

-- 
*Mars Hall
415-818-7039
Customer Facing Architect
Salesforce Platform / Heroku
San Francisco, California

Re: Issue with loading dependencies and jars

Posted by Shane Johnson <sh...@liftiq.com>.

Thanks for looking into this with me Mars and Donald. It's nice to have
experts, we are not completely stuck as our models train successfully ~50%
of the time but we are looking forward to finding a way to order the jars
and stabilize our processes.

Mars, I was reviewing the code that you are referencing, the "jars for
Spark" function, this morning and trying to see how it ties in. This code
that is outside the custom binary distribution correct, I could not find it
in the distribution that is being used by the buildpack?

Do you think the ordering of jars may need to happen in the Common.scala
file instead of the compute-classpath.sh?

Best,

Shane

*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>

On Fri, Mar 9, 2018 at 6:59 PM, Mars Hall <ma...@salesforce.com> wrote:

> Correction, it’s this “jars for Spark” function:
> https://github.com/apache/predictionio/blob/develop/
> tools/src/main/scala/org/apache/predictionio/tools/Common.scala#L105
>
> On Fri, Mar 9, 2018 at 17:54 Mars Hall <ma...@salesforce.com> wrote:
>
>> It looks like this Scala function is the source of that jars list:
>> https://github.com/apache/predictionio/blob/develop/
>> tools/src/main/scala/org/apache/predictionio/tools/Common.scala#L81
>>
>> On Fri, Mar 9, 2018 at 17:42 Mars Hall <ma...@salesforce.com> wrote:
>>
>>> Where does the classpath in spark-submit originate? Is
>>> compute-classpath.sh not the source?
>>>
>>>>
>>>> --
> *Mars Hall
> 415-818-7039 <(415)%20818-7039>
> Customer Facing Architect
> Salesforce Platform / Heroku
> San Francisco, California
>

Re: Issue with loading dependencies and jars

Posted by Mars Hall <ma...@salesforce.com>.

Correction, it’s this “jars for Spark” function:
https://github.com/apache/predictionio/blob/develop/tools/src/main/scala/org/apache/predictionio/tools/Common.scala#L105

On Fri, Mar 9, 2018 at 17:54 Mars Hall <ma...@salesforce.com> wrote:

> It looks like this Scala function is the source of that jars list:
>
> https://github.com/apache/predictionio/blob/develop/tools/src/main/scala/org/apache/predictionio/tools/Common.scala#L81
>
> On Fri, Mar 9, 2018 at 17:42 Mars Hall <ma...@salesforce.com> wrote:
>
>> Where does the classpath in spark-submit originate? Is
>> compute-classpath.sh not the source?
>>
>>>
>>> --
*Mars Hall
415-818-7039
Customer Facing Architect
Salesforce Platform / Heroku
San Francisco, California

Re: Issue with loading dependencies and jars

Posted by Mars Hall <ma...@salesforce.com>.

It looks like this Scala function is the source of that jars list:
https://github.com/apache/predictionio/blob/develop/tools/src/main/scala/org/apache/predictionio/tools/Common.scala#L81

On Fri, Mar 9, 2018 at 17:42 Mars Hall <ma...@salesforce.com> wrote:

> Where does the classpath in spark-submit originate? Is
> compute-classpath.sh not the source?
>
> As noted previously, the stable-ordering fix by me in compute-classpath.sh
> no longer seems to be effective either.
>
> Looks like some tracing of classpath assembly through the Spark command
> runner is required:
> https://github.com/apache/predictionio/blob/develop/tools/src/main/scala/org/apache/predictionio/tools/Runner.scala#L185
>
> Unless someone with more knowledge of these internals could weigh-in…
> Donald? 😬😊
>
> On Fri, Mar 9, 2018 at 15:44 Shane Johnson <sh...@liftiq.com> wrote:
>
>> One additional item that you mentioned earlier is that we would need to
>> remove or skip the aws-java-sdk.jar that is already in the CLASSPATH. Do
>> you think this has impact? I did not write anything to skip or remove the
>> existing aws-java-sdk.jar.
>>
>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script will
>>> need to skip or remove it first.
>>
>>
>> *Shane Johnson | LIFT IQ*
>> *Founder | CEO*
>>
>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>> <sh...@liftiq.com>*
>> mobile: (801) 360-3350
>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>> <https://twitter.com/SWaldenJ> |  Facebook
>> <https://www.facebook.com/shane.johnson.71653>
>>
>>
>>
>> On Fri, Mar 9, 2018 at 4:41 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>
>>> Now that I am able to deploy I reset the buildpack to
>>> ...#debug-custom-dist and redeployed. Here is the build log...URL does
>>> point to the correct distribution with the edited compute-classpath.sh file.
>>>
>>> -----> JVM Common app detected
>>>
>>> -----> Installing JDK 1.8... done
>>>
>>> -----> PredictionIO app detected
>>>
>>> -----> Install core components
>>>
>>>        + PredictionIO (https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz)
>>>
>>>        + Spark (spark-2.1.1-bin-hadoop2.7)
>>>
>>> -----> Install supplemental components
>>>
>>>        + PostgreSQL (JDBC)
>>>
>>>        + S3 HDFS (AWS SDK)
>>>
>>>        + S3 HDFS (Hadoop-AWS)
>>>
>>>          Writing default 'core-site.xml.erb'
>>>
>>>        + local Maven repo from buildpack (contents)
>>>
>>> -----> Configure PredictionIO
>>>
>>>        Writing default 'pio-env.sh'
>>>
>>>        Writing default 'spark-defaults.conf.erb'
>>>
>>>        + Maven repo from buildpack (build.sbt entry)
>>>
>>>        Set-up environment via '.profile.d/' scripts
>>>
>>> -----> Install JVM (heroku/jvm-common)
>>>
>>> -----> PredictionIO engine
>>>
>>>        Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>>>
>>>        [INFO] [Engine$] Using command '/tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0/PredictionIO-dist/sbt/sbt' at /tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0 to build.
>>>
>>>        [INFO] [Engine$] If the path above is incorrect, this process will fail.
>>>
>>>        [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent.
>>>
>>>        [INFO] [Engine$] Going to run: /tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency in /tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0
>>>
>>>        [INFO] [Engine$] Compilation finished successfully.
>>>
>>>        [INFO] [Engine$] Looking for an engine...
>>>
>>>        [INFO] [Engine$] Found template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar
>>>
>>>        [INFO] [Engine$] Found template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar
>>>
>>>        [INFO] [Engine$] Build finished successfully.
>>>
>>>        [INFO] [Pio$] Your engine is ready for training.
>>>
>>>        Using default Procfile for engine
>>>
>>> -----> Discovering process types
>>>
>>>        Procfile declares types -> release, train, web
>>>
>>> -----> Compressing...
>>>
>>>        Done: 376.7M
>>>
>>>
>>> The release log is below...I am not seeing the  */app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>> *show up at the beginning of the CLASSPATH, this is what we should see
>>> correct? I was also manipulating the compute-classpath.sh locally as well,
>>> I observed that adding a line right before echo "$CLASSPATH" was not
>>> changing what was in the logged spark-submit command as an FYI. This is
>>> what I was testing locally...
>>>
>>>
>>> CLASSPATH="*/Users/shanejohnson/Desktop/Apps/liftiq_platform/lift-s*
>>> *core/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*
>>> :$CLASSPATH"
>>> echo "$CLASSPATH"
>>>
>>> I did not see any change in the spark-submit command by adding this
>>> when building and deploying locally.
>>>
>>> Release Log with new buildpack ..#debug-custom-dist
>>>
>>> Running train on release…
>>>
>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>
>>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-hbase-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-s3-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-hdfs-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-jdbc-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env PIO_ENV_LOADED=1,PIO_EVENTSERVER_APP_NAME=classi,PIO_STORAGE_SOURCES_PGSQL_INDEX=enabled,PIO_S3_AWS_ACCESS_KEY_ID=AKIAJJX2S55QPCPZXGFQ,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/app/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost,PIO_S3_BUCKET_NAME=lift-model-devmaster,PIO_EVENTSERVER_ACCESS_KEY=5954-20848-7512-17427-21660,PIO_HOME=/app/PredictionIO-dist,PIO_FS_ENGINESDIR=/app/.pio_store/engines,PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://ec2-52-70-46-243.compute-1.amazonaws.com:5432/dbvbo86hohutvb?sslmode=require,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=PGSQL,PIO_SPARK_OPTS=--driver-memory 13g ,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=PGSQL,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_PGSQL_PASSWORD=p5c404ac780ab517d4ab249d7000809b51b4b987fdfb5c26e1bace511130337ac,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/app/PredictionIO-dist/vendors/elasticsearch,PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc,PIO_FS_TMPDIR=/app/.pio_store/tmp,PIO_STORAGE_SOURCES_PGSQL_USERNAME=ubefhv668b1s1m,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http,PIO_S3_AWS_SECRET_ACCESS_KEY=tQwL1PgYR0Y5MHG+qwVgEXNEcDcdlupaN2oO6JuR,PIO_TRAIN_SPARK_OPTS=--driver-memory 13g ,PIO_STORAGE_SOURCES_PGSQL_CONNECTIONS=8,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=PGSQL,PIO_CONF_DIR=/app/PredictionIO-dist/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200,PIO_STORAGE_SOURCES_PGSQL_PARTITIONS=4,PIO_S3_AWS_REGION=us-east-1
>>>
>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>
>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>
>>> [INFO] [Engine] Extracting datasource params...
>>>
>>> [INFO] [Engine] Datasource params: (,DataSourceParams(Some(5)))
>>>
>>> [INFO] [Engine] Extracting preparator params...
>>>
>>> [WARN] [WorkflowUtils$] Non-empty parameters supplied to org.template.liftscoring.Preparator, but its constructor does not accept any arguments. Stubbing with empty parameters.
>>>
>>> [INFO] [Engine] Preparator params: (,Empty)
>>>
>>>
>>>
>>>
>>>
>>> *Shane Johnson | LIFT IQ*
>>> *Founder | CEO*
>>>
>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>> <sh...@liftiq.com>*
>>> mobile: (801) 360-3350
>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>> <https://twitter.com/SWaldenJ> |  Facebook
>>> <https://www.facebook.com/shane.johnson.71653>
>>>
>>>
>>>
>>> On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <ma...@salesforce.com>
>>> wrote:
>>>
>>>> I'm lost as to how such direct manipulation of CLASSPATH is not
>>>> appearing in the logged spark-submit command.
>>>>
>>>> What could cause this!?
>>>>
>>>> I just pushed a version of the buildpack which should help debug.
>>>> Assuming only a single buildpack is assigned to the app, here's how to set
>>>> it:
>>>>
>>>>   heroku buildpacks:set
>>>> https://github.com/heroku/predictionio-buildpack#debug-custom-dist
>>>>
>>>> Then redeploy the engine an check the build log for the line:
>>>>
>>>>       + PredictionIO ($URL)
>>>>
>>>> Please confirm that it is the URL of your custom PredictionIO dist.
>>>>
>>>> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>>>
>>>>> Thanks Donald and Mars,
>>>>>
>>>>> I created a new distribution (
>>>>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
>>>>> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz)
>>>>> with the added CLASSPATH code and pointed to the distribution with
>>>>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
>>>>>
>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>> :$CLASSPATH"
>>>>> echo "$CLASSPATH"
>>>>>
>>>>> It didn't seem to force the aws-java-sdk to load first as I reviewed
>>>>> the release logs. Should the aws-java-sdk.jar show up as the first file
>>>>> within the --jars section when this is added CLASSPATH="
>>>>> /app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH".
>>>>>
>>>>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
>>>>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have
>>>>> other suggestions to try? I was also testing locally to change the order of
>>>>> the --jars but changes to the compute-classpath.sh didn't seem to change
>>>>> the order of the jars in the logs.
>>>>>
>>>>> Running train on release…
>>>>>
>>>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>>>
>>>>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env
>>>>>
>>>>>
>>>>> Error:
>>>>>
>>>>> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>>>>
>>>>> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>>>>
>>>>> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>>>>
>>>>> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Shane Johnson | LIFT IQ*
>>>>> *Founder | CEO*
>>>>>
>>>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>>>> <sh...@liftiq.com>*
>>>>> mobile: (801) 360-3350
>>>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>>>> <https://twitter.com/SWaldenJ> |  Facebook
>>>>> <https://www.facebook.com/shane.johnson.71653>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com>
>>>>> wrote:
>>>>>
>>>>>> Shane,
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Re: adding a line to ensure a jar is loaded first. Is this what you
>>>>>>> are referring to...(line at the bottom in red)?
>>>>>>>
>>>>>>
>>>>>>
>>>>>> I believe the code would need to look like this to effect the output
>>>>>> classpath as intended:
>>>>>>
>>>>>>
>>>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>>>> :$CLASSPATH"
>>>>>>> echo "$CLASSPATH"
>>>>>>>
>>>>>>
>>>>>>
>>>>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script
>>>>>> will need to be skip or remove it first.
>>>>>>
>>>>>> --
>>>>>> *Mars Hall
>>>>>> 415-818-7039 <(415)%20818-7039>
>>>>>> Customer Facing Architect
>>>>>> Salesforce Platform / Heroku
>>>>>> San Francisco, California
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Mars Hall
>>>> 415-818-7039 <(415)%20818-7039>
>>>> Customer Facing Architect
>>>> Salesforce Platform / Heroku
>>>> San Francisco, California
>>>>
>>>
>>>
>> --
> *Mars Hall
> 415-818-7039
> Customer Facing Architect
> Salesforce Platform / Heroku
> San Francisco, California
>
-- 
*Mars Hall
415-818-7039
Customer Facing Architect
Salesforce Platform / Heroku
San Francisco, California

Re: Issue with loading dependencies and jars

Posted by Mars Hall <ma...@salesforce.com>.

Where does the classpath in spark-submit originate? Is compute-classpath.sh
not the source?

As noted previously, the stable-ordering fix by me in compute-classpath.sh
no longer seems to be effective either.

Looks like some tracing of classpath assembly through the Spark command
runner is required:
https://github.com/apache/predictionio/blob/develop/tools/src/main/scala/org/apache/predictionio/tools/Runner.scala#L185

Unless someone with more knowledge of these internals could weigh-in…
Donald? 😬😊

On Fri, Mar 9, 2018 at 15:44 Shane Johnson <sh...@liftiq.com> wrote:

> One additional item that you mentioned earlier is that we would need to
> remove or skip the aws-java-sdk.jar that is already in the CLASSPATH. Do
> you think this has impact? I did not write anything to skip or remove the
> existing aws-java-sdk.jar.
>
> aws-java-sdk.jar is already in the CLASSPATH though, So, the script will
>> need to skip or remove it first.
>
>
> *Shane Johnson | LIFT IQ*
> *Founder | CEO*
>
> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
> <sh...@liftiq.com>*
> mobile: (801) 360-3350
> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
> <https://twitter.com/SWaldenJ> |  Facebook
> <https://www.facebook.com/shane.johnson.71653>
>
>
>
> On Fri, Mar 9, 2018 at 4:41 PM, Shane Johnson <sh...@liftiq.com> wrote:
>
>> Now that I am able to deploy I reset the buildpack to
>> ...#debug-custom-dist and redeployed. Here is the build log...URL does
>> point to the correct distribution with the edited compute-classpath.sh file.
>>
>> -----> JVM Common app detected
>>
>> -----> Installing JDK 1.8... done
>>
>> -----> PredictionIO app detected
>>
>> -----> Install core components
>>
>>        + PredictionIO (https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz)
>>
>>        + Spark (spark-2.1.1-bin-hadoop2.7)
>>
>> -----> Install supplemental components
>>
>>        + PostgreSQL (JDBC)
>>
>>        + S3 HDFS (AWS SDK)
>>
>>        + S3 HDFS (Hadoop-AWS)
>>
>>          Writing default 'core-site.xml.erb'
>>
>>        + local Maven repo from buildpack (contents)
>>
>> -----> Configure PredictionIO
>>
>>        Writing default 'pio-env.sh'
>>
>>        Writing default 'spark-defaults.conf.erb'
>>
>>        + Maven repo from buildpack (build.sbt entry)
>>
>>        Set-up environment via '.profile.d/' scripts
>>
>> -----> Install JVM (heroku/jvm-common)
>>
>> -----> PredictionIO engine
>>
>>        Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>>
>>        [INFO] [Engine$] Using command '/tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0/PredictionIO-dist/sbt/sbt' at /tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0 to build.
>>
>>        [INFO] [Engine$] If the path above is incorrect, this process will fail.
>>
>>        [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent.
>>
>>        [INFO] [Engine$] Going to run: /tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency in /tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0
>>
>>        [INFO] [Engine$] Compilation finished successfully.
>>
>>        [INFO] [Engine$] Looking for an engine...
>>
>>        [INFO] [Engine$] Found template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar
>>
>>        [INFO] [Engine$] Found template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar
>>
>>        [INFO] [Engine$] Build finished successfully.
>>
>>        [INFO] [Pio$] Your engine is ready for training.
>>
>>        Using default Procfile for engine
>>
>> -----> Discovering process types
>>
>>        Procfile declares types -> release, train, web
>>
>> -----> Compressing...
>>
>>        Done: 376.7M
>>
>>
>> The release log is below...I am not seeing the  */app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>> *show up at the beginning of the CLASSPATH, this is what we should see
>> correct? I was also manipulating the compute-classpath.sh locally as well,
>> I observed that adding a line right before echo "$CLASSPATH" was not
>> changing what was in the logged spark-submit command as an FYI. This is
>> what I was testing locally...
>>
>>
>> CLASSPATH="*/Users/shanejohnson/Desktop/Apps/liftiq_platform/lift-s*
>> *core/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*
>> :$CLASSPATH"
>> echo "$CLASSPATH"
>>
>> I did not see any change in the spark-submit command by adding this when
>> building and deploying locally.
>>
>> Release Log with new buildpack ..#debug-custom-dist
>>
>> Running train on release…
>>
>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>
>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-hbase-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-s3-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-hdfs-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-jdbc-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env PIO_ENV_LOADED=1,PIO_EVENTSERVER_APP_NAME=classi,PIO_STORAGE_SOURCES_PGSQL_INDEX=enabled,PIO_S3_AWS_ACCESS_KEY_ID=AKIAJJX2S55QPCPZXGFQ,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/app/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost,PIO_S3_BUCKET_NAME=lift-model-devmaster,PIO_EVENTSERVER_ACCESS_KEY=5954-20848-7512-17427-21660,PIO_HOME=/app/PredictionIO-dist,PIO_FS_ENGINESDIR=/app/.pio_store/engines,PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://ec2-52-70-46-243.compute-1.amazonaws.com:5432/dbvbo86hohutvb?sslmode=require,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=PGSQL,PIO_SPARK_OPTS=--driver-memory 13g ,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=PGSQL,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_PGSQL_PASSWORD=p5c404ac780ab517d4ab249d7000809b51b4b987fdfb5c26e1bace511130337ac,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/app/PredictionIO-dist/vendors/elasticsearch,PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc,PIO_FS_TMPDIR=/app/.pio_store/tmp,PIO_STORAGE_SOURCES_PGSQL_USERNAME=ubefhv668b1s1m,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http,PIO_S3_AWS_SECRET_ACCESS_KEY=tQwL1PgYR0Y5MHG+qwVgEXNEcDcdlupaN2oO6JuR,PIO_TRAIN_SPARK_OPTS=--driver-memory 13g ,PIO_STORAGE_SOURCES_PGSQL_CONNECTIONS=8,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=PGSQL,PIO_CONF_DIR=/app/PredictionIO-dist/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200,PIO_STORAGE_SOURCES_PGSQL_PARTITIONS=4,PIO_S3_AWS_REGION=us-east-1
>>
>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>
>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>
>> [INFO] [Engine] Extracting datasource params...
>>
>> [INFO] [Engine] Datasource params: (,DataSourceParams(Some(5)))
>>
>> [INFO] [Engine] Extracting preparator params...
>>
>> [WARN] [WorkflowUtils$] Non-empty parameters supplied to org.template.liftscoring.Preparator, but its constructor does not accept any arguments. Stubbing with empty parameters.
>>
>> [INFO] [Engine] Preparator params: (,Empty)
>>
>>
>>
>>
>>
>> *Shane Johnson | LIFT IQ*
>> *Founder | CEO*
>>
>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>> <sh...@liftiq.com>*
>> mobile: (801) 360-3350
>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>> <https://twitter.com/SWaldenJ> |  Facebook
>> <https://www.facebook.com/shane.johnson.71653>
>>
>>
>>
>> On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <ma...@salesforce.com>
>> wrote:
>>
>>> I'm lost as to how such direct manipulation of CLASSPATH is not
>>> appearing in the logged spark-submit command.
>>>
>>> What could cause this!?
>>>
>>> I just pushed a version of the buildpack which should help debug.
>>> Assuming only a single buildpack is assigned to the app, here's how to set
>>> it:
>>>
>>>   heroku buildpacks:set
>>> https://github.com/heroku/predictionio-buildpack#debug-custom-dist
>>>
>>> Then redeploy the engine an check the build log for the line:
>>>
>>>       + PredictionIO ($URL)
>>>
>>> Please confirm that it is the URL of your custom PredictionIO dist.
>>>
>>> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>>
>>>> Thanks Donald and Mars,
>>>>
>>>> I created a new distribution (
>>>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
>>>> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz)
>>>> with the added CLASSPATH code and pointed to the distribution with
>>>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
>>>>
>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>> :$CLASSPATH"
>>>> echo "$CLASSPATH"
>>>>
>>>> It didn't seem to force the aws-java-sdk to load first as I reviewed
>>>> the release logs. Should the aws-java-sdk.jar show up as the first file
>>>> within the --jars section when this is added CLASSPATH="
>>>> /app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH".
>>>>
>>>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
>>>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have
>>>> other suggestions to try? I was also testing locally to change the order of
>>>> the --jars but changes to the compute-classpath.sh didn't seem to change
>>>> the order of the jars in the logs.
>>>>
>>>> Running train on release…
>>>>
>>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>>
>>>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env
>>>>
>>>>
>>>> Error:
>>>>
>>>> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>>>
>>>> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>>>
>>>> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>>>
>>>> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Shane Johnson | LIFT IQ*
>>>> *Founder | CEO*
>>>>
>>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>>> <sh...@liftiq.com>*
>>>> mobile: (801) 360-3350
>>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>>> <https://twitter.com/SWaldenJ> |  Facebook
>>>> <https://www.facebook.com/shane.johnson.71653>
>>>>
>>>>
>>>>
>>>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com>
>>>> wrote:
>>>>
>>>>> Shane,
>>>>>
>>>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Re: adding a line to ensure a jar is loaded first. Is this what you
>>>>>> are referring to...(line at the bottom in red)?
>>>>>>
>>>>>
>>>>>
>>>>> I believe the code would need to look like this to effect the output
>>>>> classpath as intended:
>>>>>
>>>>>
>>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>>> :$CLASSPATH"
>>>>>> echo "$CLASSPATH"
>>>>>>
>>>>>
>>>>>
>>>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script
>>>>> will need to be skip or remove it first.
>>>>>
>>>>> --
>>>>> *Mars Hall
>>>>> 415-818-7039 <(415)%20818-7039>
>>>>> Customer Facing Architect
>>>>> Salesforce Platform / Heroku
>>>>> San Francisco, California
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Mars Hall
>>> 415-818-7039 <(415)%20818-7039>
>>> Customer Facing Architect
>>> Salesforce Platform / Heroku
>>> San Francisco, California
>>>
>>
>>
> --
*Mars Hall
415-818-7039
Customer Facing Architect
Salesforce Platform / Heroku
San Francisco, California

Re: Issue with loading dependencies and jars

Posted by Shane Johnson <sh...@liftiq.com>.

One additional item that you mentioned earlier is that we would need to
remove or skip the aws-java-sdk.jar that is already in the CLASSPATH. Do
you think this has impact? I did not write anything to skip or remove the
existing aws-java-sdk.jar.

aws-java-sdk.jar is already in the CLASSPATH though, So, the script will
> need to skip or remove it first.


*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>



On Fri, Mar 9, 2018 at 4:41 PM, Shane Johnson <sh...@liftiq.com> wrote:

> Now that I am able to deploy I reset the buildpack to
> ...#debug-custom-dist and redeployed. Here is the build log...URL does
> point to the correct distribution with the edited compute-classpath.sh file.
>
> -----> JVM Common app detected
>
> -----> Installing JDK 1.8... done
>
> -----> PredictionIO app detected
>
> -----> Install core components
>
>        + PredictionIO (https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz)
>
>        + Spark (spark-2.1.1-bin-hadoop2.7)
>
> -----> Install supplemental components
>
>        + PostgreSQL (JDBC)
>
>        + S3 HDFS (AWS SDK)
>
>        + S3 HDFS (Hadoop-AWS)
>
>          Writing default 'core-site.xml.erb'
>
>        + local Maven repo from buildpack (contents)
>
> -----> Configure PredictionIO
>
>        Writing default 'pio-env.sh'
>
>        Writing default 'spark-defaults.conf.erb'
>
>        + Maven repo from buildpack (build.sbt entry)
>
>        Set-up environment via '.profile.d/' scripts
>
> -----> Install JVM (heroku/jvm-common)
>
> -----> PredictionIO engine
>
>        Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>
>        [INFO] [Engine$] Using command '/tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0/PredictionIO-dist/sbt/sbt' at /tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0 to build.
>
>        [INFO] [Engine$] If the path above is incorrect, this process will fail.
>
>        [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent.
>
>        [INFO] [Engine$] Going to run: /tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency in /tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0
>
>        [INFO] [Engine$] Compilation finished successfully.
>
>        [INFO] [Engine$] Looking for an engine...
>
>        [INFO] [Engine$] Found template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar
>
>        [INFO] [Engine$] Found template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar
>
>        [INFO] [Engine$] Build finished successfully.
>
>        [INFO] [Pio$] Your engine is ready for training.
>
>        Using default Procfile for engine
>
> -----> Discovering process types
>
>        Procfile declares types -> release, train, web
>
> -----> Compressing...
>
>        Done: 376.7M
>
>
> The release log is below...I am not seeing the  */app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
> *show up at the beginning of the CLASSPATH, this is what we should see
> correct? I was also manipulating the compute-classpath.sh locally as well,
> I observed that adding a line right before echo "$CLASSPATH" was not
> changing what was in the logged spark-submit command as an FYI. This is
> what I was testing locally...
>
>
> CLASSPATH="*/Users/shanejohnson/Desktop/Apps/liftiq_platform/lift-s*
> *core/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*
> :$CLASSPATH"
> echo "$CLASSPATH"
>
> I did not see any change in the spark-submit command by adding this when
> building and deploying locally.
>
> Release Log with new buildpack ..#debug-custom-dist
>
> Running train on release…
>
> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>
> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-hbase-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-s3-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-hdfs-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-jdbc-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env PIO_ENV_LOADED=1,PIO_EVENTSERVER_APP_NAME=classi,PIO_STORAGE_SOURCES_PGSQL_INDEX=enabled,PIO_S3_AWS_ACCESS_KEY_ID=AKIAJJX2S55QPCPZXGFQ,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/app/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost,PIO_S3_BUCKET_NAME=lift-model-devmaster,PIO_EVENTSERVER_ACCESS_KEY=5954-20848-7512-17427-21660,PIO_HOME=/app/PredictionIO-dist,PIO_FS_ENGINESDIR=/app/.pio_store/engines,PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://ec2-52-70-46-243.compute-1.amazonaws.com:5432/dbvbo86hohutvb?sslmode=require,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=PGSQL,PIO_SPARK_OPTS=--driver-memory 13g ,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=PGSQL,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_PGSQL_PASSWORD=p5c404ac780ab517d4ab249d7000809b51b4b987fdfb5c26e1bace511130337ac,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/app/PredictionIO-dist/vendors/elasticsearch,PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc,PIO_FS_TMPDIR=/app/.pio_store/tmp,PIO_STORAGE_SOURCES_PGSQL_USERNAME=ubefhv668b1s1m,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http,PIO_S3_AWS_SECRET_ACCESS_KEY=tQwL1PgYR0Y5MHG+qwVgEXNEcDcdlupaN2oO6JuR,PIO_TRAIN_SPARK_OPTS=--driver-memory 13g ,PIO_STORAGE_SOURCES_PGSQL_CONNECTIONS=8,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=PGSQL,PIO_CONF_DIR=/app/PredictionIO-dist/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200,PIO_STORAGE_SOURCES_PGSQL_PARTITIONS=4,PIO_S3_AWS_REGION=us-east-1
>
> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>
> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>
> [INFO] [Engine] Extracting datasource params...
>
> [INFO] [Engine] Datasource params: (,DataSourceParams(Some(5)))
>
> [INFO] [Engine] Extracting preparator params...
>
> [WARN] [WorkflowUtils$] Non-empty parameters supplied to org.template.liftscoring.Preparator, but its constructor does not accept any arguments. Stubbing with empty parameters.
>
> [INFO] [Engine] Preparator params: (,Empty)
>
>
>
>
>
> *Shane Johnson | LIFT IQ*
> *Founder | CEO*
>
> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
> <sh...@liftiq.com>*
> mobile: (801) 360-3350
> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
> <https://twitter.com/SWaldenJ> |  Facebook
> <https://www.facebook.com/shane.johnson.71653>
>
>
>
> On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <ma...@salesforce.com>
> wrote:
>
>> I'm lost as to how such direct manipulation of CLASSPATH is not appearing
>> in the logged spark-submit command.
>>
>> What could cause this!?
>>
>> I just pushed a version of the buildpack which should help debug.
>> Assuming only a single buildpack is assigned to the app, here's how to set
>> it:
>>
>>   heroku buildpacks:set https://github.com/heroku/pred
>> ictionio-buildpack#debug-custom-dist
>>
>> Then redeploy the engine an check the build log for the line:
>>
>>       + PredictionIO ($URL)
>>
>> Please confirm that it is the URL of your custom PredictionIO dist.
>>
>> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>
>>> Thanks Donald and Mars,
>>>
>>> I created a new distribution (
>>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
>>> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incu
>>> bating/apache-predictionio-0.12.0-incubating-bin.tar.gz) with the added
>>> CLASSPATH code and pointed to the distribution with
>>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
>>>
>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
>>> echo "$CLASSPATH"
>>>
>>> It didn't seem to force the aws-java-sdk to load first as I reviewed the
>>> release logs. Should the aws-java-sdk.jar show up as the first file within
>>> the --jars section when this is added CLASSPATH="/app/PredictionIO-dist/
>>> lib/spark/aws-java-sdk.jar:$CLASSPATH".
>>>
>>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
>>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have
>>> other suggestions to try? I was also testing locally to change the order of
>>> the --jars but changes to the compute-classpath.sh didn't seem to change
>>> the order of the jars in the logs.
>>>
>>> Running train on release…
>>>
>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>
>>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env
>>>
>>>
>>> Error:
>>>
>>> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>>
>>> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>>
>>> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>>
>>> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>>
>>>
>>>
>>>
>>>
>>> *Shane Johnson | LIFT IQ*
>>> *Founder | CEO*
>>>
>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>> <sh...@liftiq.com>*
>>> mobile: (801) 360-3350
>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>> <https://twitter.com/SWaldenJ> |  Facebook
>>> <https://www.facebook.com/shane.johnson.71653>
>>>
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com>
>>> wrote:
>>>
>>>> Shane,
>>>>
>>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com> wrote:
>>>>
>>>>>
>>>>> Re: adding a line to ensure a jar is loaded first. Is this what you
>>>>> are referring to...(line at the bottom in red)?
>>>>>
>>>>
>>>>
>>>> I believe the code would need to look like this to effect the output
>>>> classpath as intended:
>>>>
>>>>
>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>> :$CLASSPATH"
>>>>> echo "$CLASSPATH"
>>>>>
>>>>
>>>>
>>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script
>>>> will need to be skip or remove it first.
>>>>
>>>> --
>>>> *Mars Hall
>>>> 415-818-7039 <(415)%20818-7039>
>>>> Customer Facing Architect
>>>> Salesforce Platform / Heroku
>>>> San Francisco, California
>>>>
>>>
>>>
>>
>>
>> --
>> *Mars Hall
>> 415-818-7039 <(415)%20818-7039>
>> Customer Facing Architect
>> Salesforce Platform / Heroku
>> San Francisco, California
>>
>
>

Re: Issue with loading dependencies and jars

Posted by Shane Johnson <sh...@liftiq.com>.

Now that I am able to deploy I reset the buildpack to ...#debug-custom-dist
and redeployed. Here is the build log...URL does point to the correct
distribution with the edited compute-classpath.sh file.

-----> JVM Common app detected

-----> Installing JDK 1.8... done

-----> PredictionIO app detected

-----> Install core components

       + PredictionIO
(https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz)

       + Spark (spark-2.1.1-bin-hadoop2.7)

-----> Install supplemental components

       + PostgreSQL (JDBC)

       + S3 HDFS (AWS SDK)

       + S3 HDFS (Hadoop-AWS)

         Writing default 'core-site.xml.erb'

       + local Maven repo from buildpack (contents)

-----> Configure PredictionIO

       Writing default 'pio-env.sh'

       Writing default 'spark-defaults.conf.erb'

       + Maven repo from buildpack (build.sbt entry)

       Set-up environment via '.profile.d/' scripts

-----> Install JVM (heroku/jvm-common)

-----> PredictionIO engine

       Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)

       [INFO] [Engine$] Using command
'/tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0/PredictionIO-dist/sbt/sbt'
at /tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0
to build.

       [INFO] [Engine$] If the path above is incorrect, this process will fail.

       [INFO] [Engine$] Uber JAR disabled. Making sure
lib/pio-assembly-0.12.0-incubating.jar is absent.

       [INFO] [Engine$] Going to run:
/tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0/PredictionIO-dist/sbt/sbt
 package assemblyPackageDependency in
/tmp/build_67e7942abed821fccc839c9a79faf0eb/lift-iq-score-e92ed3de9212d04972e0e67e68b5407489e0c8d0

       [INFO] [Engine$] Compilation finished successfully.

       [INFO] [Engine$] Looking for an engine...

       [INFO] [Engine$] Found
template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar

       [INFO] [Engine$] Found
template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar

       [INFO] [Engine$] Build finished successfully.

       [INFO] [Pio$] Your engine is ready for training.

       Using default Procfile for engine

-----> Discovering process types

       Procfile declares types -> release, train, web

-----> Compressing...

       Done: 376.7M


The release log is below...I am not seeing the
*/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
*show up at the beginning of the CLASSPATH, this is what we should see
correct? I was also manipulating the compute-classpath.sh locally as well,
I observed that adding a line right before echo "$CLASSPATH" was not
changing what was in the logged spark-submit command as an FYI. This is
what I was testing locally...


CLASSPATH="*/Users/shanejohnson/Desktop/Apps/liftiq_platform/lift-s*
*core/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*
:$CLASSPATH"
echo "$CLASSPATH"

I did not see any change in the spark-submit command by adding this when
building and deploying locally.

Release Log with new buildpack ..#debug-custom-dist

Running train on release…

Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8

[INFO] [Runner$] Submission command:
/app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit
--driver-memory 13g --class
org.apache.predictionio.workflow.CreateWorkflow --jars
file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-hbase-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-s3-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-hdfs-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/._pio-data-jdbc-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar*
--files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml
--driver-class-path
/app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf
--driver-java-options -Dpio.log.dir=/app
file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar
--engine-id org.template.liftscoring.LiftScoringEngine
--engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20
--engine-variant file:/app/engine.json --verbosity 0 --json-extractor
Both --env PIO_ENV_LOADED=1,PIO_EVENTSERVER_APP_NAME=classi,PIO_STORAGE_SOURCES_PGSQL_INDEX=enabled,PIO_S3_AWS_ACCESS_KEY_ID=AKIAJJX2S55QPCPZXGFQ,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/app/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost,PIO_S3_BUCKET_NAME=lift-model-devmaster,PIO_EVENTSERVER_ACCESS_KEY=5954-20848-7512-17427-21660,PIO_HOME=/app/PredictionIO-dist,PIO_FS_ENGINESDIR=/app/.pio_store/engines,PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://ec2-52-70-46-243.compute-1.amazonaws.com:5432/dbvbo86hohutvb?sslmode=require,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=PGSQL,PIO_SPARK_OPTS=--driver-memory
13g ,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=PGSQL,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_PGSQL_PASSWORD=p5c404ac780ab517d4ab249d7000809b51b4b987fdfb5c26e1bace511130337ac,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/app/PredictionIO-dist/vendors/elasticsearch,PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc,PIO_FS_TMPDIR=/app/.pio_store/tmp,PIO_STORAGE_SOURCES_PGSQL_USERNAME=ubefhv668b1s1m,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http,PIO_S3_AWS_SECRET_ACCESS_KEY=tQwL1PgYR0Y5MHG+qwVgEXNEcDcdlupaN2oO6JuR,PIO_TRAIN_SPARK_OPTS=--driver-memory
13g ,PIO_STORAGE_SOURCES_PGSQL_CONNECTIONS=8,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=PGSQL,PIO_CONF_DIR=/app/PredictionIO-dist/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200,PIO_STORAGE_SOURCES_PGSQL_PARTITIONS=4,PIO_S3_AWS_REGION=us-east-1

Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8

Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8

[INFO] [Engine] Extracting datasource params...

[INFO] [Engine] Datasource params: (,DataSourceParams(Some(5)))

[INFO] [Engine] Extracting preparator params...

[WARN] [WorkflowUtils$] Non-empty parameters supplied to
org.template.liftscoring.Preparator, but its constructor does not
accept any arguments. Stubbing with empty parameters.

[INFO] [Engine] Preparator params: (,Empty)





*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>



On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <ma...@salesforce.com> wrote:

> I'm lost as to how such direct manipulation of CLASSPATH is not appearing
> in the logged spark-submit command.
>
> What could cause this!?
>
> I just pushed a version of the buildpack which should help debug. Assuming
> only a single buildpack is assigned to the app, here's how to set it:
>
>   heroku buildpacks:set https://github.com/heroku/
> predictionio-buildpack#debug-custom-dist
>
> Then redeploy the engine an check the build log for the line:
>
>       + PredictionIO ($URL)
>
> Please confirm that it is the URL of your custom PredictionIO dist.
>
> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote:
>
>> Thanks Donald and Mars,
>>
>> I created a new distribution (
>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
>> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-
>> incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz) with the
>> added CLASSPATH code and pointed to the distribution with
>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
>>
>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
>> echo "$CLASSPATH"
>>
>> It didn't seem to force the aws-java-sdk to load first as I reviewed the
>> release logs. Should the aws-java-sdk.jar show up as the first file within
>> the --jars section when this is added CLASSPATH="/app/PredictionIO-dist/
>> lib/spark/aws-java-sdk.jar:$CLASSPATH".
>>
>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have
>> other suggestions to try? I was also testing locally to change the order of
>> the --jars but changes to the compute-classpath.sh didn't seem to change
>> the order of the jars in the logs.
>>
>> Running train on release…
>>
>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>
>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env
>>
>>
>> Error:
>>
>> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>
>> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>
>> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>
>> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>
>>
>>
>>
>>
>> *Shane Johnson | LIFT IQ*
>> *Founder | CEO*
>>
>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>> <sh...@liftiq.com>*
>> mobile: (801) 360-3350
>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>> <https://twitter.com/SWaldenJ> |  Facebook
>> <https://www.facebook.com/shane.johnson.71653>
>>
>>
>>
>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com>
>> wrote:
>>
>>> Shane,
>>>
>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com> wrote:
>>>
>>>>
>>>> Re: adding a line to ensure a jar is loaded first. Is this what you are
>>>> referring to...(line at the bottom in red)?
>>>>
>>>
>>>
>>> I believe the code would need to look like this to effect the output
>>> classpath as intended:
>>>
>>>
>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>> :$CLASSPATH"
>>>> echo "$CLASSPATH"
>>>>
>>>
>>>
>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script will
>>> need to be skip or remove it first.
>>>
>>> --
>>> *Mars Hall
>>> 415-818-7039 <(415)%20818-7039>
>>> Customer Facing Architect
>>> Salesforce Platform / Heroku
>>> San Francisco, California
>>>
>>
>>
>
>
> --
> *Mars Hall
> 415-818-7039 <(415)%20818-7039>
> Customer Facing Architect
> Salesforce Platform / Heroku
> San Francisco, California
>

Re: Issue with loading dependencies and jars

Posted by Shane Johnson <sh...@liftiq.com>.

Thanks Mars, trying to get back to a stable state and then I will respond
with what I find re: Classpath. Here is some context in the meantime. I was
able to deploy to Heroku again. Not sure how I introduced that error,
perhaps what Donald said.


> Does this error occur locally when you `pio build` this engine?


The error does not occur locally.

Are there any diffs in your local code?


There was not any diffs in my local code. I am trying to find a previous
commit to see if I can deploy it successfully. I'm in git land right now
that is why I haven't gotten back to you until now:) I was able to get a
older version of the code to deploy to heroku so I think I will be able to
solve deploying and then I will resume were we left off on confirming my
URL is pointing to the correct distribution.

How are you running locally? Are you using the buildpack's "local dev"
> setup?


Yes, I am using the buildpacks "local dev" setup.
https://github.com/heroku/predictionio-buildpack/blob/master/DEV.md

*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>



On Fri, Mar 9, 2018 at 12:59 PM, Donald Szeto <do...@apache.org> wrote:

> This error looks like pio build was not run at the engine template
> directory, which would have the proper configuration to enable that command.
>
> On Fri, Mar 9, 2018 at 11:28 AM Mars Hall <ma...@salesforce.com>
> wrote:
>
>> At this point, I'm just searching the internet for "Not a valid command:
>> assemblyPackageDependency" errors, which I image you are too.
>>
>> Does this error occur locally when you `pio build` this engine?
>>
>> Are there any diffs in your local code?
>>
>> How are you running locally? Are you using the buildpack's "local dev"
>> setup?
>>
>> On Fri, Mar 9, 2018 at 6:48 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>
>>> Mars, to test what may be happening I just reverted to the original
>>> buildpack (https://github.com/heroku/predictionio-buildpack.git) and
>>> removed the variable PREDICTIONIO_DIST_URL and I am still getting the
>>> same error. I don't know where I would have introduced this. Have you seen
>>> it before, perhaps this is what is causing the custom dist to not be used?
>>> I'm guessing the build failed before and I was overlooking that. Now I need
>>> to find out why the build is failing:)
>>>
>>>  Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>>>
>>>        [INFO] [Engine$] Using command '/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt' at /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e to build.
>>>
>>>        [INFO] [Engine$] If the path above is incorrect, this process will fail.
>>>
>>>        [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent.
>>>
>>>        [INFO] [Engine$] Going to run: /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency in /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
>>>
>>>        [ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency
>>>
>>>        [ERROR] [Engine$] [error] Not a valid key: assemblyPackageDependency (similar: sbtDependency)
>>>
>>>        [ERROR] [Engine$] [error] assemblyPackageDependency
>>>
>>>        [ERROR] [Engine$] [error]                          ^
>>>
>>>        [ERROR] [Engine$] Return code of build command: /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency is 1. Aborting.
>>>
>>>  !     Push rejected, failed to compile PredictionIO app.
>>>
>>>  !     Push failed
>>>
>>>
>>>
>>>
>>> *Shane Johnson | LIFT IQ*
>>> *Founder | CEO*
>>>
>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>> <sh...@liftiq.com>*
>>> mobile: (801) 360-3350
>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>> <https://twitter.com/SWaldenJ> |  Facebook
>>> <https://www.facebook.com/shane.johnson.71653>
>>>
>>>
>>>
>>> On Fri, Mar 9, 2018 at 11:35 AM, Shane Johnson <sh...@liftiq.com> wrote:
>>>
>>>> It looks like the URL is the correct URL of the custom PredictionIO
>>>> dist. It looks like there is another error that might be occurring here
>>>> when I went to deploy.
>>>>
>>>> -----> JVM Common app detected
>>>>
>>>> -----> Installing JDK 1.8... done
>>>>
>>>> -----> PredictionIO app detected
>>>>
>>>> -----> Install core components
>>>>
>>>>        *+ PredictionIO (https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>)*
>>>>
>>>>        + Spark (spark-2.1.1-bin-hadoop2.7)
>>>>
>>>> -----> Install supplemental components
>>>>
>>>>        + PostgreSQL (JDBC)
>>>>
>>>>        + S3 HDFS (AWS SDK)
>>>>
>>>>        + S3 HDFS (Hadoop-AWS)
>>>>
>>>>          Writing default 'core-site.xml.erb'
>>>>
>>>>        + local Maven repo from buildpack (contents)
>>>>
>>>> -----> Configure PredictionIO
>>>>
>>>>        Writing default 'pio-env.sh'
>>>>
>>>>        Writing default 'spark-defaults.conf.erb'
>>>>
>>>>        + Maven repo from buildpack (build.sbt entry)
>>>>
>>>>        Set-up environment via '.profile.d/' scripts
>>>>
>>>> -----> Install JVM (heroku/jvm-common)
>>>>
>>>> -----> PredictionIO engine
>>>>
>>>>        Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>>>>
>>>>        [INFO] [Engine$] Using command '/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt' at /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e to build.
>>>>
>>>>        [INFO] [Engine$] If the path above is incorrect, this process will fail.
>>>>
>>>>        [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent.
>>>>
>>>>        [INFO] [Engine$] Going to run: /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency in /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
>>>>
>>>>        [ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency
>>>>
>>>>        [ERROR] [Engine$] [error] Not a valid key: assemblyPackageDependency (similar: sbtDependency)
>>>>
>>>>        [ERROR] [Engine$] [error] assemblyPackageDependency
>>>>
>>>>        [ERROR] [Engine$] [error]                          ^
>>>>
>>>>        [ERROR] [Engine$] Return code of build command: /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency is 1. Aborting.
>>>>
>>>>  !     Push rejected, failed to compile PredictionIO app.
>>>>
>>>>  !     Push failed
>>>>
>>>>
>>>>
>>>>
>>>> *Shane Johnson | LIFT IQ*
>>>> *Founder | CEO*
>>>>
>>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>>> <sh...@liftiq.com>*
>>>> mobile: (801) 360-3350
>>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>>> <https://twitter.com/SWaldenJ> |  Facebook
>>>> <https://www.facebook.com/shane.johnson.71653>
>>>>
>>>>
>>>>
>>>> On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <ma...@salesforce.com>
>>>> wrote:
>>>>
>>>>> I'm lost as to how such direct manipulation of CLASSPATH is not
>>>>> appearing in the logged spark-submit command.
>>>>>
>>>>> What could cause this!?
>>>>>
>>>>> I just pushed a version of the buildpack which should help debug.
>>>>> Assuming only a single buildpack is assigned to the app, here's how to set
>>>>> it:
>>>>>
>>>>>   heroku buildpacks:set https://github.com/heroku/
>>>>> predictionio-buildpack#debug-custom-dist
>>>>>
>>>>> Then redeploy the engine an check the build log for the line:
>>>>>
>>>>>       + PredictionIO ($URL)
>>>>>
>>>>> Please confirm that it is the URL of your custom PredictionIO dist.
>>>>>
>>>>> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Donald and Mars,
>>>>>>
>>>>>> I created a new distribution (
>>>>>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
>>>>>> https://s3-us-west-1.amazonaws.com/predictionio/0.
>>>>>> 12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz)
>>>>>> with the added CLASSPATH code and pointed to the distribution with
>>>>>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
>>>>>>
>>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>>> :$CLASSPATH"
>>>>>> echo "$CLASSPATH"
>>>>>>
>>>>>> It didn't seem to force the aws-java-sdk to load first as I reviewed
>>>>>> the release logs. Should the aws-java-sdk.jar show up as the first file
>>>>>> within the --jars section when this is added CLASSPATH="/app/
>>>>>> PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH".
>>>>>>
>>>>>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
>>>>>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you
>>>>>> have other suggestions to try? I was also testing locally to change the
>>>>>> order of the --jars but changes to the compute-classpath.sh didn't seem to
>>>>>> change the order of the jars in the logs.
>>>>>>
>>>>>> Running train on release…
>>>>>>
>>>>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>>>>
>>>>>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env
>>>>>>
>>>>>>
>>>>>> Error:
>>>>>>
>>>>>> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>>>>>
>>>>>> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>>>>>
>>>>>> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>>>>>
>>>>>> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Shane Johnson | LIFT IQ*
>>>>>> *Founder | CEO*
>>>>>>
>>>>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>>>>> <sh...@liftiq.com>*
>>>>>> mobile: (801) 360-3350
>>>>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>>>>> <https://twitter.com/SWaldenJ> |  Facebook
>>>>>> <https://www.facebook.com/shane.johnson.71653>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Shane,
>>>>>>>
>>>>>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Re: adding a line to ensure a jar is loaded first. Is this what you
>>>>>>>> are referring to...(line at the bottom in red)?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I believe the code would need to look like this to effect the output
>>>>>>> classpath as intended:
>>>>>>>
>>>>>>>
>>>>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>>>>> :$CLASSPATH"
>>>>>>>> echo "$CLASSPATH"
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script
>>>>>>> will need to be skip or remove it first.
>>>>>>>
>>>>>>> --
>>>>>>> *Mars Hall
>>>>>>> 415-818-7039 <(415)%20818-7039>
>>>>>>> Customer Facing Architect
>>>>>>> Salesforce Platform / Heroku
>>>>>>> San Francisco, California
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Mars Hall
>>>>> 415-818-7039 <(415)%20818-7039>
>>>>> Customer Facing Architect
>>>>> Salesforce Platform / Heroku
>>>>> San Francisco, California
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> *Mars Hall
>> 415-818-7039 <(415)%20818-7039>
>> Customer Facing Architect
>> Salesforce Platform / Heroku
>> San Francisco, California
>>
>

Re: Issue with loading dependencies and jars

Posted by Donald Szeto <do...@apache.org>.

This error looks like pio build was not run at the engine template
directory, which would have the proper configuration to enable that command.

On Fri, Mar 9, 2018 at 11:28 AM Mars Hall <ma...@salesforce.com> wrote:

> At this point, I'm just searching the internet for "Not a valid command:
> assemblyPackageDependency" errors, which I image you are too.
>
> Does this error occur locally when you `pio build` this engine?
>
> Are there any diffs in your local code?
>
> How are you running locally? Are you using the buildpack's "local dev"
> setup?
>
> On Fri, Mar 9, 2018 at 6:48 PM, Shane Johnson <sh...@liftiq.com> wrote:
>
>> Mars, to test what may be happening I just reverted to the original
>> buildpack (https://github.com/heroku/predictionio-buildpack.git) and
>> removed the variable PREDICTIONIO_DIST_URL and I am still getting the
>> same error. I don't know where I would have introduced this. Have you seen
>> it before, perhaps this is what is causing the custom dist to not be used?
>> I'm guessing the build failed before and I was overlooking that. Now I need
>> to find out why the build is failing:)
>>
>>  Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>>
>>        [INFO] [Engine$] Using command '/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt' at /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e to build.
>>
>>        [INFO] [Engine$] If the path above is incorrect, this process will fail.
>>
>>        [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent.
>>
>>        [INFO] [Engine$] Going to run: /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency in /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
>>
>>        [ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency
>>
>>        [ERROR] [Engine$] [error] Not a valid key: assemblyPackageDependency (similar: sbtDependency)
>>
>>        [ERROR] [Engine$] [error] assemblyPackageDependency
>>
>>        [ERROR] [Engine$] [error]                          ^
>>
>>        [ERROR] [Engine$] Return code of build command: /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency is 1. Aborting.
>>
>>  !     Push rejected, failed to compile PredictionIO app.
>>
>>  !     Push failed
>>
>>
>>
>>
>> *Shane Johnson | LIFT IQ*
>> *Founder | CEO*
>>
>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>> <sh...@liftiq.com>*
>> mobile: (801) 360-3350
>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>> <https://twitter.com/SWaldenJ> |  Facebook
>> <https://www.facebook.com/shane.johnson.71653>
>>
>>
>>
>> On Fri, Mar 9, 2018 at 11:35 AM, Shane Johnson <sh...@liftiq.com> wrote:
>>
>>> It looks like the URL is the correct URL of the custom PredictionIO
>>> dist. It looks like there is another error that might be occurring here
>>> when I went to deploy.
>>>
>>> -----> JVM Common app detected
>>>
>>> -----> Installing JDK 1.8... done
>>>
>>> -----> PredictionIO app detected
>>>
>>> -----> Install core components
>>>
>>>        *+ PredictionIO (https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>)*
>>>
>>>        + Spark (spark-2.1.1-bin-hadoop2.7)
>>>
>>> -----> Install supplemental components
>>>
>>>        + PostgreSQL (JDBC)
>>>
>>>        + S3 HDFS (AWS SDK)
>>>
>>>        + S3 HDFS (Hadoop-AWS)
>>>
>>>          Writing default 'core-site.xml.erb'
>>>
>>>        + local Maven repo from buildpack (contents)
>>>
>>> -----> Configure PredictionIO
>>>
>>>        Writing default 'pio-env.sh'
>>>
>>>        Writing default 'spark-defaults.conf.erb'
>>>
>>>        + Maven repo from buildpack (build.sbt entry)
>>>
>>>        Set-up environment via '.profile.d/' scripts
>>>
>>> -----> Install JVM (heroku/jvm-common)
>>>
>>> -----> PredictionIO engine
>>>
>>>        Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>>>
>>>        [INFO] [Engine$] Using command '/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt' at /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e to build.
>>>
>>>        [INFO] [Engine$] If the path above is incorrect, this process will fail.
>>>
>>>        [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent.
>>>
>>>        [INFO] [Engine$] Going to run: /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency in /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
>>>
>>>        [ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency
>>>
>>>        [ERROR] [Engine$] [error] Not a valid key: assemblyPackageDependency (similar: sbtDependency)
>>>
>>>        [ERROR] [Engine$] [error] assemblyPackageDependency
>>>
>>>        [ERROR] [Engine$] [error]                          ^
>>>
>>>        [ERROR] [Engine$] Return code of build command: /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency is 1. Aborting.
>>>
>>>  !     Push rejected, failed to compile PredictionIO app.
>>>
>>>  !     Push failed
>>>
>>>
>>>
>>>
>>> *Shane Johnson | LIFT IQ*
>>> *Founder | CEO*
>>>
>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>> <sh...@liftiq.com>*
>>> mobile: (801) 360-3350
>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>> <https://twitter.com/SWaldenJ> |  Facebook
>>> <https://www.facebook.com/shane.johnson.71653>
>>>
>>>
>>>
>>> On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <ma...@salesforce.com>
>>> wrote:
>>>
>>>> I'm lost as to how such direct manipulation of CLASSPATH is not
>>>> appearing in the logged spark-submit command.
>>>>
>>>> What could cause this!?
>>>>
>>>> I just pushed a version of the buildpack which should help debug.
>>>> Assuming only a single buildpack is assigned to the app, here's how to set
>>>> it:
>>>>
>>>>   heroku buildpacks:set
>>>> https://github.com/heroku/predictionio-buildpack#debug-custom-dist
>>>>
>>>> Then redeploy the engine an check the build log for the line:
>>>>
>>>>       + PredictionIO ($URL)
>>>>
>>>> Please confirm that it is the URL of your custom PredictionIO dist.
>>>>
>>>> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>>>
>>>>> Thanks Donald and Mars,
>>>>>
>>>>> I created a new distribution (
>>>>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
>>>>> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz)
>>>>> with the added CLASSPATH code and pointed to the distribution with
>>>>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
>>>>>
>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>> :$CLASSPATH"
>>>>> echo "$CLASSPATH"
>>>>>
>>>>> It didn't seem to force the aws-java-sdk to load first as I reviewed
>>>>> the release logs. Should the aws-java-sdk.jar show up as the first file
>>>>> within the --jars section when this is added CLASSPATH="
>>>>> /app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH".
>>>>>
>>>>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
>>>>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have
>>>>> other suggestions to try? I was also testing locally to change the order of
>>>>> the --jars but changes to the compute-classpath.sh didn't seem to change
>>>>> the order of the jars in the logs.
>>>>>
>>>>> Running train on release…
>>>>>
>>>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>>>
>>>>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env
>>>>>
>>>>>
>>>>> Error:
>>>>>
>>>>> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>>>>
>>>>> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>>>>
>>>>> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>>>>
>>>>> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Shane Johnson | LIFT IQ*
>>>>> *Founder | CEO*
>>>>>
>>>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>>>> <sh...@liftiq.com>*
>>>>> mobile: (801) 360-3350
>>>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>>>> <https://twitter.com/SWaldenJ> |  Facebook
>>>>> <https://www.facebook.com/shane.johnson.71653>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com>
>>>>> wrote:
>>>>>
>>>>>> Shane,
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Re: adding a line to ensure a jar is loaded first. Is this what you
>>>>>>> are referring to...(line at the bottom in red)?
>>>>>>>
>>>>>>
>>>>>>
>>>>>> I believe the code would need to look like this to effect the output
>>>>>> classpath as intended:
>>>>>>
>>>>>>
>>>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>>>> :$CLASSPATH"
>>>>>>> echo "$CLASSPATH"
>>>>>>>
>>>>>>
>>>>>>
>>>>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script
>>>>>> will need to be skip or remove it first.
>>>>>>
>>>>>> --
>>>>>> *Mars Hall
>>>>>> 415-818-7039 <(415)%20818-7039>
>>>>>> Customer Facing Architect
>>>>>> Salesforce Platform / Heroku
>>>>>> San Francisco, California
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Mars Hall
>>>> 415-818-7039 <(415)%20818-7039>
>>>> Customer Facing Architect
>>>> Salesforce Platform / Heroku
>>>> San Francisco, California
>>>>
>>>
>>>
>>
>
>
> --
> *Mars Hall
> 415-818-7039
> Customer Facing Architect
> Salesforce Platform / Heroku
> San Francisco, California
>

Re: Issue with loading dependencies and jars

Posted by Mars Hall <ma...@salesforce.com>.

At this point, I'm just searching the internet for "Not a valid command:
assemblyPackageDependency" errors, which I image you are too.

Does this error occur locally when you `pio build` this engine?

Are there any diffs in your local code?

How are you running locally? Are you using the buildpack's "local dev"
setup?

On Fri, Mar 9, 2018 at 6:48 PM, Shane Johnson <sh...@liftiq.com> wrote:

> Mars, to test what may be happening I just reverted to the original
> buildpack (https://github.com/heroku/predictionio-buildpack.git) and
> removed the variable PREDICTIONIO_DIST_URL and I am still getting the
> same error. I don't know where I would have introduced this. Have you seen
> it before, perhaps this is what is causing the custom dist to not be used?
> I'm guessing the build failed before and I was overlooking that. Now I need
> to find out why the build is failing:)
>
>  Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>
>        [INFO] [Engine$] Using command '/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt' at /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e to build.
>
>        [INFO] [Engine$] If the path above is incorrect, this process will fail.
>
>        [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent.
>
>        [INFO] [Engine$] Going to run: /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency in /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
>
>        [ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency
>
>        [ERROR] [Engine$] [error] Not a valid key: assemblyPackageDependency (similar: sbtDependency)
>
>        [ERROR] [Engine$] [error] assemblyPackageDependency
>
>        [ERROR] [Engine$] [error]                          ^
>
>        [ERROR] [Engine$] Return code of build command: /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency is 1. Aborting.
>
>  !     Push rejected, failed to compile PredictionIO app.
>
>  !     Push failed
>
>
>
>
> *Shane Johnson | LIFT IQ*
> *Founder | CEO*
>
> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
> <sh...@liftiq.com>*
> mobile: (801) 360-3350
> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
> <https://twitter.com/SWaldenJ> |  Facebook
> <https://www.facebook.com/shane.johnson.71653>
>
>
>
> On Fri, Mar 9, 2018 at 11:35 AM, Shane Johnson <sh...@liftiq.com> wrote:
>
>> It looks like the URL is the correct URL of the custom PredictionIO dist.
>> It looks like there is another error that might be occurring here when I
>> went to deploy.
>>
>> -----> JVM Common app detected
>>
>> -----> Installing JDK 1.8... done
>>
>> -----> PredictionIO app detected
>>
>> -----> Install core components
>>
>>        *+ PredictionIO (https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>)*
>>
>>        + Spark (spark-2.1.1-bin-hadoop2.7)
>>
>> -----> Install supplemental components
>>
>>        + PostgreSQL (JDBC)
>>
>>        + S3 HDFS (AWS SDK)
>>
>>        + S3 HDFS (Hadoop-AWS)
>>
>>          Writing default 'core-site.xml.erb'
>>
>>        + local Maven repo from buildpack (contents)
>>
>> -----> Configure PredictionIO
>>
>>        Writing default 'pio-env.sh'
>>
>>        Writing default 'spark-defaults.conf.erb'
>>
>>        + Maven repo from buildpack (build.sbt entry)
>>
>>        Set-up environment via '.profile.d/' scripts
>>
>> -----> Install JVM (heroku/jvm-common)
>>
>> -----> PredictionIO engine
>>
>>        Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>>
>>        [INFO] [Engine$] Using command '/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt' at /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e to build.
>>
>>        [INFO] [Engine$] If the path above is incorrect, this process will fail.
>>
>>        [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent.
>>
>>        [INFO] [Engine$] Going to run: /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency in /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
>>
>>        [ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency
>>
>>        [ERROR] [Engine$] [error] Not a valid key: assemblyPackageDependency (similar: sbtDependency)
>>
>>        [ERROR] [Engine$] [error] assemblyPackageDependency
>>
>>        [ERROR] [Engine$] [error]                          ^
>>
>>        [ERROR] [Engine$] Return code of build command: /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency is 1. Aborting.
>>
>>  !     Push rejected, failed to compile PredictionIO app.
>>
>>  !     Push failed
>>
>>
>>
>>
>> *Shane Johnson | LIFT IQ*
>> *Founder | CEO*
>>
>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>> <sh...@liftiq.com>*
>> mobile: (801) 360-3350
>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>> <https://twitter.com/SWaldenJ> |  Facebook
>> <https://www.facebook.com/shane.johnson.71653>
>>
>>
>>
>> On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <ma...@salesforce.com>
>> wrote:
>>
>>> I'm lost as to how such direct manipulation of CLASSPATH is not
>>> appearing in the logged spark-submit command.
>>>
>>> What could cause this!?
>>>
>>> I just pushed a version of the buildpack which should help debug.
>>> Assuming only a single buildpack is assigned to the app, here's how to set
>>> it:
>>>
>>>   heroku buildpacks:set https://github.com/heroku/pred
>>> ictionio-buildpack#debug-custom-dist
>>>
>>> Then redeploy the engine an check the build log for the line:
>>>
>>>       + PredictionIO ($URL)
>>>
>>> Please confirm that it is the URL of your custom PredictionIO dist.
>>>
>>> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>>
>>>> Thanks Donald and Mars,
>>>>
>>>> I created a new distribution (
>>>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
>>>> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incu
>>>> bating/apache-predictionio-0.12.0-incubating-bin.tar.gz) with the
>>>> added CLASSPATH code and pointed to the distribution with
>>>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
>>>>
>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>> :$CLASSPATH"
>>>> echo "$CLASSPATH"
>>>>
>>>> It didn't seem to force the aws-java-sdk to load first as I reviewed
>>>> the release logs. Should the aws-java-sdk.jar show up as the first file
>>>> within the --jars section when this is added CLASSPATH="/app/Predicti
>>>> onIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH".
>>>>
>>>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
>>>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have
>>>> other suggestions to try? I was also testing locally to change the order of
>>>> the --jars but changes to the compute-classpath.sh didn't seem to change
>>>> the order of the jars in the logs.
>>>>
>>>> Running train on release…
>>>>
>>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>>
>>>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env
>>>>
>>>>
>>>> Error:
>>>>
>>>> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>>>
>>>> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>>>
>>>> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>>>
>>>> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Shane Johnson | LIFT IQ*
>>>> *Founder | CEO*
>>>>
>>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>>> <sh...@liftiq.com>*
>>>> mobile: (801) 360-3350
>>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>>> <https://twitter.com/SWaldenJ> |  Facebook
>>>> <https://www.facebook.com/shane.johnson.71653>
>>>>
>>>>
>>>>
>>>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com>
>>>> wrote:
>>>>
>>>>> Shane,
>>>>>
>>>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Re: adding a line to ensure a jar is loaded first. Is this what you
>>>>>> are referring to...(line at the bottom in red)?
>>>>>>
>>>>>
>>>>>
>>>>> I believe the code would need to look like this to effect the output
>>>>> classpath as intended:
>>>>>
>>>>>
>>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>>> :$CLASSPATH"
>>>>>> echo "$CLASSPATH"
>>>>>>
>>>>>
>>>>>
>>>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script
>>>>> will need to be skip or remove it first.
>>>>>
>>>>> --
>>>>> *Mars Hall
>>>>> 415-818-7039 <(415)%20818-7039>
>>>>> Customer Facing Architect
>>>>> Salesforce Platform / Heroku
>>>>> San Francisco, California
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Mars Hall
>>> 415-818-7039 <(415)%20818-7039>
>>> Customer Facing Architect
>>> Salesforce Platform / Heroku
>>> San Francisco, California
>>>
>>
>>
>


-- 
*Mars Hall
415-818-7039
Customer Facing Architect
Salesforce Platform / Heroku
San Francisco, California

Re: Issue with loading dependencies and jars

Posted by Shane Johnson <sh...@liftiq.com>.

Mars, to test what may be happening I just reverted to the original
buildpack (https://github.com/heroku/predictionio-buildpack.git) and
removed the variable PREDICTIONIO_DIST_URL and I am still getting the same
error. I don't know where I would have introduced this. Have you seen it
before, perhaps this is what is causing the custom dist to not be used? I'm
guessing the build failed before and I was overlooking that. Now I need to
find out why the build is failing:)

 Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)

       [INFO] [Engine$] Using command
'/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt'
at /tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
to build.

       [INFO] [Engine$] If the path above is incorrect, this process will fail.

       [INFO] [Engine$] Uber JAR disabled. Making sure
lib/pio-assembly-0.12.0-incubating.jar is absent.

       [INFO] [Engine$] Going to run:
/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt
 package assemblyPackageDependency in
/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e

       [ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency

       [ERROR] [Engine$] [error] Not a valid key:
assemblyPackageDependency (similar: sbtDependency)

       [ERROR] [Engine$] [error] assemblyPackageDependency

       [ERROR] [Engine$] [error]                          ^

       [ERROR] [Engine$] Return code of build command:
/tmp/build_28c7923a4f807cd0eaa37b606989fe11/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt
 package assemblyPackageDependency is 1. Aborting.

 !     Push rejected, failed to compile PredictionIO app.

 !     Push failed




*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>



On Fri, Mar 9, 2018 at 11:35 AM, Shane Johnson <sh...@liftiq.com> wrote:

> It looks like the URL is the correct URL of the custom PredictionIO dist.
> It looks like there is another error that might be occurring here when I
> went to deploy.
>
> -----> JVM Common app detected
>
> -----> Installing JDK 1.8... done
>
> -----> PredictionIO app detected
>
> -----> Install core components
>
>        *+ PredictionIO (https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>)*
>
>        + Spark (spark-2.1.1-bin-hadoop2.7)
>
> -----> Install supplemental components
>
>        + PostgreSQL (JDBC)
>
>        + S3 HDFS (AWS SDK)
>
>        + S3 HDFS (Hadoop-AWS)
>
>          Writing default 'core-site.xml.erb'
>
>        + local Maven repo from buildpack (contents)
>
> -----> Configure PredictionIO
>
>        Writing default 'pio-env.sh'
>
>        Writing default 'spark-defaults.conf.erb'
>
>        + Maven repo from buildpack (build.sbt entry)
>
>        Set-up environment via '.profile.d/' scripts
>
> -----> Install JVM (heroku/jvm-common)
>
> -----> PredictionIO engine
>
>        Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)
>
>        [INFO] [Engine$] Using command '/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt' at /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e to build.
>
>        [INFO] [Engine$] If the path above is incorrect, this process will fail.
>
>        [INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.12.0-incubating.jar is absent.
>
>        [INFO] [Engine$] Going to run: /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency in /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
>
>        [ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency
>
>        [ERROR] [Engine$] [error] Not a valid key: assemblyPackageDependency (similar: sbtDependency)
>
>        [ERROR] [Engine$] [error] assemblyPackageDependency
>
>        [ERROR] [Engine$] [error]                          ^
>
>        [ERROR] [Engine$] Return code of build command: /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt  package assemblyPackageDependency is 1. Aborting.
>
>  !     Push rejected, failed to compile PredictionIO app.
>
>  !     Push failed
>
>
>
>
> *Shane Johnson | LIFT IQ*
> *Founder | CEO*
>
> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
> <sh...@liftiq.com>*
> mobile: (801) 360-3350
> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
> <https://twitter.com/SWaldenJ> |  Facebook
> <https://www.facebook.com/shane.johnson.71653>
>
>
>
> On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <ma...@salesforce.com>
> wrote:
>
>> I'm lost as to how such direct manipulation of CLASSPATH is not appearing
>> in the logged spark-submit command.
>>
>> What could cause this!?
>>
>> I just pushed a version of the buildpack which should help debug.
>> Assuming only a single buildpack is assigned to the app, here's how to set
>> it:
>>
>>   heroku buildpacks:set https://github.com/heroku/pred
>> ictionio-buildpack#debug-custom-dist
>>
>> Then redeploy the engine an check the build log for the line:
>>
>>       + PredictionIO ($URL)
>>
>> Please confirm that it is the URL of your custom PredictionIO dist.
>>
>> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>
>>> Thanks Donald and Mars,
>>>
>>> I created a new distribution (
>>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
>>> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incu
>>> bating/apache-predictionio-0.12.0-incubating-bin.tar.gz) with the added
>>> CLASSPATH code and pointed to the distribution with
>>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
>>>
>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
>>> echo "$CLASSPATH"
>>>
>>> It didn't seem to force the aws-java-sdk to load first as I reviewed the
>>> release logs. Should the aws-java-sdk.jar show up as the first file within
>>> the --jars section when this is added CLASSPATH="/app/PredictionIO-dist/
>>> lib/spark/aws-java-sdk.jar:$CLASSPATH".
>>>
>>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
>>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have
>>> other suggestions to try? I was also testing locally to change the order of
>>> the --jars but changes to the compute-classpath.sh didn't seem to change
>>> the order of the jars in the logs.
>>>
>>> Running train on release…
>>>
>>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>>
>>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env
>>>
>>>
>>> Error:
>>>
>>> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>>
>>> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>>
>>> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>>
>>> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>>
>>>
>>>
>>>
>>>
>>> *Shane Johnson | LIFT IQ*
>>> *Founder | CEO*
>>>
>>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>>> <sh...@liftiq.com>*
>>> mobile: (801) 360-3350
>>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>>> <https://twitter.com/SWaldenJ> |  Facebook
>>> <https://www.facebook.com/shane.johnson.71653>
>>>
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com>
>>> wrote:
>>>
>>>> Shane,
>>>>
>>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com> wrote:
>>>>
>>>>>
>>>>> Re: adding a line to ensure a jar is loaded first. Is this what you
>>>>> are referring to...(line at the bottom in red)?
>>>>>
>>>>
>>>>
>>>> I believe the code would need to look like this to effect the output
>>>> classpath as intended:
>>>>
>>>>
>>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>>> :$CLASSPATH"
>>>>> echo "$CLASSPATH"
>>>>>
>>>>
>>>>
>>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script
>>>> will need to be skip or remove it first.
>>>>
>>>> --
>>>> *Mars Hall
>>>> 415-818-7039 <(415)%20818-7039>
>>>> Customer Facing Architect
>>>> Salesforce Platform / Heroku
>>>> San Francisco, California
>>>>
>>>
>>>
>>
>>
>> --
>> *Mars Hall
>> 415-818-7039 <(415)%20818-7039>
>> Customer Facing Architect
>> Salesforce Platform / Heroku
>> San Francisco, California
>>
>
>

Re: Issue with loading dependencies and jars

Posted by Shane Johnson <sh...@liftiq.com>.

It looks like the URL is the correct URL of the custom PredictionIO dist.
It looks like there is another error that might be occurring here when I
went to deploy.

-----> JVM Common app detected

-----> Installing JDK 1.8... done

-----> PredictionIO app detected

-----> Install core components

       *+ PredictionIO
(https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz
<https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>)*

       + Spark (spark-2.1.1-bin-hadoop2.7)

-----> Install supplemental components

       + PostgreSQL (JDBC)

       + S3 HDFS (AWS SDK)

       + S3 HDFS (Hadoop-AWS)

         Writing default 'core-site.xml.erb'

       + local Maven repo from buildpack (contents)

-----> Configure PredictionIO

       Writing default 'pio-env.sh'

       Writing default 'spark-defaults.conf.erb'

       + Maven repo from buildpack (build.sbt entry)

       Set-up environment via '.profile.d/' scripts

-----> Install JVM (heroku/jvm-common)

-----> PredictionIO engine

       Quietly logging. (Set `PIO_VERBOSE=true` for detailed build log.)

       [INFO] [Engine$] Using command
'/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt'
at /tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e
to build.

       [INFO] [Engine$] If the path above is incorrect, this process will fail.

       [INFO] [Engine$] Uber JAR disabled. Making sure
lib/pio-assembly-0.12.0-incubating.jar is absent.

       [INFO] [Engine$] Going to run:
/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt
 package assemblyPackageDependency in
/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e

       [ERROR] [Engine$] [error] Not a valid command: assemblyPackageDependency

       [ERROR] [Engine$] [error] Not a valid key:
assemblyPackageDependency (similar: sbtDependency)

       [ERROR] [Engine$] [error] assemblyPackageDependency

       [ERROR] [Engine$] [error]                          ^

       [ERROR] [Engine$] Return code of build command:
/tmp/build_a27b71b6803e5ee2db3367fa69981626/lift-iq-score-f7694693324fc7acebdc6effb0134c218eb7434e/PredictionIO-dist/sbt/sbt
 package assemblyPackageDependency is 1. Aborting.

 !     Push rejected, failed to compile PredictionIO app.

 !     Push failed




*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>



On Fri, Mar 9, 2018 at 11:17 AM, Mars Hall <ma...@salesforce.com> wrote:

> I'm lost as to how such direct manipulation of CLASSPATH is not appearing
> in the logged spark-submit command.
>
> What could cause this!?
>
> I just pushed a version of the buildpack which should help debug. Assuming
> only a single buildpack is assigned to the app, here's how to set it:
>
>   heroku buildpacks:set https://github.com/heroku/
> predictionio-buildpack#debug-custom-dist
>
> Then redeploy the engine an check the build log for the line:
>
>       + PredictionIO ($URL)
>
> Please confirm that it is the URL of your custom PredictionIO dist.
>
> On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote:
>
>> Thanks Donald and Mars,
>>
>> I created a new distribution (
>> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
>> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-
>> incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz) with the
>> added CLASSPATH code and pointed to the distribution with
>> the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.
>>
>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
>> echo "$CLASSPATH"
>>
>> It didn't seem to force the aws-java-sdk to load first as I reviewed the
>> release logs. Should the aws-java-sdk.jar show up as the first file within
>> the --jars section when this is added CLASSPATH="/app/PredictionIO-dist/
>> lib/spark/aws-java-sdk.jar:$CLASSPATH".
>>
>> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
>> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have
>> other suggestions to try? I was also testing locally to change the order of
>> the --jars but changes to the compute-classpath.sh didn't seem to change
>> the order of the jars in the logs.
>>
>> Running train on release…
>>
>> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>>
>> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env
>>
>>
>> Error:
>>
>> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>
>> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>
>> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>
>> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>
>>
>>
>>
>>
>> *Shane Johnson | LIFT IQ*
>> *Founder | CEO*
>>
>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>> <sh...@liftiq.com>*
>> mobile: (801) 360-3350
>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>> <https://twitter.com/SWaldenJ> |  Facebook
>> <https://www.facebook.com/shane.johnson.71653>
>>
>>
>>
>> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com>
>> wrote:
>>
>>> Shane,
>>>
>>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com> wrote:
>>>
>>>>
>>>> Re: adding a line to ensure a jar is loaded first. Is this what you are
>>>> referring to...(line at the bottom in red)?
>>>>
>>>
>>>
>>> I believe the code would need to look like this to effect the output
>>> classpath as intended:
>>>
>>>
>>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
>>>> :$CLASSPATH"
>>>> echo "$CLASSPATH"
>>>>
>>>
>>>
>>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script will
>>> need to be skip or remove it first.
>>>
>>> --
>>> *Mars Hall
>>> 415-818-7039 <(415)%20818-7039>
>>> Customer Facing Architect
>>> Salesforce Platform / Heroku
>>> San Francisco, California
>>>
>>
>>
>
>
> --
> *Mars Hall
> 415-818-7039 <(415)%20818-7039>
> Customer Facing Architect
> Salesforce Platform / Heroku
> San Francisco, California
>

Re: Issue with loading dependencies and jars

Posted by Mars Hall <ma...@salesforce.com>.

I'm lost as to how such direct manipulation of CLASSPATH is not appearing
in the logged spark-submit command.

What could cause this!?

I just pushed a version of the buildpack which should help debug. Assuming
only a single buildpack is assigned to the app, here's how to set it:

  heroku buildpacks:set
https://github.com/heroku/predictionio-buildpack#debug-custom-dist

Then redeploy the engine an check the build log for the line:

      + PredictionIO ($URL)

Please confirm that it is the URL of your custom PredictionIO dist.

On Fri, Mar 9, 2018 at 2:47 PM, Shane Johnson <sh...@liftiq.com> wrote:

> Thanks Donald and Mars,
>
> I created a new distribution (
> <https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
> https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-
> predictionio-0.12.0-incubating-bin.tar.gz) with the added CLASSPATH code
> and pointed to the distribution with the PREDICTIONIO_DIST_URL variable
> within the engine app in Heroku.
>
> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
> echo "$CLASSPATH"
>
> It didn't seem to force the aws-java-sdk to load first as I reviewed the
> release logs. Should the aws-java-sdk.jar show up as the first file within
> the --jars section when this is added CLASSPATH="/app/PredictionIO-dist/
> lib/spark/aws-java-sdk.jar:$CLASSPATH".
>
> I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
> after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have
> other suggestions to try? I was also testing locally to change the order of
> the --jars but changes to the compute-classpath.sh didn't seem to change
> the order of the jars in the logs.
>
> Running train on release…
>
> Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8
>
> [INFO] [Runner$] Submission command: /app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit --driver-memory 13g --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar* --files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml --driver-class-path /app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf --driver-java-options -Dpio.log.dir=/app file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.template.liftscoring.LiftScoringEngine --engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20 --engine-variant file:/app/engine.json --verbosity 0 --json-extractor Both --env
>
>
> Error:
>
> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>
> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>
> 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>
> 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>
>
>
>
>
> *Shane Johnson | LIFT IQ*
> *Founder | CEO*
>
> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
> <sh...@liftiq.com>*
> mobile: (801) 360-3350
> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
> <https://twitter.com/SWaldenJ> |  Facebook
> <https://www.facebook.com/shane.johnson.71653>
>
>
>
> On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com>
> wrote:
>
>> Shane,
>>
>> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com> wrote:
>>
>>>
>>> Re: adding a line to ensure a jar is loaded first. Is this what you are
>>> referring to...(line at the bottom in red)?
>>>
>>
>>
>> I believe the code would need to look like this to effect the output
>> classpath as intended:
>>
>>
>>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
>>> echo "$CLASSPATH"
>>>
>>
>>
>> aws-java-sdk.jar is already in the CLASSPATH though, So, the script will
>> need to be skip or remove it first.
>>
>> --
>> *Mars Hall
>> 415-818-7039 <(415)%20818-7039>
>> Customer Facing Architect
>> Salesforce Platform / Heroku
>> San Francisco, California
>>
>
>


-- 
*Mars Hall
415-818-7039
Customer Facing Architect
Salesforce Platform / Heroku
San Francisco, California

Re: Issue with loading dependencies and jars

Posted by Shane Johnson <sh...@liftiq.com>.

Thanks Donald and Mars,

I created a new distribution (
<https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz>
https://s3-us-west-1.amazonaws.com/predictionio/0.12.0-incubating/apache-predictionio-0.12.0-incubating-bin.tar.gz)
with the added CLASSPATH code and pointed to the distribution with
the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.

CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
echo "$CLASSPATH"

It didn't seem to force the aws-java-sdk to load first as I reviewed the
release logs. Should the aws-java-sdk.jar show up as the first file within
the --jars section when this is added CLASSPATH="/app/PredictionIO-dist/
lib/spark/aws-java-sdk.jar:$CLASSPATH".

I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have other
suggestions to try? I was also testing locally to change the order of the
--jars but changes to the compute-classpath.sh didn't seem to change the
order of the jars in the logs.

Running train on release…

Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8

[INFO] [Runner$] Submission command:
/app/PredictionIO-dist/vendors/spark-hadoop/bin/spark-submit
--driver-memory 13g --class
org.apache.predictionio.workflow.CreateWorkflow --jars
file:/app/PredictionIO-dist/lib/postgresql_jdbc.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring-assembly-0.1-SNAPSHOT-deps.jar,file:/app/target/scala-2.11/template-scala-parallel-liftscoring_2.11-0.1-SNAPSHOT.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,file:/app/PredictionIO-dist/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar*,file:/app/PredictionIO-dist/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,*file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar*
--files file:/app/PredictionIO-dist/conf/log4j.properties,file:/app/PredictionIO-dist/conf/core-site.xml
--driver-class-path
/app/PredictionIO-dist/conf:/app/PredictionIO-dist/conf:/app/PredictionIO-dist/lib/postgresql_jdbc.jar:/app/PredictionIO-dist/conf
--driver-java-options -Dpio.log.dir=/app
file:/app/PredictionIO-dist/lib/pio-assembly-0.12.0-incubating.jar
--engine-id org.template.liftscoring.LiftScoringEngine
--engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20
--engine-variant file:/app/engine.json --verbosity 0 --json-extractor
Both --env


Error:

Exception in thread "main" java.lang.NoSuchMethodError:
com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V

	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)

	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)





*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>



On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <ma...@salesforce.com> wrote:

> Shane,
>
> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com> wrote:
>
>>
>> Re: adding a line to ensure a jar is loaded first. Is this what you are
>> referring to...(line at the bottom in red)?
>>
>
>
> I believe the code would need to look like this to effect the output
> classpath as intended:
>
>
>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
>> echo "$CLASSPATH"
>>
>
>
> aws-java-sdk.jar is already in the CLASSPATH though, So, the script will
> need to be skip or remove it first.
>
> --
> *Mars Hall
> 415-818-7039 <(415)%20818-7039>
> Customer Facing Architect
> Salesforce Platform / Heroku
> San Francisco, California
>

Re: Issue with loading dependencies and jars

Posted by Mars Hall <ma...@salesforce.com>.

Shane,

On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <sh...@liftiq.com> wrote:

>
> Re: adding a line to ensure a jar is loaded first. Is this what you are
> referring to...(line at the bottom in red)?
>

I believe the code would need to look like this to effect the output
classpath as intended:

> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
> echo "$CLASSPATH"
>

aws-java-sdk.jar is already in the CLASSPATH though, So, the script will
need to be skip or remove it first.

-- 
*Mars Hall
415-818-7039
Customer Facing Architect
Salesforce Platform / Heroku
San Francisco, California

Re: Issue with loading dependencies and jars

Posted by Shane Johnson <sh...@liftiq.com>.

Thanks Donald,

Re: adding a line to ensure a jar is loaded first. Is this what you are
referring to...(line at the bottom in red)?


# Add hadoop conf dir if given -- otherwise FileSystem.*, etc fail ! Note,
this
# assumes that there is either a HADOOP_CONF_DIR or YARN_CONF_DIR which
hosts
# the configurtion files.
if [ -n "$HADOOP_CONF_DIR" ]; then
  CLASSPATH="$CLASSPATH:$HADOOP_CONF_DIR"
fi
if [ -n "$YARN_CONF_DIR" ]; then
  CLASSPATH="$CLASSPATH:$YARN_CONF_DIR"
fi
if [ -n "$HBASE_CONF_DIR" ]; then
  CLASSPATH="$CLASSPATH:$HBASE_CONF_DIR"
fi
if [ -n "$ES_CONF_DIR" ]; then
  CLASSPATH="$CLASSPATH:$ES_CONF_DIR"
fi
if [ -n "$POSTGRES_JDBC_DRIVER" ]; then
  CLASSPATH="$CLASSPATH:$POSTGRES_JDBC_DRIVER"
  ASSEMBLY_JARS="$ASSEMBLY_JARS,$POSTGRES_JDBC_DRIVER"
fi
if [ -n "$MYSQL_JDBC_DRIVER" ]; then
  CLASSPATH="$CLASSPATH:$MYSQL_JDBC_DRIVER"
  ASSEMBLY_JARS="$ASSEMBLY_JARS,$MYSQL_JDBC_DRIVER"
fi

file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar
echo "$CLASSPATH"



*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>



On Tue, Mar 6, 2018 at 6:58 PM, Donald Szeto <do...@apache.org> wrote:

> Even easier: skip cloning, and just edit the shell script directly in the
> binary distribution. Hope that works.
>
> Regards,
> Donald
>
> On Tue, Mar 6, 2018 at 5:41 PM Shane Johnson <sh...@liftiq.com> wrote:
>
>> Thanks Mars and Donald. I think this gets me to next steps:
>>
>>    - Clone PredictionIO 0.12 and adjust the bin/compute-classpath.sh to
>>    have aws-java-sdk-1.7.4 loaded first.
>>    - Create custom binary distribution of PredicionIO 0.12.
>>    - Add config var to point to custom binary distribution.
>>
>> This is very helpful. Thank you!
>>
>> *Shane Johnson | LIFT IQ*
>> *Founder | CEO*
>>
>> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
>> <sh...@liftiq.com>*
>> mobile: (801) 360-3350
>> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
>> <https://twitter.com/SWaldenJ> |  Facebook
>> <https://www.facebook.com/shane.johnson.71653>
>>
>>
>>
>> On Tue, Mar 6, 2018 at 4:52 PM, Mars Hall <ma...@salesforce.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Mar 6, 2018 at 11:39 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>>
>>>>
>>>> Do you know the version of hadoop-aws.jar and aws-java-sdk.jar that you
>>>> are using?
>>>>
>>>> I do not know what version is being used. Is this something that I can
>>>> specify or control? I am using the PredictionIO buildpack
>>>> https://github.com/heroku/predictionio-buildpack. I am *not*
>>>> specifying these in my build.sbt currently.
>>>>
>>>
>>> The versions are specified in the "bin/common/setup-runtime" script of
>>> the buildpack:
>>>   https://github.com/heroku/predictionio-buildpack/blob/
>>> master/bin/common/setup-runtime#L50
>>>
>>> Currently they are:
>>>
>>>    - hadoop-aws-2.7.3
>>>    - aws-java-sdk-1.7.4
>>>
>>> If you fork the buildpack, those download URLs (currently hosted in my
>>> S3 bucket) can be changed.
>>>
>>>
>>>
>>>> You are also right that you can modify the class path in
>>>> bin/compute-classpath.sh as a short term fix. The current order is
>>>> following the output of your target system's `ls`, so the order is not
>>>> guaranteed like you speak. Right before the last line (echo $CLASSPATH),
>>>> you can add a line to make sure that the JAR you want to be loaded first is
>>>> at the very beginning.
>>>>
>>>> I believe this would need to be something that is edited on master (
>>>> https://github.com/apache/predictionio) as the buildpack leverages it
>>>> vs. a cloned version of the code that I can edit, am I thinking through
>>>> that correctly? I may need to circle back with Mars to see if there are any
>>>> other options to get this to work with Heroku. Is this something that can
>>>> be committed to master? The aws-java-sdk.jar is the one that we need
>>>> to load first.
>>>>
>>>
>>> If you build your own custom binary distribution of PredictionIO 0.12
>>> and upload it to a publicly accessible URL, then the buildpack can be
>>> configured to use it by setting config var `PREDICTIONIO_DIST_URL` which is
>>> used by the same setup script as above:
>>>   https://github.com/heroku/predictionio-buildpack/blob/
>>> master/bin/common/setup-runtime#L82
>>>
>>> This is how I tested my pre-release builds on Heroku.
>>>
>>> --
>>> *Mars Hall
>>> 415-818-7039 <(415)%20818-7039>
>>> Customer Facing Architect
>>> Salesforce Platform / Heroku
>>> San Francisco, California
>>>
>>
>>

Re: Issue with loading dependencies and jars

Posted by Donald Szeto <do...@apache.org>.

Even easier: skip cloning, and just edit the shell script directly in the
binary distribution. Hope that works.

Regards,
Donald

On Tue, Mar 6, 2018 at 5:41 PM Shane Johnson <sh...@liftiq.com> wrote:

> Thanks Mars and Donald. I think this gets me to next steps:
>
>    - Clone PredictionIO 0.12 and adjust the bin/compute-classpath.sh to
>    have aws-java-sdk-1.7.4 loaded first.
>    - Create custom binary distribution of PredicionIO 0.12.
>    - Add config var to point to custom binary distribution.
>
> This is very helpful. Thank you!
>
> *Shane Johnson | LIFT IQ*
> *Founder | CEO*
>
> *www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
> <sh...@liftiq.com>*
> mobile: (801) 360-3350
> LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
> <https://twitter.com/SWaldenJ> |  Facebook
> <https://www.facebook.com/shane.johnson.71653>
>
>
>
> On Tue, Mar 6, 2018 at 4:52 PM, Mars Hall <ma...@salesforce.com>
> wrote:
>
>>
>>
>> On Tue, Mar 6, 2018 at 11:39 PM, Shane Johnson <sh...@liftiq.com> wrote:
>>
>>>
>>> Do you know the version of hadoop-aws.jar and aws-java-sdk.jar that you
>>> are using?
>>>
>>> I do not know what version is being used. Is this something that I can
>>> specify or control? I am using the PredictionIO buildpack
>>> https://github.com/heroku/predictionio-buildpack. I am *not* specifying
>>> these in my build.sbt currently.
>>>
>>
>> The versions are specified in the "bin/common/setup-runtime" script of
>> the buildpack:
>>
>> https://github.com/heroku/predictionio-buildpack/blob/master/bin/common/setup-runtime#L50
>>
>> Currently they are:
>>
>>    - hadoop-aws-2.7.3
>>    - aws-java-sdk-1.7.4
>>
>> If you fork the buildpack, those download URLs (currently hosted in my S3
>> bucket) can be changed.
>>
>>
>>
>>> You are also right that you can modify the class path in
>>> bin/compute-classpath.sh as a short term fix. The current order is
>>> following the output of your target system's `ls`, so the order is not
>>> guaranteed like you speak. Right before the last line (echo $CLASSPATH),
>>> you can add a line to make sure that the JAR you want to be loaded first is
>>> at the very beginning.
>>>
>>> I believe this would need to be something that is edited on master (
>>> https://github.com/apache/predictionio) as the buildpack leverages it
>>> vs. a cloned version of the code that I can edit, am I thinking through
>>> that correctly? I may need to circle back with Mars to see if there are any
>>> other options to get this to work with Heroku. Is this something that can
>>> be committed to master? The aws-java-sdk.jar is the one that we need to
>>> load first.
>>>
>>
>> If you build your own custom binary distribution of PredictionIO 0.12 and
>> upload it to a publicly accessible URL, then the buildpack can be
>> configured to use it by setting config var `PREDICTIONIO_DIST_URL` which is
>> used by the same setup script as above:
>>
>> https://github.com/heroku/predictionio-buildpack/blob/master/bin/common/setup-runtime#L82
>>
>> This is how I tested my pre-release builds on Heroku.
>>
>> --
>> *Mars Hall
>> 415-818-7039 <(415)%20818-7039>
>> Customer Facing Architect
>> Salesforce Platform / Heroku
>> San Francisco, California
>>
>
>

Re: Issue with loading dependencies and jars

Posted by Shane Johnson <sh...@liftiq.com>.

Thanks Mars and Donald. I think this gets me to next steps:

   - Clone PredictionIO 0.12 and adjust the bin/compute-classpath.sh to
   have aws-java-sdk-1.7.4 loaded first.
   - Create custom binary distribution of PredicionIO 0.12.
   - Add config var to point to custom binary distribution.

This is very helpful. Thank you!

*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>



On Tue, Mar 6, 2018 at 4:52 PM, Mars Hall <ma...@salesforce.com> wrote:

>
>
> On Tue, Mar 6, 2018 at 11:39 PM, Shane Johnson <sh...@liftiq.com> wrote:
>
>>
>> Do you know the version of hadoop-aws.jar and aws-java-sdk.jar that you
>> are using?
>>
>> I do not know what version is being used. Is this something that I can
>> specify or control? I am using the PredictionIO buildpack
>> https://github.com/heroku/predictionio-buildpack. I am *not* specifying
>> these in my build.sbt currently.
>>
>
> The versions are specified in the "bin/common/setup-runtime" script of the
> buildpack:
>   https://github.com/heroku/predictionio-buildpack/blob/
> master/bin/common/setup-runtime#L50
>
> Currently they are:
>
>    - hadoop-aws-2.7.3
>    - aws-java-sdk-1.7.4
>
> If you fork the buildpack, those download URLs (currently hosted in my S3
> bucket) can be changed.
>
>
>
>> You are also right that you can modify the class path in
>> bin/compute-classpath.sh as a short term fix. The current order is
>> following the output of your target system's `ls`, so the order is not
>> guaranteed like you speak. Right before the last line (echo $CLASSPATH),
>> you can add a line to make sure that the JAR you want to be loaded first is
>> at the very beginning.
>>
>> I believe this would need to be something that is edited on master (
>> https://github.com/apache/predictionio) as the buildpack leverages it
>> vs. a cloned version of the code that I can edit, am I thinking through
>> that correctly? I may need to circle back with Mars to see if there are any
>> other options to get this to work with Heroku. Is this something that can
>> be committed to master? The aws-java-sdk.jar is the one that we need to
>> load first.
>>
>
> If you build your own custom binary distribution of PredictionIO 0.12 and
> upload it to a publicly accessible URL, then the buildpack can be
> configured to use it by setting config var `PREDICTIONIO_DIST_URL` which is
> used by the same setup script as above:
>   https://github.com/heroku/predictionio-buildpack/blob/
> master/bin/common/setup-runtime#L82
>
> This is how I tested my pre-release builds on Heroku.
>
> --
> *Mars Hall
> 415-818-7039 <(415)%20818-7039>
> Customer Facing Architect
> Salesforce Platform / Heroku
> San Francisco, California
>

Re: Issue with loading dependencies and jars

Posted by Mars Hall <ma...@salesforce.com>.

On Tue, Mar 6, 2018 at 11:39 PM, Shane Johnson <sh...@liftiq.com> wrote:

>
> Do you know the version of hadoop-aws.jar and aws-java-sdk.jar that you
> are using?
>
> I do not know what version is being used. Is this something that I can
> specify or control? I am using the PredictionIO buildpack
> https://github.com/heroku/predictionio-buildpack. I am *not* specifying
> these in my build.sbt currently.
>

The versions are specified in the "bin/common/setup-runtime" script of the
buildpack:

https://github.com/heroku/predictionio-buildpack/blob/master/bin/common/setup-runtime#L50

Currently they are:

   - hadoop-aws-2.7.3
   - aws-java-sdk-1.7.4

If you fork the buildpack, those download URLs (currently hosted in my S3
bucket) can be changed.



> You are also right that you can modify the class path in
> bin/compute-classpath.sh as a short term fix. The current order is
> following the output of your target system's `ls`, so the order is not
> guaranteed like you speak. Right before the last line (echo $CLASSPATH),
> you can add a line to make sure that the JAR you want to be loaded first is
> at the very beginning.
>
> I believe this would need to be something that is edited on master (
> https://github.com/apache/predictionio) as the buildpack leverages it vs.
> a cloned version of the code that I can edit, am I thinking through that
> correctly? I may need to circle back with Mars to see if there are any
> other options to get this to work with Heroku. Is this something that can
> be committed to master? The aws-java-sdk.jar is the one that we need to
> load first.
>

If you build your own custom binary distribution of PredictionIO 0.12 and
upload it to a publicly accessible URL, then the buildpack can be
configured to use it by setting config var `PREDICTIONIO_DIST_URL` which is
used by the same setup script as above:

https://github.com/heroku/predictionio-buildpack/blob/master/bin/common/setup-runtime#L82

This is how I tested my pre-release builds on Heroku.

-- 
*Mars Hall
415-818-7039
Customer Facing Architect
Salesforce Platform / Heroku
San Francisco, California

Re: Issue with loading dependencies and jars

Posted by Shane Johnson <sh...@liftiq.com>.

Thanks for the response Donald.

Do you know the version of hadoop-aws.jar and aws-java-sdk.jar that you are
using?

I do not know what version is being used. Is this something that I can
specify or control? I am using the PredictionIO buildpack
https://github.com/heroku/predictionio-buildpack. I am *not* specifying
these in my build.sbt currently.

You are also right that you can modify the class path in
bin/compute-classpath.sh as a short term fix. The current order is
following the output of your target system's `ls`, so the order is not
guaranteed like you speak. Right before the last line (echo $CLASSPATH),
you can add a line to make sure that the JAR you want to be loaded first is
at the very beginning.

I believe this would need to be something that is edited on master (
https://github.com/apache/predictionio) as the buildpack leverages it vs. a
cloned version of the code that I can edit, am I thinking through that
correctly? I may need to circle back with Mars to see if there are any
other options to get this to work with Heroku. Is this something that can
be committed to master? The aws-java-sdk.jar is the one that we need to
load first.

Best,

Shane


*Shane Johnson | LIFT IQ*
*Founder | CEO*

*www.liftiq.com <http://www.liftiq.com/>* or *shane@liftiq.com
<sh...@liftiq.com>*
mobile: (801) 360-3350
LinkedIn <https://www.linkedin.com/in/shanewjohnson/>  |  Twitter
<https://twitter.com/SWaldenJ> |  Facebook
<https://www.facebook.com/shane.johnson.71653>



On Tue, Mar 6, 2018 at 4:08 PM, Donald Szeto <do...@apache.org> wrote:

> Hi Shane,
>
> Do you know the version of hadoop-aws.jar and aws-java-sdk.jar that you
> are using?
>
> You are also right that you can modify the class path in
> bin/compute-classpath.sh as a short term fix. The current order is
> following the output of your target system's `ls`, so the order is not
> guaranteed like you speak. Right before the last line (echo $CLASSPATH),
> you can add a line to make sure that the JAR you want to be loaded first is
> at the very beginning.
>
> The long term fix would be matching the hadoop-aws.jar, aws-java-sdk.jar,
> and PIO's S3 storage JAR version to have a binary compatible transitive
> dependency tree. This could be quite time consuming though.
>
> Regards,
> Donald
>
> On Mon, Mar 5, 2018 at 6:57 AM, Shane Johnson <
> shanewaldenjohnson@gmail.com> wrote:
>
>> Hi team,
>>
>> I have been working with Mars to get a little more insight to an error I
>> am seeing. After narrowing down the issue, I am trying to find a way to
>> force the order of loading the jars. I am a novice in Scala and am looking
>> for a way to continue leveraging PredictionIO on Heroku but have run into a
>> roadblock as I believe ordering the jars would be a change to PredictionIO
>> itself. I would greatly appreciate your expertise and guidance.
>>
>> Error:
>>
>> Exception
>>
>>  in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>>
>> 2018-03-02T02:49:42.245083+00:00 app[release.7253]: 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>>
>> 2018-03-02T02:49:42.245086+00:00 app[release.7253]: 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>
>> 2018-03-02T02:49:42.245087+00:00 app[release.7253]: 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>>
>> 2018-03-02T02:49:42.245088+00:00 app[release.7253]: 	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
>>
>> 2018-03-02T02:49:42.245090+00:00 app[release.7253]: 	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
>>
>>
>> Context:
>>
>>    - I am using the predictionIO buildpack on heroku and using version
>>    12.
>>    - I am using S3 to Persist the model objects as I am using the
>>    PAlgorithm vs L2LAlgorithm.
>>    - I see an intermittent error when the Algorithm.scala file tries to
>>    connect to S3. Sometimes it works and sometime is does not.
>>    - When it fails I get the message above - I think I have narrowed
>>    this down to the ordering of loading the jars.
>>
>> *I have done a number of tests and can see that the problem always occurs
>> with a certain order of jars. This issue appears to be intermittent and
>> could be solved by forcing the order of the jars. I am assuming this could
>> be a straightforward fix, *
>> *I believe it is in the compute-classpath.sh. Perhaps it's something I
>> can even do on my end as an end user but I don't believe that to be the
>> case.*
>>
>> *Please let me know if this is something that can be changed and
>> committed or if I can contribute in someway. Again, I am fairly new to
>> Scala and big data infrastructure so I am intimidated by the thought of
>> making the change myself but we are at a roadblock:)*
>>
>> *When aws-java-sdk.jar is loaded before
>> (hadoop-aws.jar and pio-data-s3-assembly-0.12.0-incubating.jar) it is
>> always successful. If it is loaded after it fails with the message above.*
>>
>> *Success Examples:*
>>
>> --jars
>>   file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
>>   file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
>>   file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0
>> .12.0-incubating.jar,
>>   ...
>>
>> --jars
>>   file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
>>   file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0
>> .12.0-incubating.jar,
>>   file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
>>   ...
>>
>> *Failure Examples:*
>>
>> --jars
>>   file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
>>   file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0
>> .12.0-incubating.jar,
>>   file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
>>   ...
>>
>> --jars
>>   file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-0
>> .12.0-incubating.jar,
>>   file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
>>   file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
>>   ...
>>
>>
>> Thank you again for your help!
>>
>> *Shane Johnson | 801.360.3350 <(801)%20360-3350>*
>> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook
>> <https://www.facebook.com/shane.johnson.71653>
>>
>
>

Re: Issue with loading dependencies and jars

Posted by Donald Szeto <do...@apache.org>.

Hi Shane,

Do you know the version of hadoop-aws.jar and aws-java-sdk.jar that you are
using?

You are also right that you can modify the class path in
bin/compute-classpath.sh as a short term fix. The current order is
following the output of your target system's `ls`, so the order is not
guaranteed like you speak. Right before the last line (echo $CLASSPATH),
you can add a line to make sure that the JAR you want to be loaded first is
at the very beginning.

The long term fix would be matching the hadoop-aws.jar, aws-java-sdk.jar,
and PIO's S3 storage JAR version to have a binary compatible transitive
dependency tree. This could be quite time consuming though.

Regards,
Donald

On Mon, Mar 5, 2018 at 6:57 AM, Shane Johnson <sh...@gmail.com>
wrote:

> Hi team,
>
> I have been working with Mars to get a little more insight to an error I
> am seeing. After narrowing down the issue, I am trying to find a way to
> force the order of loading the jars. I am a novice in Scala and am looking
> for a way to continue leveraging PredictionIO on Heroku but have run into a
> roadblock as I believe ordering the jars would be a change to PredictionIO
> itself. I would greatly appreciate your expertise and guidance.
>
> Error:
>
> Exception
>
>  in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
>
> 2018-03-02T02:49:42.245083+00:00 app[release.7253]: 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
>
> 2018-03-02T02:49:42.245086+00:00 app[release.7253]: 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>
> 2018-03-02T02:49:42.245087+00:00 app[release.7253]: 	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
>
> 2018-03-02T02:49:42.245088+00:00 app[release.7253]: 	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
>
> 2018-03-02T02:49:42.245090+00:00 app[release.7253]: 	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
>
>
> Context:
>
>    - I am using the predictionIO buildpack on heroku and using version 12.
>    - I am using S3 to Persist the model objects as I am using the
>    PAlgorithm vs L2LAlgorithm.
>    - I see an intermittent error when the Algorithm.scala file tries to
>    connect to S3. Sometimes it works and sometime is does not.
>    - When it fails I get the message above - I think I have narrowed this
>    down to the ordering of loading the jars.
>
> *I have done a number of tests and can see that the problem always occurs
> with a certain order of jars. This issue appears to be intermittent and
> could be solved by forcing the order of the jars. I am assuming this could
> be a straightforward fix, *
> *I believe it is in the compute-classpath.sh. Perhaps it's something I can
> even do on my end as an end user but I don't believe that to be the case.*
>
> *Please let me know if this is something that can be changed and committed
> or if I can contribute in someway. Again, I am fairly new to Scala and big
> data infrastructure so I am intimidated by the thought of making the change
> myself but we are at a roadblock:)*
>
> *When aws-java-sdk.jar is loaded before
> (hadoop-aws.jar and pio-data-s3-assembly-0.12.0-incubating.jar) it is
> always successful. If it is loaded after it fails with the message above.*
>
> *Success Examples:*
>
> --jars
>   file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
>   file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
>   file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-
> 0.12.0-incubating.jar,
>   ...
>
> --jars
>   file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
>   file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-
> 0.12.0-incubating.jar,
>   file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
>   ...
>
> *Failure Examples:*
>
> --jars
>   file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
>   file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-
> 0.12.0-incubating.jar,
>   file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
>   ...
>
> --jars
>   file:/app/PredictionIO-dist/lib/spark/pio-data-s3-assembly-
> 0.12.0-incubating.jar,
>   file:/app/PredictionIO-dist/lib/spark/hadoop-aws.jar,
>   file:/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar,
>   ...
>
>
> Thank you again for your help!
>
> *Shane Johnson | 801.360.3350 <(801)%20360-3350>*
> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook
> <https://www.facebook.com/shane.johnson.71653>
>