[jira] [Comment Edited] (MAHOUT-2023) Drivers broken, scopt classes not found

["Pat Ferrel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAHOUT-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16258255#comment-16258255 ] 

Pat Ferrel edited comment on MAHOUT-2023 at 11/18/17 11:32 PM:
---------------------------------------------------------------

ok not that MAHOUT-2020 is resolved, I looked at the scopt issue and found:
* all the correct scopt artifact exist in remote repos for all scala versions and they are being found by the mahout build.
* the ids for artifact etc are correct as per ^^^
* I checked all the tagged versions of Mahout back to 12.0. Not sure when the drivers stopped working but there has been no change to any reference to scopt in any POM. And since people have been using it and asking questions on the mailing list I will assume that up till the last build changes the drivers worked.
* The vienna-cl and java to c bindings are in the assembly pom so these classes are getting to the Spark Executors.
* I've checked compute-classpath.sh and the mahout script where changes were small and not relevant. 
* I've looked at the contents of the mahout*dependency-reduced.jar, which should have the things listed below and it does not, in only had guava, apache commons and fastutils. It is supposed to have:

  <dependencySets>
    <dependencySet>
      <unpack>true</unpack>
      <unpackOptions>
      <!-- MAHOUT-1126 -->
      <excludes>
         <exclude>META-INF/LICENSE</exclude>
      </excludes>
      </unpackOptions>
      <scope>runtime</scope>
      <outputDirectory>/</outputDirectory>
      <useTransitiveFiltering>true</useTransitiveFiltering>
      <includes>
        <!-- guava only included to get Preconditions in mahout-math and mahout-hdfs -->
        <include>com.google.guava:guava</include>
        <include>com.github.scopt_${scala.compat.version}</include>
        <include>com.tdunning:t-digest</include>
        <include>org.apache.commons:commons-math3</include>
        <include>it.unimi.dsi:fastutil</include>
        <include>org.apache.mahout:mahout-native-viennacl_${scala.compat.version}</include>
        <include>org.apache.mahout:mahout-native-viennacl-omp_${scala.compat.version}</include>
        <include>org.bytedeco:javacpp</include>
      </includes>
    </dependencySet>
  

This all leads me to believe that something in the build no longer makes that dependency-reduced.jar available to the Java Driver code since those other libs in the assembly are probably all hadoop or Spark Executor code, not needed in the Mahout driver. This is likely to have been a side effect of the build refactoring

[~rawkintrevo_apache] does "dependencies-reduced.jar" which contains Scopt get its scala.compat.version fixed? It seems like the jar is missing anything with scala.compat.version but this may be a red herring.




was (Author: pferrel):
ok not that MAHOUT-2020 is resolved, I looked at the scopt issue and found:
* all the correct scopt artifact exist in remote repos for all scala versions and they are being found by the mahout build.
* the ids for artifact etc are correct as per ^^^
* I checked all the tagged versions of Mahout back to 12.0. Not sure when the drivers stopped working but there has been no change to any reference to scopt in any POM. And since people have been using it and asking questions on the mailing list I will assume that up till the last build changes the drivers worked.
* The vienna-cl and java to c bindings are in the assembly pom so these classes are getting to the Spark Executors.
* I've checked compute-classpath.sh and the mahout script where changes were small and not relevant. 
* I've looked at the contents of the mahout*dependency-reduced.jar, which should have the things listed below and it does not, in only had guava, apache commons and fastutils. It is supposed to have:

 {{ <dependencySets>
    <dependencySet>
      <unpack>true</unpack>
      <unpackOptions>
      <!-- MAHOUT-1126 -->
      <excludes>
         <exclude>META-INF/LICENSE</exclude>
      </excludes>
      </unpackOptions>
      <scope>runtime</scope>
      <outputDirectory>/</outputDirectory>
      <useTransitiveFiltering>true</useTransitiveFiltering>
      <includes>
        <!-- guava only included to get Preconditions in mahout-math and mahout-hdfs -->
        <include>com.google.guava:guava</include>
        <include>com.github.scopt_${scala.compat.version}</include>
        <include>com.tdunning:t-digest</include>
        <include>org.apache.commons:commons-math3</include>
        <include>it.unimi.dsi:fastutil</include>
        <include>org.apache.mahout:mahout-native-viennacl_${scala.compat.version}</include>
        <include>org.apache.mahout:mahout-native-viennacl-omp_${scala.compat.version}</include>
        <include>org.bytedeco:javacpp</include>
      </includes>
    </dependencySet>
  }}

This all leads me to believe that something in the build no longer makes that dependency-reduced.jar available to the Java Driver code since those other libs in the assembly are probably all hadoop or Spark Executor code, not needed in the Mahout driver. This is likely to have been a side effect of the build refactoring

[~rawkintrevo_apache] does "dependencies-reduced.jar" which contains Scopt get its scala.compat.version fixed? This doesn't seem to be the problem but is aa question nonetheless.



> Drivers broken, scopt classes not found
> ---------------------------------------
>
>                 Key: MAHOUT-2023
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-2023
>             Project: Mahout
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.13.1
>         Environment: any
>            Reporter: Pat Ferrel
>            Assignee: Pat Ferrel
>            Priority: Blocker
>             Fix For: 0.13.1
>
>
> Type `mahout spark-itemsimilarity` after Mahout is installed properly and you get a fatal exception due to missing scopt classes.
> Probably a build issue related to incorrect versions of scopt being looked for.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)