You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by janardhan shetty <ja...@gmail.com> on 2016/09/06 18:26:45 UTC
Re: Using spark package XGBoost
Is this merged to Spark ML ? If so which version ?
On Tue, Sep 6, 2016 at 12:58 AM, Takeshi Yamamuro <li...@gmail.com>
wrote:
> Hi,
>
> Sorry to bother you, but I'ld like to inform you our activities.
> We'll start incubating our product, Hivemall, in Apache and this is a
> scalable ML library
> for Hive/Spark/Pig.
>
> - http://wiki.apache.org/incubator/HivemallProposal
> - http://markmail.org/thread/mjwyyd4btthk3626
>
> I made a pr for XGBoost integration on DataFrame/Spark(https://
> github.com/myui/hivemall/pull/281)
> and this pr has already been merged in a master.
> I wrote how to use the integration on my gist:
> https://gist.github.com/maropu/33794b293ee937e99b8fb0788843fa3f
>
> If you are interested in the integration, could you please you try using
> it and
> let me know the issues that you get stuck in?
>
> Best regards,
> takeshi
>
> // maropu
>
>
>
> On Mon, Aug 15, 2016 at 1:04 PM, Brandon White <bw...@gmail.com>
> wrote:
>
>> The XGBoost integration with Spark is currently only supported for RDDs,
>> there is a ticket for dataframe and folks calm to be working on it.
>>
>> On Aug 14, 2016 8:15 PM, "Jacek Laskowski" <ja...@japila.pl> wrote:
>>
>>> Hi,
>>>
>>> I've never worked with the library and speaking about sbt setup only.
>>>
>>> It appears that the project didn't release 2.11-compatible jars (only
>>> 2.10) [1] so you need to build the project yourself and uber-jar it
>>> (using sbt-assembly plugin).
>>>
>>> [1] https://spark-packages.org/package/rotationsymmetry/sparkxgboost
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> ----
>>> https://medium.com/@jaceklaskowski/
>>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>>> Follow me at https://twitter.com/jaceklaskowski
>>>
>>>
>>> On Sun, Aug 14, 2016 at 7:13 AM, janardhan shetty
>>> <ja...@gmail.com> wrote:
>>> > Any leads how to do acheive this?
>>> >
>>> > On Aug 12, 2016 6:33 PM, "janardhan shetty" <ja...@gmail.com>
>>> wrote:
>>> >>
>>> >> I tried using sparkxgboost package in build.sbt file but it failed.
>>> >> Spark 2.0
>>> >> Scala 2.11.8
>>> >>
>>> >> Error:
>>> >> [warn]
>>> >> http://dl.bintray.com/spark-packages/maven/rotationsymmetry/
>>> sparkxgboost/0.2.1-s_2.10/sparkxgboost-0.2.1-s_2.10-javadoc.jar
>>> >> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> >> [warn] :: FAILED DOWNLOADS ::
>>> >> [warn] :: ^ see resolution messages for details ^ ::
>>> >> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> >> [warn] ::
>>> >> rotationsymmetry#sparkxgboost;0.2.1-s_2.10!sparkxgboost.jar(src)
>>> >> [warn] ::
>>> >> rotationsymmetry#sparkxgboost;0.2.1-s_2.10!sparkxgboost.jar(doc)
>>> >>
>>> >> build.sbt:
>>> >>
>>> >> scalaVersion := "2.11.8"
>>> >>
>>> >> libraryDependencies ++= {
>>> >> val sparkVersion = "2.0.0-preview"
>>> >> Seq(
>>> >> "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
>>> >> "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
>>> >> "org.apache.spark" %% "spark-streaming" % sparkVersion %
>>> "provided",
>>> >> "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided"
>>> >> )
>>> >> }
>>> >>
>>> >> resolvers += "Spark Packages Repo" at
>>> >> "http://dl.bintray.com/spark-packages/maven"
>>> >>
>>> >> libraryDependencies += "rotationsymmetry" % "sparkxgboost" %
>>> >> "0.2.1-s_2.10"
>>> >>
>>> >> assemblyMergeStrategy in assembly := {
>>> >> case PathList("META-INF", "MANIFEST.MF") =>
>>> >> MergeStrategy.discard
>>> >> case PathList("javax", "servlet", xs @ _*) =>
>>> >> MergeStrategy.first
>>> >> case PathList(ps @ _*) if ps.last endsWith ".html" =>
>>> >> MergeStrategy.first
>>> >> case "application.conf" =>
>>> >> MergeStrategy.concat
>>> >> case "unwanted.txt" =>
>>> >> MergeStrategy.discard
>>> >>
>>> >> case x => val oldStrategy = (assemblyMergeStrategy in
>>> assembly).value
>>> >> oldStrategy(x)
>>> >>
>>> >> }
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Fri, Aug 12, 2016 at 3:35 PM, janardhan shetty <
>>> janardhanp22@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Is there a dataframe version of XGBoost in spark-ml ?.
>>> >>> Has anyone used sparkxgboost package ?
>>> >>
>>> >>
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>
>>>
>
>
> --
> ---
> Takeshi Yamamuro
>
Re: Using spark package XGBoost
Posted by janardhan shetty <ja...@gmail.com>.
Tried to implement spark package in 2.0
https://spark-packages.org/package/rotationsymmetry/sparkxgboost
but it is throwing the error:
error: not found: type SparkXGBoostClassifier
On Tue, Sep 6, 2016 at 11:26 AM, janardhan shetty <ja...@gmail.com>
wrote:
> Is this merged to Spark ML ? If so which version ?
>
> On Tue, Sep 6, 2016 at 12:58 AM, Takeshi Yamamuro <li...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Sorry to bother you, but I'ld like to inform you our activities.
>> We'll start incubating our product, Hivemall, in Apache and this is a
>> scalable ML library
>> for Hive/Spark/Pig.
>>
>> - http://wiki.apache.org/incubator/HivemallProposal
>> - http://markmail.org/thread/mjwyyd4btthk3626
>>
>> I made a pr for XGBoost integration on DataFrame/Spark(https://github
>> .com/myui/hivemall/pull/281)
>> and this pr has already been merged in a master.
>> I wrote how to use the integration on my gist:
>> https://gist.github.com/maropu/33794b293ee937e99b8fb0788843fa3f
>>
>> If you are interested in the integration, could you please you try using
>> it and
>> let me know the issues that you get stuck in?
>>
>> Best regards,
>> takeshi
>>
>> // maropu
>>
>>
>>
>> On Mon, Aug 15, 2016 at 1:04 PM, Brandon White <bw...@gmail.com>
>> wrote:
>>
>>> The XGBoost integration with Spark is currently only supported for RDDs,
>>> there is a ticket for dataframe and folks calm to be working on it.
>>>
>>> On Aug 14, 2016 8:15 PM, "Jacek Laskowski" <ja...@japila.pl> wrote:
>>>
>>>> Hi,
>>>>
>>>> I've never worked with the library and speaking about sbt setup only.
>>>>
>>>> It appears that the project didn't release 2.11-compatible jars (only
>>>> 2.10) [1] so you need to build the project yourself and uber-jar it
>>>> (using sbt-assembly plugin).
>>>>
>>>> [1] https://spark-packages.org/package/rotationsymmetry/sparkxgboost
>>>>
>>>> Pozdrawiam,
>>>> Jacek Laskowski
>>>> ----
>>>> https://medium.com/@jaceklaskowski/
>>>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>>>> Follow me at https://twitter.com/jaceklaskowski
>>>>
>>>>
>>>> On Sun, Aug 14, 2016 at 7:13 AM, janardhan shetty
>>>> <ja...@gmail.com> wrote:
>>>> > Any leads how to do acheive this?
>>>> >
>>>> > On Aug 12, 2016 6:33 PM, "janardhan shetty" <ja...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> I tried using sparkxgboost package in build.sbt file but it failed.
>>>> >> Spark 2.0
>>>> >> Scala 2.11.8
>>>> >>
>>>> >> Error:
>>>> >> [warn]
>>>> >> http://dl.bintray.com/spark-packages/maven/rotationsymmetry/
>>>> sparkxgboost/0.2.1-s_2.10/sparkxgboost-0.2.1-s_2.10-javadoc.jar
>>>> >> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>>> >> [warn] :: FAILED DOWNLOADS ::
>>>> >> [warn] :: ^ see resolution messages for details ^ ::
>>>> >> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>>> >> [warn] ::
>>>> >> rotationsymmetry#sparkxgboost;0.2.1-s_2.10!sparkxgboost.jar(src)
>>>> >> [warn] ::
>>>> >> rotationsymmetry#sparkxgboost;0.2.1-s_2.10!sparkxgboost.jar(doc)
>>>> >>
>>>> >> build.sbt:
>>>> >>
>>>> >> scalaVersion := "2.11.8"
>>>> >>
>>>> >> libraryDependencies ++= {
>>>> >> val sparkVersion = "2.0.0-preview"
>>>> >> Seq(
>>>> >> "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
>>>> >> "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
>>>> >> "org.apache.spark" %% "spark-streaming" % sparkVersion %
>>>> "provided",
>>>> >> "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided"
>>>> >> )
>>>> >> }
>>>> >>
>>>> >> resolvers += "Spark Packages Repo" at
>>>> >> "http://dl.bintray.com/spark-packages/maven"
>>>> >>
>>>> >> libraryDependencies += "rotationsymmetry" % "sparkxgboost" %
>>>> >> "0.2.1-s_2.10"
>>>> >>
>>>> >> assemblyMergeStrategy in assembly := {
>>>> >> case PathList("META-INF", "MANIFEST.MF") =>
>>>> >> MergeStrategy.discard
>>>> >> case PathList("javax", "servlet", xs @ _*) =>
>>>> >> MergeStrategy.first
>>>> >> case PathList(ps @ _*) if ps.last endsWith ".html" =>
>>>> >> MergeStrategy.first
>>>> >> case "application.conf" =>
>>>> >> MergeStrategy.concat
>>>> >> case "unwanted.txt" =>
>>>> >> MergeStrategy.discard
>>>> >>
>>>> >> case x => val oldStrategy = (assemblyMergeStrategy in
>>>> assembly).value
>>>> >> oldStrategy(x)
>>>> >>
>>>> >> }
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Fri, Aug 12, 2016 at 3:35 PM, janardhan shetty <
>>>> janardhanp22@gmail.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> Is there a dataframe version of XGBoost in spark-ml ?.
>>>> >>> Has anyone used sparkxgboost package ?
>>>> >>
>>>> >>
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>
>>>>
>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>
>