You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mengxr <gi...@git.apache.org> on 2015/03/26 03:32:43 UTC
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
GitHub user mengxr opened a pull request:
https://github.com/apache/spark/pull/5196
[SPARK-5885][MLLIB] Add VectorAssembler as a feature transformer
VectorAssembler merges multiple columns into a vector column. This PR contains content from #5195.
- [ ] carry ML attributes
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mengxr/spark SPARK-5885
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5196.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5196
----
commit 4ff16cef4fe82555191403e980cd2fd438441d3b
Author: Xiangrui Meng <me...@databricks.com>
Date: 2015-03-25T23:19:16Z
add VectorAssembler
commit 85f3106fb422f872747c4be7b0a83cc5b2ec8930
Author: Xiangrui Meng <me...@databricks.com>
Date: 2015-03-26T01:58:10Z
add CreateStruct
commit adb71c4a941e848cb9219c44addfc857ae072e0b
Author: Xiangrui Meng <me...@databricks.com>
Date: 2015-03-26T01:58:47Z
Merge branch 'SPARK-6542' into SPARK-5885
commit 29fb6acbb79a80893cc86b0a3788718212343617
Author: Xiangrui Meng <me...@databricks.com>
Date: 2015-03-26T02:24:13Z
use CreateStruct
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5196#issuecomment-91483359
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30018/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5196#issuecomment-86305463
[Test build #29199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29199/consoleFull) for PR 5196 at commit [`29fb6ac`](https://github.com/apache/spark/commit/29fb6acbb79a80893cc86b0a3788718212343617).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/5196#issuecomment-86361537
The build failure was caused by codegen. When the `ExpressionEvaluationSuite.CreateStruct` test case is executed separately, codegen is not enabled by default, and `CreateStruct.dataType` is not touched. But on Jenkins, some test suite executed earlier enabled codegen, then `CreateStruct.dataType` got called and threw exception because `BoundReference` used in the test case if not `NamedExpression`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5196#issuecomment-86324803
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29199/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5196#issuecomment-91462798
[Test build #30018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30018/consoleFull) for PR 5196 at commit [`a52b101`](https://github.com/apache/spark/commit/a52b10162a4d1cca9cb86d726771b481844499c9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5196#discussion_r28126284
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.AlphaComponent
+import org.apache.spark.ml.Transformer
+import org.apache.spark.ml.param.{HasInputCols, HasOutputCol, ParamMap}
+import org.apache.spark.mllib.linalg.{Vector, VectorUDT, Vectors}
+import org.apache.spark.sql.{Column, DataFrame, Row}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.CreateStruct
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * :: AlphaComponent ::
+ * A feature transformer than merge multiple columns into a vector column.
+ */
+@AlphaComponent
+class VectorAssembler extends Transformer with HasInputCols with HasOutputCol {
+
+ /** @group setParam */
+ def setInputCols(value: Array[String]): this.type = set(inputCols, value)
+
+ /** @group setParam */
+ def setOutputCol(value: String): this.type = set(outputCol, value)
+
+ override def transform(dataset: DataFrame, paramMap: ParamMap): DataFrame = {
+ val map = this.paramMap ++ paramMap
+ val assembleFunc = udf { r: Row =>
+ VectorAssembler.assemble(r.toSeq: _*)
+ }
+ val args = map(inputCols).map(c => UnresolvedAttribute(c))
+ dataset.select(col("*"), assembleFunc(new Column(CreateStruct(args))).as(map(outputCol)))
+ }
+
+ override def transformSchema(schema: StructType, paramMap: ParamMap): StructType = {
+ val map = this.paramMap ++ paramMap
+ val inputColNames = map(inputCols)
+ val outputColName = map(outputCol)
+ val inputDataTypes = inputColNames.map(name => schema(name).dataType)
+ for (dataType <- inputDataTypes) {
+ if (!(dataType == DoubleType || dataType.isInstanceOf[VectorUDT])) {
+ throw new IllegalArgumentException(s"Data type $dataType is not supported.")
+ }
+ }
+ if (schema.fieldNames.contains(outputColName)) {
+ throw new IllegalArgumentException(s"Output column $outputColName already exists.")
+ }
+ StructType(schema.fields :+ new StructField(outputColName, new VectorUDT, false))
+ }
+}
+
+@AlphaComponent
+object VectorAssembler {
+
+ private[feature] def assemble(vv: Any*): Vector = {
+ val indices = ArrayBuilder.make[Int]
+ val values = ArrayBuilder.make[Double]
+ var cur = 0
+ vv.foreach {
+ case v: Double =>
--- End diff --
Done with casting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by petro-rudenko <gi...@git.apache.org>.
Github user petro-rudenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/5196#discussion_r27739585
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.AlphaComponent
+import org.apache.spark.ml.Transformer
+import org.apache.spark.ml.param.{HasInputCols, HasOutputCol, ParamMap}
+import org.apache.spark.mllib.linalg.{Vector, VectorUDT, Vectors}
+import org.apache.spark.sql.{Column, DataFrame, Row}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.CreateStruct
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * :: AlphaComponent ::
+ * A feature transformer than merge multiple columns into a vector column.
+ */
+@AlphaComponent
+class VectorAssembler extends Transformer with HasInputCols with HasOutputCol {
+
+ /** @group setParam */
+ def setInputCols(value: Array[String]): this.type = set(inputCols, value)
+
+ /** @group setParam */
+ def setOutputCol(value: String): this.type = set(outputCol, value)
+
+ override def transform(dataset: DataFrame, paramMap: ParamMap): DataFrame = {
+ val map = this.paramMap ++ paramMap
+ val assembleFunc = udf { r: Row =>
+ VectorAssembler.assemble(r.toSeq: _*)
+ }
+ val args = map(inputCols).map(c => UnresolvedAttribute(c))
+ dataset.select(col("*"), assembleFunc(new Column(CreateStruct(args))).as(map(outputCol)))
+ }
+
+ override def transformSchema(schema: StructType, paramMap: ParamMap): StructType = {
+ val map = this.paramMap ++ paramMap
+ val inputColNames = map(inputCols)
+ val outputColName = map(outputCol)
+ val inputDataTypes = inputColNames.map(name => schema(name).dataType)
+ for (dataType <- inputDataTypes) {
+ if (!(dataType == DoubleType || dataType.isInstanceOf[VectorUDT])) {
+ throw new IllegalArgumentException(s"Data type $dataType is not supported.")
+ }
+ }
+ if (schema.fieldNames.contains(outputColName)) {
+ throw new IllegalArgumentException(s"Output column $outputColName already exists.")
+ }
+ StructType(schema.fields :+ new StructField(outputColName, new VectorUDT, false))
+ }
+}
+
+@AlphaComponent
+object VectorAssembler {
+
+ private[feature] def assemble(vv: Any*): Vector = {
+ val indices = ArrayBuilder.make[Int]
+ val values = ArrayBuilder.make[Double]
+ var cur = 0
+ vv.foreach {
+ case v: Double =>
--- End diff --
Would be good to support Integers also and just convert them to double.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/5196#issuecomment-92220404
Merged into master. Will send a follow-up PR to handle attributes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5196#discussion_r27191305
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypes.scala ---
@@ -142,3 +142,29 @@ case class CreateArray(children: Seq[Expression]) extends Expression {
override def toString: String = s"Array(${children.mkString(",")})"
}
+
+/**
+ * Returns a Row containing the evaluation of all children expressions.
+ */
+case class CreateStruct(children: Seq[Expression]) extends Expression {
--- End diff --
Would be better to enforce the type of `children` to be `Seq[NamedExpression]`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5196#issuecomment-91483350
[Test build #30018 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30018/consoleFull) for PR 5196 at commit [`a52b101`](https://github.com/apache/spark/commit/a52b10162a4d1cca9cb86d726771b481844499c9).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class VectorAssembler extends Transformer with HasInputCols with HasOutputCol `
* `class SparseMatrix(Matrix):`
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by petro-rudenko <gi...@git.apache.org>.
Github user petro-rudenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/5196#discussion_r27645880
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.AlphaComponent
+import org.apache.spark.ml.Transformer
+import org.apache.spark.ml.param.{HasInputCols, HasOutputCol, ParamMap}
+import org.apache.spark.mllib.linalg.{Vector, VectorUDT, Vectors}
+import org.apache.spark.sql.{Column, DataFrame, Row}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.CreateStruct
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * :: AlphaComponent ::
+ * A feature transformer than merge multiple columns into a vector column.
+ */
+@AlphaComponent
+class VectorAssembler extends Transformer with HasInputCols with HasOutputCol {
--- End diff --
Maybe call it FeatureUnion to keep the same semantic with [sklearn](http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.FeatureUnion.html)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5196#discussion_r27191279
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypes.scala ---
@@ -142,3 +142,29 @@ case class CreateArray(children: Seq[Expression]) extends Expression {
override def toString: String = s"Array(${children.mkString(",")})"
}
+
+/**
+ * Returns a Row containing the evaluation of all children expressions.
+ */
+case class CreateStruct(children: Seq[Expression]) extends Expression {
+ override type EvaluatedType = Row
+
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ override lazy val resolved: Boolean = childrenResolved
+
+ override def dataType: StructType = {
+ assert(resolved, s"CreateStruct is called with unresolved children: $children.")
--- End diff --
The error message is confusing. `CreateStruct` *can* be called with unresolved children (and you actually did this [here] [1]), but `CreateStruct.dataType` can't, because child types may be unknown.
[1]: https://github.com/apache/spark/pull/5196/files#diff-9c84e1d27f25714e256cb482069359cfR52
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5196#issuecomment-86324798
[Test build #29199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29199/consoleFull) for PR 5196 at commit [`29fb6ac`](https://github.com/apache/spark/commit/29fb6acbb79a80893cc86b0a3788718212343617).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class VectorAssembler extends Transformer with HasInputCols with HasOutputCol `
* `case class CreateStruct(children: Seq[Expression]) extends Expression `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/5196#issuecomment-86368320
Actually, the failed test suites are `GeneratedEvaluationSuite` and `GeneratedMutableEvaluationSuite`, both of which just extend `ExpressionEvaluationSuite`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5196#discussion_r27927014
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.AlphaComponent
+import org.apache.spark.ml.Transformer
+import org.apache.spark.ml.param.{HasInputCols, HasOutputCol, ParamMap}
+import org.apache.spark.mllib.linalg.{Vector, VectorUDT, Vectors}
+import org.apache.spark.sql.{Column, DataFrame, Row}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.CreateStruct
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * :: AlphaComponent ::
+ * A feature transformer than merge multiple columns into a vector column.
+ */
+@AlphaComponent
+class VectorAssembler extends Transformer with HasInputCols with HasOutputCol {
--- End diff --
This is not exactly sklearn's FeatureUnion that takes transformers. `VectorAssembler` merges arbitrary columns. I'm a little worried that if we use the same name people would expect the same behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5196
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5196#discussion_r27191304
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvaluationSuite.scala ---
@@ -1080,4 +1080,11 @@ class ExpressionEvaluationSuite extends FunSuite {
checkEvaluation(c1 ^ c2, 3, row)
checkEvaluation(~c1, -2, row)
}
+
+ test("CreateStruct") {
+ val row = Row(1, 2, 3)
+ val c1 = 'a.int.at(0)
--- End diff --
`BoundReference` is not `NamedExpression`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5885][MLLIB] Add VectorAssembler as a f...
Posted by oefirouz <gi...@git.apache.org>.
Github user oefirouz commented on a diff in the pull request:
https://github.com/apache/spark/pull/5196#discussion_r27684795
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/VectorAssembler.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.collection.mutable.ArrayBuilder
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.AlphaComponent
+import org.apache.spark.ml.Transformer
+import org.apache.spark.ml.param.{HasInputCols, HasOutputCol, ParamMap}
+import org.apache.spark.mllib.linalg.{Vector, VectorUDT, Vectors}
+import org.apache.spark.sql.{Column, DataFrame, Row}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.expressions.CreateStruct
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types._
+
+/**
+ * :: AlphaComponent ::
+ * A feature transformer than merge multiple columns into a vector column.
+ */
+@AlphaComponent
+class VectorAssembler extends Transformer with HasInputCols with HasOutputCol {
--- End diff --
+1. I like the idea of trying to keep naming and architecture as similar to sklearn as possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org