You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by culler <gi...@git.apache.org> on 2014/11/03 04:37:40 UTC
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
GitHub user culler opened a pull request:
https://github.com/apache/spark/pull/3066
Timestamp and Date classes which work in the catalyst DSL.
These Timestamp and Date classes support the usual comparison operators, so the can be used in catalyst Expressions. The come with implicit conversions to and from the underlying Java classes. They also have a convenient initializer which accepts a String.
Additional implicit conversions fix an issue with the DSL where expressions like 0 < 'x are not recognized.
An independent change adds a method to Row for creating a row from a sequence of Strings and a schema.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/culler/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3066.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3066
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19720121
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala ---
@@ -187,7 +187,8 @@ case object NullType extends DataType
object NativeType {
val all = Seq(
- IntegerType, BooleanType, LongType, DoubleType, FloatType, ShortType, ByteType, StringType)
+ IntegerType, BooleanType, LongType, DoubleType, FloatType, ShortType,
+ ByteType, StringType, DateType, TimestampType)
--- End diff --
The original line doesn't exceed 100 columns thus doesn't require a wrap.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19784424
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnType.scala ---
@@ -372,6 +372,11 @@ private[sql] object TIMESTAMP extends NativeColumnType(TimestampType, 9, 12) {
override def setField(row: MutableRow, ordinal: Int, value: Timestamp): Unit = {
row(ordinal) = value
}
+
+ def append(v: Date, buffer: ByteBuffer) {
+ buffer.putLong(v.getTime)
+ }
+
--- End diff --
Sure? There's already an `append` method https://github.com/apache/spark/pull/3066/files#diff-b71f7eb2042698f801a345e8c7e90c5cR364 Otherwise the master branch couldn't have passed Jenkins build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19730958
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala ---
@@ -146,6 +146,31 @@ package object dsl {
def upper(e: Expression) = Upper(e)
def lower(e: Expression) = Lower(e)
+ /*
+ * Conversions to provide the standard operators in the special case
+ * where a literal is being combined with a symbol. Without these an
+ * expression such as 0 < 'x is not recognized.
+ */
+ implicit class InitialLiteral(x: Any) {
+ val literal = Literal(x)
+ def + (other: Symbol):Expression = {literal + other}
+ def - (other: Symbol):Expression = {literal - other}
+ def * (other: Symbol):Expression = {literal * other}
+ def / (other: Symbol):Expression = {literal / other}
+ def % (other: Symbol):Expression = {literal % other}
+
+ def && (other: Symbol):Expression = {literal && other}
+ def || (other: Symbol):Expression = {literal || other}
+
+ def < (other: Symbol):Expression = {literal < other}
+ def <= (other: Symbol):Expression = {literal <= other}
+ def > (other: Symbol):Expression = {literal > other}
+ def >= (other: Symbol):Expression = {literal >= other}
+ def === (other: Symbol):Expression = {literal === other}
+ def <=> (other: Symbol):Expression = {literal <=> other}
+ def !== (other: Symbol):Expression = {literal !== other}
--- End diff --
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19722086
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/package.scala ---
@@ -451,4 +451,33 @@ package object sql {
* Builder for [[Metadata]]. If there is a key collision, the latter will overwrite the former.
*/
type MetadataBuilder = catalyst.util.MetadataBuilder
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * A Date class which support the standard comparison operators, for
+ * use in DSL expressions. Implicit conversions to java.sql.Date
+ * are provided. The class intializer accepts a String, e.g.
+ *
+ * val ts = Date("2014-01-01")
--- End diff --
Please use proper Scaladoc format here, i.e.:
```
{{{
val ts = Date("2014-01-01")
}}}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61599608
Hi Cheng,
I will happily add tests.
There is one thing I could use some help with, though. It would be very
helpful if I could run the existing test suite. But I am currently unable
to compile several of the tests involving catalyst expressions.
The problem is that the compiler objects whenever a triple = ( === ) is
used inside of an assert command. I realize that this is a feature of
FunSuite. But with the current spark source distribution and the FunSuite
jar that it installs, this feature does not work for me. The compiler says
that the === operator expects a Symbol argument, which suggests that it is
somehow being redefined by catalyst libraries. (It works if I just import
FunSuite alone.) Can you tell me the trick for getting FunSuite working
without having to change all of those test suite files? Thanks, and sorry
to have to bother you with this.
- Marc
On Mon, Nov 3, 2014 at 10:31 PM, Cheng Lian <no...@github.com>
wrote:
> Hey @culler <https://github.com/culler>, would you please add tests for
> all the new features you introduced in this PR?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3066#issuecomment-61592236>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61468734
[Test build #22811 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22811/consoleFull) for PR 3066 at commit [`0c389a7`](https://github.com/apache/spark/commit/0c389a7e485d346c4dc26d09a993695d9826f37c).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` implicit class InitialLiteral(x: Any) `
* `final class MutableDate extends MutableValue `
* `final class MutableTimestamp extends MutableValue `
* `class Date(milliseconds: Long) extends JDate(milliseconds) `
* `class Timestamp(milliseconds: Long) extends JTimestamp(milliseconds) `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61761929
Hey @culler, saw you comment about testing yesterday (gone now?) but didn't have time to response. Yes, the `===` operator is used by in both Spark SQL DSL and ScalaTest. To workaround this, you can either add your test case in `DslQuerySuite` and use `checkAnswer` for assertion, or simply avoid to use `===`, there're some other assertion functions in ScalaTest, for example, `assertResult` is one of my favorite :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19720008
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala ---
@@ -139,6 +140,12 @@ class JoinedRow extends Row {
def getString(i: Int): String =
if (i < row1.size) row1.getString(i) else row2.getString(i - row1.size)
+ def getDate(i: Int): Date =
+ if (i < row1.size) row1.getDate(i) else row2.getDate(i - row1.size)
+
+ def getTimestamp(i: Int): Timestamp =
+ if (i < row1.size) row1.getTimestamp(i) else row2.getTimestamp(i - row1.size)
+
--- End diff --
And you can always replace these specialized getters with `Row.getAs[T]`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by culler <gi...@git.apache.org>.
Github user culler closed the pull request at:
https://github.com/apache/spark/pull/3066
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19733740
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/timetypes.scala ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import java.sql.{Date => JDate, Timestamp => JTimestamp}
+import scala.language.implicitConversions
+
+/*
+ * Subclass of java.sql.Date which provides the usual comparison
+ * operators (as required for catalyst expressions) and which can
+ * be constructed from a string.
+ *
+ * scala> val d1 = Date("2014-02-01")
+ * d1: Date = 2014-02-01
+ *
+ * scala> val d2 = Date("2014-02-02")
+ * d2: Date = 2014-02-02
+ *
+ * scala> d1 < d2
+ * res1: Boolean = true
+ */
+
+class Date(milliseconds: Long) extends JDate(milliseconds) {
--- End diff --
OK. That makes sense and avoids problems.
But how about if they are still imported as org.apache.spark.sql.Date and org.apache.spark.sql.Timestamp at the top level? My reasoning is based on providing consistency for a user. A user is not required to use RichInts or RichStrings as literals in a catalyst expression. So why would they expect to have to use RichDates or RichTimestamps?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19719985
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala ---
@@ -139,6 +140,12 @@ class JoinedRow extends Row {
def getString(i: Int): String =
if (i < row1.size) row1.getString(i) else row2.getString(i - row1.size)
+ def getDate(i: Int): Date =
+ if (i < row1.size) row1.getDate(i) else row2.getDate(i - row1.size)
+
+ def getTimestamp(i: Int): Timestamp =
+ if (i < row1.size) row1.getTimestamp(i) else row2.getTimestamp(i - row1.size)
+
--- End diff --
Getters like `getDate` and `getTimestamp` are not implemented for a reason, please refer to this PR discussion for details https://github.com/apache/spark/pull/2284#issuecomment-54917551
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61437893
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19784873
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Row.scala ---
@@ -42,6 +44,31 @@ object Row {
* This method can be used to construct a [[Row]] from a [[Seq]] of values.
*/
def fromSeq(values: Seq[Any]): Row = new GenericRow(values.toArray)
+
+ /**
+ * This method can be used to construct a [[Row]] from a [[Seq]] of Strings,
+ * converting each item to the type specified in a [[StructType]] schema.
+ * Only primitive types can be used.
+ */
+ def fromStringsBySchema(strings: Seq[String], schema: StructType): Row = {
+ val values = for {
+ (field, str) <- schema.fields zip strings
+ item = field.dataType match {
+ case IntegerType => str.toInt
+ case LongType => str.toLong
+ case DoubleType => str.toDouble
+ case FloatType => str.toFloat
+ case ByteType => str.toByte
+ case ShortType => str.toShort
+ case StringType => str
+ case BooleanType => (str != "")
+ case DateType => Date.valueOf(str)
+ case TimestampType => Timestamp.valueOf(str)
+ case DecimalType() => new BigDecimal(str)
+ }
+ } yield item
+ new GenericRow(values.toArray)
+ }
--- End diff --
OK I see you point. However, `Row` is used extremely frequently on many critical paths, so in general we choose to detect type errors while query planning, rather than introduce per row type checking cost here. It's somewhat similar to static typed programming languages like C++: once your program compiles, types are guaranteed to be matched (unless you add nasty type casting tricks intentionally), and you don't need to check types everywhere manually. The query planner is just the compiler here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19719774
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala ---
@@ -146,6 +146,31 @@ package object dsl {
def upper(e: Expression) = Upper(e)
def lower(e: Expression) = Lower(e)
+ /*
+ * Conversions to provide the standard operators in the special case
+ * where a literal is being combined with a symbol. Without these an
+ * expression such as 0 < 'x is not recognized.
+ */
+ implicit class InitialLiteral(x: Any) {
+ val literal = Literal(x)
+ def + (other: Symbol):Expression = {literal + other}
+ def - (other: Symbol):Expression = {literal - other}
+ def * (other: Symbol):Expression = {literal * other}
+ def / (other: Symbol):Expression = {literal / other}
+ def % (other: Symbol):Expression = {literal % other}
+
+ def && (other: Symbol):Expression = {literal && other}
+ def || (other: Symbol):Expression = {literal || other}
+
+ def < (other: Symbol):Expression = {literal < other}
+ def <= (other: Symbol):Expression = {literal <= other}
+ def > (other: Symbol):Expression = {literal > other}
+ def >= (other: Symbol):Expression = {literal >= other}
+ def === (other: Symbol):Expression = {literal === other}
+ def <=> (other: Symbol):Expression = {literal <=> other}
+ def !== (other: Symbol):Expression = {literal !== other}
--- End diff --
Please add a space after `:`, and remove the braces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19737251
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnType.scala ---
@@ -372,6 +372,11 @@ private[sql] object TIMESTAMP extends NativeColumnType(TimestampType, 9, 12) {
override def setField(row: MutableRow, ordinal: Int, value: Timestamp): Unit = {
row(ordinal) = value
}
+
+ def append(v: Date, buffer: ByteBuffer) {
+ buffer.putLong(v.getTime)
+ }
+
--- End diff --
This won't compile without an implementation of the append method, because it is declared in the abstract ColumnType class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19784466
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala ---
@@ -187,7 +187,8 @@ case object NullType extends DataType
object NativeType {
val all = Seq(
- IntegerType, BooleanType, LongType, DoubleType, FloatType, ShortType, ByteType, StringType)
+ IntegerType, BooleanType, LongType, DoubleType, FloatType, ShortType,
+ ByteType, StringType, DateType, TimestampType)
--- End diff --
Oh yes, sorry I missed the newly added `DateType` and `TimestampType` here. Then would you mind to also add these two in `NativeType.defaultSizeOf`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19733162
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Row.scala ---
@@ -42,6 +44,31 @@ object Row {
* This method can be used to construct a [[Row]] from a [[Seq]] of values.
*/
def fromSeq(values: Seq[Any]): Row = new GenericRow(values.toArray)
+
+ /**
+ * This method can be used to construct a [[Row]] from a [[Seq]] of Strings,
+ * converting each item to the type specified in a [[StructType]] schema.
+ * Only primitive types can be used.
+ */
+ def fromStringsBySchema(strings: Seq[String], schema: StructType): Row = {
+ val values = for {
+ (field, str) <- schema.fields zip strings
+ item = field.dataType match {
+ case IntegerType => str.toInt
+ case LongType => str.toLong
+ case DoubleType => str.toDouble
+ case FloatType => str.toFloat
+ case ByteType => str.toByte
+ case ShortType => str.toShort
+ case StringType => str
+ case BooleanType => (str != "")
+ case DateType => Date.valueOf(str)
+ case TimestampType => Timestamp.valueOf(str)
+ case DecimalType() => new BigDecimal(str)
+ }
+ } yield item
+ new GenericRow(values.toArray)
+ }
--- End diff --
It doesn't need to be a Row method. I will move it outside ( and worry about the Boolean case.)
There is an issue lurking here, however. A SchemaRDD should have Rows which match its schema. But mismatches generate no errors until the last stage, e.g. when one collects the results of a query. My thought was that it would be helpful to have a "safe" Row constructor which would check its data against a schema and raise an error if they do not match (after applying whatever implicit conversions are available. (Yes, this is a separate issue and does not belong here.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61468491
[Test build #22811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22811/consoleFull) for PR 3066 at commit [`0c389a7`](https://github.com/apache/spark/commit/0c389a7e485d346c4dc26d09a993695d9826f37c).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61830155
Thanks @liancheng. I deleted the comment to avoid wasting your time since
I figured out what was going on.
Actually it was not my test suite that i was worried about. Many existing
tests would not compile because my new conversions add an === operator to
types such as Int when the new conversions are in scope, as they are when
testing the DSL. I followed scalatest's advice for dealing with cases
where an === operator already exists. Namely, to apply their conversion
explicitly:
assert(X === Y) --> assert(convertToEqualizer(X).===(Y))
Will that be OK? (I import convertToEqualizer as EQ for brevity.)
I wanted to pass the existing tests before adding any new ones. I am
checking that now. It looks good so far, but unfortunately, 11 test files
had to be edited as above.
On Tue, Nov 4, 2014 at 11:31 PM, Cheng Lian <no...@github.com>
wrote:
> Hey @culler <https://github.com/culler>, saw you comment about testing
> yesterday (gone now?) but didn't have time to response. Yes, the ===
> operator is used by in both Spark SQL DSL and ScalaTest. To workaround
> this, you can either add your test case in DslQuerySuite and use
> checkAnswer for assertion, or simply avoid to use ===, there're some
> other assertion functions in ScalaTest, for example, assertResult is one
> of my favorite :)
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3066#issuecomment-61761929>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19736442
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/package.scala ---
@@ -451,4 +451,33 @@ package object sql {
* Builder for [[Metadata]]. If there is a key collision, the latter will overwrite the former.
*/
type MetadataBuilder = catalyst.util.MetadataBuilder
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * A Date class which support the standard comparison operators, for
+ * use in DSL expressions. Implicit conversions to java.sql.Date
+ * are provided. The class intializer accepts a String, e.g.
+ *
+ * val ts = Date("2014-01-01")
--- End diff --
And more thanks for reading the code so carefully and for all of the thoughtful comments and helpful advice. I am pushing fixes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19735638
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/package.scala ---
@@ -451,4 +451,33 @@ package object sql {
* Builder for [[Metadata]]. If there is a key collision, the latter will overwrite the former.
*/
type MetadataBuilder = catalyst.util.MetadataBuilder
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * A Date class which support the standard comparison operators, for
+ * use in DSL expressions. Implicit conversions to java.sql.Date
+ * are provided. The class intializer accepts a String, e.g.
+ *
+ * val ts = Date("2014-01-01")
--- End diff --
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-62257326
Thanks! I'm proud to be a member of the club. :^)
On Fri, Nov 7, 2014 at 11:16 PM, Cheng Lian <no...@github.com>
wrote:
> @culler <https://github.com/culler> Don't worry, I guess every
> contributor had screwed up rebasing at least once :)
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3066#issuecomment-62246351>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61468737
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22811/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61498140
Hmmm. The GitHub docs seem to suggest that I am supposed to mention @liancheng and @rxin in my comments. So I am doing that. My apologies if this is superfluous or annoying. - @culler
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61447235
Hey @culler, in general I like the DSL improvements, thanks for the work! Left some in-line comments, and two additional hight level comments here:
1. Usually we encourage one PR for one code change. It's harder to track and review if developers mix several unrelated or loosely related changes into a single PR.
2. Please always add sufficient tests for changes made in the PR unless they are trivial (e.g. renaming or styling changes).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-62158264
Hi @liancheng , now that I have completely screwed up this PR by attempting to rebase the repository, I will close it and open a new one which will hopefully be clean.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19731906
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala ---
@@ -139,6 +140,12 @@ class JoinedRow extends Row {
def getString(i: Int): String =
if (i < row1.size) row1.getString(i) else row2.getString(i - row1.size)
+ def getDate(i: Int): Date =
+ if (i < row1.size) row1.getDate(i) else row2.getDate(i - row1.size)
+
+ def getTimestamp(i: Int): Timestamp =
+ if (i < row1.size) row1.getTimestamp(i) else row2.getTimestamp(i - row1.size)
+
--- End diff --
I see. I am removing them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19784664
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala ---
@@ -139,6 +140,12 @@ class JoinedRow extends Row {
def getString(i: Int): String =
if (i < row1.size) row1.getString(i) else row2.getString(i - row1.size)
+ def getDate(i: Int): Date =
+ if (i < row1.size) row1.getDate(i) else row2.getDate(i - row1.size)
+
+ def getTimestamp(i: Int): Timestamp =
+ if (i < row1.size) row1.getTimestamp(i) else row2.getTimestamp(i - row1.size)
+
--- End diff --
Sorry for not being clear enough, all getters/setters for `Date` and `Timestamp` in all `Row` classes should be removed, including:
- [MutableRow](https://github.com/apache/spark/pull/3066/files#diff-d065227bfe580403f937c59a1c9e3fadR104)
- [EmptyRow](https://github.com/apache/spark/pull/3066/files#diff-d065227bfe580403f937c59a1c9e3fadR126)
- [GenericRow](https://github.com/apache/spark/pull/3066/files#diff-d065227bfe580403f937c59a1c9e3fadR193)
- [GenericMutableRow](https://github.com/apache/spark/pull/3066/files#diff-d065227bfe580403f937c59a1c9e3fadR246)
- [MutableDate and MutableTimestamp](https://github.com/apache/spark/pull/3066/files#diff-ea0766c16ccb1684d2bf67df274d0d5bR173) & [SpecificMutableRow](https://github.com/apache/spark/pull/3066/files#diff-ea0766c16ccb1684d2bf67df274d0d5bR340) (`SpecificMutableRow` was designed for avoiding boxing/unboxing costs when getting/setting primitive types like `Int` and `Double`. Adding specialized code for `Date` and `Timestamp` doesn't make sense here.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61870151
Hi @liancheng. Tests have been added. The PR builds, satisfies lint, and pass the tests on my system. (Except there were two failures in the SparkSubmitSuite, which I think are not related to this PR.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61468129
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19733534
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala ---
@@ -187,7 +187,8 @@ case object NullType extends DataType
object NativeType {
val all = Seq(
- IntegerType, BooleanType, LongType, DoubleType, FloatType, ShortType, ByteType, StringType)
+ IntegerType, BooleanType, LongType, DoubleType, FloatType, ShortType,
+ ByteType, StringType, DateType, TimestampType)
--- End diff --
Yes, but the new line does exceed 100 columns due to the addition of two new items DateType and TimestampType. (Or so the compiler tells me.)
Perhaps there is a reason why these types should not be regarded as "Native" but I could not see any such reason.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19722088
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/package.scala ---
@@ -451,4 +451,33 @@ package object sql {
* Builder for [[Metadata]]. If there is a key collision, the latter will overwrite the former.
*/
type MetadataBuilder = catalyst.util.MetadataBuilder
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * A Date class which support the standard comparison operators, for
+ * use in DSL expressions. Implicit conversions to java.sql.Date
+ * are provided. The class intializer accepts a String, e.g.
+ *
+ * val ts = Date("2014-01-01")
+ *
+ * @group dataType
+ */
+ @DeveloperApi
+ val Date = catalyst.expressions.Date
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * A Timestamp class which support the standard comparison
+ * operators, for use in DSL expressions. Implicit conversions to
+ * java.sql.timestamp are provided. The class intializer accepts a
+ * String, e.g.
+ *
+ * val ts = Timestamp("2014-01-01 12:34:56.78")
--- End diff --
Same as above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-62036504
The file sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala was added after this PR and caused a test to fail due to the collision between === operators. I am adding the whole file to this PR (with explicit conversions for === added to it). I hope that is the right way to handle this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61592236
Hey @culler, would you please add tests for all the new features you introduced in this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61442159
@liancheng - can you take a look at this first? Thanks.
@culler - can you create a JIRA ticket at https://issues.apache.org/jira/browse/SPARK and then update the pull request title to include the JIRA ticket number? e.g. "[SPARK-9999][SQL] DSL support for TimeStamp & Date"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19720295
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala ---
@@ -124,9 +124,9 @@ object ScalaReflection {
case obj: LongType.JvmType => LongType
case obj: FloatType.JvmType => FloatType
case obj: DoubleType.JvmType => DoubleType
- case obj: DateType.JvmType => DateType
case obj: BigDecimal => DecimalType.Unlimited
case obj: Decimal => DecimalType.Unlimited
+ case obj: DateType.JvmType => DateType
--- End diff --
I guess you moved this line here because `DateType` and `TimestampType` are more tightly related. This is consideration is legitimate and I'd also choose this order if I were the original author. However, this change introduces larger probability of future merge conflicts, while the benefit is rather limited. Same principle applies to all other similar reordering changes related to `Date` and `Timestamp` in this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4205][SQL] Timestamp and Date classes w...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-62246351
@culler Don't worry, I guess every contributor had screwed up rebasing at least once :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19734582
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala ---
@@ -124,9 +124,9 @@ object ScalaReflection {
case obj: LongType.JvmType => LongType
case obj: FloatType.JvmType => FloatType
case obj: DoubleType.JvmType => DoubleType
- case obj: DateType.JvmType => DateType
case obj: BigDecimal => DecimalType.Unlimited
case obj: Decimal => DecimalType.Unlimited
+ case obj: DateType.JvmType => DateType
--- End diff --
Yes. Well, the truth is that I was originally working with the source for Spark 1.1.0. When I pulled the current code it turned out that many lines such as this one had already been added after the release of 1.1.0, so I had to go through and remove many duplicates. In the course of that process it seemed better for me to have some logical scheme to guide me in choosing which of the two duplicates to delete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by culler <gi...@git.apache.org>.
Github user culler commented on the pull request:
https://github.com/apache/spark/pull/3066#issuecomment-61488635
I have corrected the issues pointed out in the comments. Please retest.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Timestamp and Date classes which work in the c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3066#discussion_r19721009
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala ---
@@ -146,6 +146,31 @@ package object dsl {
def upper(e: Expression) = Upper(e)
def lower(e: Expression) = Lower(e)
+ /*
+ * Conversions to provide the standard operators in the special case
+ * where a literal is being combined with a symbol. Without these an
+ * expression such as 0 < 'x is not recognized.
+ */
+ implicit class InitialLiteral(x: Any) {
--- End diff --
How about `LhsLiteral` (left hand side literal)? Also, converting `Any` to this class implicitly seems a bit dangerous. I'd prefer this style:
```scala
case class LhsLiteral(x: Any) {
...
}
// Similar to all the xxToLiteral implicit conversions:
implicit booleanToLhsLiteral(b: Boolean) = new LhsLiteral(b)
implicit byteToLhsLiteral(b: Byte) = new LhsLiteral(b)
...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org