You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/10/08 02:57:21 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
HyukjinKwon opened a new pull request #29973:
URL: https://github.com/apache/spark/pull/29973
### What changes were proposed in this pull request?
This PR removes the leftover of Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script.
- `test-hive1.2` title is not used anymore in Jenkins
- Remove some comments related to Hive 1.2
- Remove unused codes in `OrcFilters.scala` @ Hive
- Test `spark.sql.hive.convertMetastoreOrc` disabled case for the tests added at SPARK-19809 and SPARK-22267
### Why are the changes needed?
To remove unused codes.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
It will be tested in CI in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705327077
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705432934
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705329875
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705301728
**[Test build #129536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129536/testReport)** for PR 29973 at commit [`c33d969`](https://github.com/apache/spark/commit/c33d969bfee34ff0aaa517039e9e408da7bee8ec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705472628
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34159/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29973:
URL: https://github.com/apache/spark/pull/29973#discussion_r501419736
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala
##########
@@ -64,185 +64,7 @@ import org.apache.spark.sql.types._
*/
private[orc] object OrcFilters extends Logging {
- private def findMethod(klass: Class[_], name: String, args: Class[_]*): Method = {
- val method = klass.getMethod(name, args: _*)
- method.setAccessible(true)
- method
- }
-
def createFilter(schema: StructType, filters: Array[Filter]): Option[SearchArgument] = {
- DatasourceOrcFilters.createFilter(schema, filters).asInstanceOf[Option[SearchArgument]]
- }
-
- def convertibleFilters(
- schema: StructType,
- dataTypeMap: Map[String, DataType],
- filters: Seq[Filter]): Seq[Filter] = {
- import org.apache.spark.sql.sources._
-
- def convertibleFiltersHelper(
- filter: Filter,
- canPartialPushDown: Boolean): Option[Filter] = filter match {
- // At here, it is not safe to just convert one side and remove the other side
- // if we do not understand what the parent filters are.
- //
- // Here is an example used to explain the reason.
- // Let's say we have NOT(a = 2 AND b in ('1')) and we do not understand how to
- // convert b in ('1'). If we only convert a = 2, we will end up with a filter
- // NOT(a = 2), which will generate wrong results.
- //
- // Pushing one side of AND down is only safe to do at the top level or in the child
- // AND before hitting NOT or OR conditions, and in this case, the unsupported predicate
- // can be safely removed.
- case And(left, right) =>
- val leftResultOptional = convertibleFiltersHelper(left, canPartialPushDown)
- val rightResultOptional = convertibleFiltersHelper(right, canPartialPushDown)
- (leftResultOptional, rightResultOptional) match {
- case (Some(leftResult), Some(rightResult)) => Some(And(leftResult, rightResult))
- case (Some(leftResult), None) if canPartialPushDown => Some(leftResult)
- case (None, Some(rightResult)) if canPartialPushDown => Some(rightResult)
- case _ => None
- }
-
- // The Or predicate is convertible when both of its children can be pushed down.
- // That is to say, if one/both of the children can be partially pushed down, the Or
- // predicate can be partially pushed down as well.
- //
- // Here is an example used to explain the reason.
- // Let's say we have
- // (a1 AND a2) OR (b1 AND b2),
- // a1 and b1 is convertible, while a2 and b2 is not.
- // The predicate can be converted as
- // (a1 OR b1) AND (a1 OR b2) AND (a2 OR b1) AND (a2 OR b2)
- // As per the logical in And predicate, we can push down (a1 OR b1).
- case Or(left, right) =>
- for {
- lhs <- convertibleFiltersHelper(left, canPartialPushDown)
- rhs <- convertibleFiltersHelper(right, canPartialPushDown)
- } yield Or(lhs, rhs)
- case Not(pred) =>
- val childResultOptional = convertibleFiltersHelper(pred, canPartialPushDown = false)
- childResultOptional.map(Not)
- case other =>
- for (_ <- buildLeafSearchArgument(dataTypeMap, other, newBuilder())) yield other
- }
- filters.flatMap { filter =>
- convertibleFiltersHelper(filter, true)
- }
- }
-
- /**
- * Build a SearchArgument and return the builder so far.
- *
- * @param dataTypeMap a map from the attribute name to its data type.
- * @param expression the input predicates, which should be fully convertible to SearchArgument.
- * @param builder the input SearchArgument.Builder.
- * @return the builder so far.
- */
- private def buildSearchArgument(
- dataTypeMap: Map[String, DataType],
- expression: Filter,
- builder: Builder): Builder = {
- expression match {
- case And(left, right) =>
- val lhs = buildSearchArgument(dataTypeMap, left, builder.startAnd())
- val rhs = buildSearchArgument(dataTypeMap, right, lhs)
- rhs.end()
-
- case Or(left, right) =>
- val lhs = buildSearchArgument(dataTypeMap, left, builder.startOr())
- val rhs = buildSearchArgument(dataTypeMap, right, lhs)
- rhs.end()
-
- case Not(child) =>
- buildSearchArgument(dataTypeMap, child, builder.startNot()).end()
-
- case other =>
- buildLeafSearchArgument(dataTypeMap, other, builder).getOrElse {
- throw new SparkException(
- "The input filter of OrcFilters.buildSearchArgument should be fully convertible.")
- }
- }
- }
-
- /**
- * Build a SearchArgument for a leaf predicate and return the builder so far.
- *
- * @param dataTypeMap a map from the attribute name to its data type.
- * @param expression the input filter predicates.
- * @param builder the input SearchArgument.Builder.
- * @return the builder so far.
- */
- private def buildLeafSearchArgument(
- dataTypeMap: Map[String, DataType],
- expression: Filter,
- builder: Builder): Option[Builder] = {
- def isSearchableType(dataType: DataType): Boolean = dataType match {
- // Only the values in the Spark types below can be recognized by
- // the `SearchArgumentImpl.BuilderImpl.boxLiteral()` method.
- case ByteType | ShortType | FloatType | DoubleType => true
- case IntegerType | LongType | StringType | BooleanType => true
- case TimestampType | _: DecimalType => true
- case _ => false
- }
-
- import org.apache.spark.sql.sources._
-
- // NOTE: For all case branches dealing with leaf predicates below, the additional `startAnd()`
- // call is mandatory. ORC `SearchArgument` builder requires that all leaf predicates must be
- // wrapped by a "parent" predicate (`And`, `Or`, or `Not`).
- expression match {
- // NOTE: For all case branches dealing with leaf predicates below, the additional `startAnd()`
- // call is mandatory. ORC `SearchArgument` builder requires that all leaf predicates must be
- // wrapped by a "parent" predicate (`And`, `Or`, or `Not`).
-
- case EqualTo(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "equals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case EqualNullSafe(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "nullSafeEquals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case LessThan(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "lessThan", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case LessThanOrEqual(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "lessThanEquals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case GreaterThan(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startNot()
- val method = findMethod(bd.getClass, "lessThanEquals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case GreaterThanOrEqual(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startNot()
- val method = findMethod(bd.getClass, "lessThan", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case IsNull(attribute) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "isNull", classOf[String])
- Some(method.invoke(bd, attribute).asInstanceOf[Builder].end())
-
- case IsNotNull(attribute) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startNot()
- val method = findMethod(bd.getClass, "isNull", classOf[String])
- Some(method.invoke(bd, attribute).asInstanceOf[Builder].end())
-
- case In(attribute, values) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "in", classOf[String], classOf[Array[Object]])
- Some(method.invoke(bd, attribute, values.map(_.asInstanceOf[AnyRef]))
- .asInstanceOf[Builder].end())
-
- case _ => None
- }
+ DatasourceOrcFilters.createFilter(schema, filters)
Review comment:
It seems not used anymore after https://github.com/apache/spark/commit/a127387a53e1a24e76de83c5a1858fcdbd38c3a2#diff-6cac9bc2656e3782b0312dceb8c55d47L77-L88
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705311353
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34141/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705316413
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29973:
URL: https://github.com/apache/spark/pull/29973#discussion_r501439874
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala
##########
@@ -64,185 +64,7 @@ import org.apache.spark.sql.types._
*/
private[orc] object OrcFilters extends Logging {
- private def findMethod(klass: Class[_], name: String, args: Class[_]*): Method = {
- val method = klass.getMethod(name, args: _*)
- method.setAccessible(true)
- method
- }
-
def createFilter(schema: StructType, filters: Array[Filter]): Option[SearchArgument] = {
- DatasourceOrcFilters.createFilter(schema, filters).asInstanceOf[Option[SearchArgument]]
- }
-
- def convertibleFilters(
- schema: StructType,
- dataTypeMap: Map[String, DataType],
- filters: Seq[Filter]): Seq[Filter] = {
- import org.apache.spark.sql.sources._
-
- def convertibleFiltersHelper(
- filter: Filter,
- canPartialPushDown: Boolean): Option[Filter] = filter match {
- // At here, it is not safe to just convert one side and remove the other side
- // if we do not understand what the parent filters are.
- //
- // Here is an example used to explain the reason.
- // Let's say we have NOT(a = 2 AND b in ('1')) and we do not understand how to
- // convert b in ('1'). If we only convert a = 2, we will end up with a filter
- // NOT(a = 2), which will generate wrong results.
- //
- // Pushing one side of AND down is only safe to do at the top level or in the child
- // AND before hitting NOT or OR conditions, and in this case, the unsupported predicate
- // can be safely removed.
- case And(left, right) =>
- val leftResultOptional = convertibleFiltersHelper(left, canPartialPushDown)
- val rightResultOptional = convertibleFiltersHelper(right, canPartialPushDown)
- (leftResultOptional, rightResultOptional) match {
- case (Some(leftResult), Some(rightResult)) => Some(And(leftResult, rightResult))
- case (Some(leftResult), None) if canPartialPushDown => Some(leftResult)
- case (None, Some(rightResult)) if canPartialPushDown => Some(rightResult)
- case _ => None
- }
-
- // The Or predicate is convertible when both of its children can be pushed down.
- // That is to say, if one/both of the children can be partially pushed down, the Or
- // predicate can be partially pushed down as well.
- //
- // Here is an example used to explain the reason.
- // Let's say we have
- // (a1 AND a2) OR (b1 AND b2),
- // a1 and b1 is convertible, while a2 and b2 is not.
- // The predicate can be converted as
- // (a1 OR b1) AND (a1 OR b2) AND (a2 OR b1) AND (a2 OR b2)
- // As per the logical in And predicate, we can push down (a1 OR b1).
- case Or(left, right) =>
- for {
- lhs <- convertibleFiltersHelper(left, canPartialPushDown)
- rhs <- convertibleFiltersHelper(right, canPartialPushDown)
- } yield Or(lhs, rhs)
- case Not(pred) =>
- val childResultOptional = convertibleFiltersHelper(pred, canPartialPushDown = false)
- childResultOptional.map(Not)
- case other =>
- for (_ <- buildLeafSearchArgument(dataTypeMap, other, newBuilder())) yield other
- }
- filters.flatMap { filter =>
- convertibleFiltersHelper(filter, true)
- }
- }
-
- /**
- * Build a SearchArgument and return the builder so far.
- *
- * @param dataTypeMap a map from the attribute name to its data type.
- * @param expression the input predicates, which should be fully convertible to SearchArgument.
- * @param builder the input SearchArgument.Builder.
- * @return the builder so far.
- */
- private def buildSearchArgument(
- dataTypeMap: Map[String, DataType],
- expression: Filter,
- builder: Builder): Builder = {
- expression match {
- case And(left, right) =>
- val lhs = buildSearchArgument(dataTypeMap, left, builder.startAnd())
- val rhs = buildSearchArgument(dataTypeMap, right, lhs)
- rhs.end()
-
- case Or(left, right) =>
- val lhs = buildSearchArgument(dataTypeMap, left, builder.startOr())
- val rhs = buildSearchArgument(dataTypeMap, right, lhs)
- rhs.end()
-
- case Not(child) =>
- buildSearchArgument(dataTypeMap, child, builder.startNot()).end()
-
- case other =>
- buildLeafSearchArgument(dataTypeMap, other, builder).getOrElse {
- throw new SparkException(
- "The input filter of OrcFilters.buildSearchArgument should be fully convertible.")
- }
- }
- }
-
- /**
- * Build a SearchArgument for a leaf predicate and return the builder so far.
- *
- * @param dataTypeMap a map from the attribute name to its data type.
- * @param expression the input filter predicates.
- * @param builder the input SearchArgument.Builder.
- * @return the builder so far.
- */
- private def buildLeafSearchArgument(
- dataTypeMap: Map[String, DataType],
- expression: Filter,
- builder: Builder): Option[Builder] = {
- def isSearchableType(dataType: DataType): Boolean = dataType match {
- // Only the values in the Spark types below can be recognized by
- // the `SearchArgumentImpl.BuilderImpl.boxLiteral()` method.
- case ByteType | ShortType | FloatType | DoubleType => true
- case IntegerType | LongType | StringType | BooleanType => true
- case TimestampType | _: DecimalType => true
- case _ => false
- }
-
- import org.apache.spark.sql.sources._
-
- // NOTE: For all case branches dealing with leaf predicates below, the additional `startAnd()`
- // call is mandatory. ORC `SearchArgument` builder requires that all leaf predicates must be
- // wrapped by a "parent" predicate (`And`, `Or`, or `Not`).
- expression match {
- // NOTE: For all case branches dealing with leaf predicates below, the additional `startAnd()`
- // call is mandatory. ORC `SearchArgument` builder requires that all leaf predicates must be
- // wrapped by a "parent" predicate (`And`, `Or`, or `Not`).
-
- case EqualTo(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "equals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case EqualNullSafe(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "nullSafeEquals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case LessThan(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "lessThan", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case LessThanOrEqual(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "lessThanEquals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case GreaterThan(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startNot()
- val method = findMethod(bd.getClass, "lessThanEquals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case GreaterThanOrEqual(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startNot()
- val method = findMethod(bd.getClass, "lessThan", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case IsNull(attribute) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "isNull", classOf[String])
- Some(method.invoke(bd, attribute).asInstanceOf[Builder].end())
-
- case IsNotNull(attribute) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startNot()
- val method = findMethod(bd.getClass, "isNull", classOf[String])
- Some(method.invoke(bd, attribute).asInstanceOf[Builder].end())
-
- case In(attribute, values) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "in", classOf[String], classOf[Array[Object]])
- Some(method.invoke(bd, attribute, values.map(_.asInstanceOf[AnyRef]))
- .asInstanceOf[Builder].end())
-
- case _ => None
- }
+ DatasourceOrcFilters.createFilter(schema, filters)
Review comment:
Could we remove this file? We can use `org.apache.spark.sql.execution.datasources.orc.OrcFilters` directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705352021
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34146/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705352040
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705432934
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-706000169
Remove [spark-deps-hadoop-2.7-hive-1.2](https://github.com/apache/spark/blob/master/dev/deps/spark-deps-hadoop-2.7-hive-1.2)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705331653
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705327077
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-706000169
Remove [spark-deps-hadoop-2.7-hive-1.2](https://github.com/apache/spark/blob/master/dev/deps/spark-deps-hadoop-2.7-hive-1.2)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705316413
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705331111
**[Test build #129536 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129536/testReport)** for PR 29973 at commit [`c33d969`](https://github.com/apache/spark/commit/c33d969bfee34ff0aaa517039e9e408da7bee8ec).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705331420
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705299873
cc @dongjoon-hyun and @wangyum FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705327077
**[Test build #129539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129539/testReport)** for PR 29973 at commit [`66ec0ff`](https://github.com/apache/spark/commit/66ec0ffa48fb6df4fb7603112e4bdcf6aca37b0b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705520245
**[Test build #129553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129553/testReport)** for PR 29973 at commit [`2bad37c`](https://github.com/apache/spark/commit/2bad37cdb3c1ffda4b01b168820b70a5e6e30846).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705376576
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29973:
URL: https://github.com/apache/spark/pull/29973#discussion_r501420104
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcFilterSuite.scala
##########
@@ -296,11 +295,6 @@ class HiveOrcFilterSuite extends OrcTest with TestHiveSingleton {
test("filter pushdown - combinations with logical operators") {
withOrcDataFrame((1 to 4).map(i => Tuple1(Option(i)))) { implicit df =>
- // Because `ExpressionTree` is not accessible at Hive 1.2.x, this should be checked
- // in string form in order to check filter creation including logical operators
- // such as `and`, `or` or `not`. So, this function uses `SearchArgument.toString()`
- // to produce string expression and then compare it to given string expression below.
- // This might have to be changed after Hive version is upgraded.
Review comment:
Even if we can access `ExpressionTree` from `SearchArgument`, we can't construct them to compare to use in tests. This seems what Hive does as well, see https://github.com/apache/hive/blob/master/ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29973:
URL: https://github.com/apache/spark/pull/29973
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705352040
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705521238
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705331659
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34145/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705301728
**[Test build #129536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129536/testReport)** for PR 29973 at commit [`c33d969`](https://github.com/apache/spark/commit/c33d969bfee34ff0aaa517039e9e408da7bee8ec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705472646
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-706008368
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705436291
**[Test build #129553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129553/testReport)** for PR 29973 at commit [`2bad37c`](https://github.com/apache/spark/commit/2bad37cdb3c1ffda4b01b168820b70a5e6e30846).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705329875
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705331170
**[Test build #129540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129540/testReport)** for PR 29973 at commit [`2bad37c`](https://github.com/apache/spark/commit/2bad37cdb3c1ffda4b01b168820b70a5e6e30846).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29973:
URL: https://github.com/apache/spark/pull/29973
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-706091745
Merged to master for Apache Spark 3.1.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705327077
**[Test build #129539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129539/testReport)** for PR 29973 at commit [`66ec0ff`](https://github.com/apache/spark/commit/66ec0ffa48fb6df4fb7603112e4bdcf6aca37b0b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705376016
**[Test build #129540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129540/testReport)** for PR 29973 at commit [`2bad37c`](https://github.com/apache/spark/commit/2bad37cdb3c1ffda4b01b168820b70a5e6e30846).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705459420
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34159/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29973:
URL: https://github.com/apache/spark/pull/29973#discussion_r501421971
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcFilterSuite.scala
##########
@@ -296,11 +295,6 @@ class HiveOrcFilterSuite extends OrcTest with TestHiveSingleton {
test("filter pushdown - combinations with logical operators") {
withOrcDataFrame((1 to 4).map(i => Tuple1(Option(i)))) { implicit df =>
- // Because `ExpressionTree` is not accessible at Hive 1.2.x, this should be checked
- // in string form in order to check filter creation including logical operators
- // such as `and`, `or` or `not`. So, this function uses `SearchArgument.toString()`
- // to produce string expression and then compare it to given string expression below.
- // This might have to be changed after Hive version is upgraded.
Review comment:
BTW, I wrote this comment. I am sure we can remove it :-).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705969339
Sure will update!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705329875
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705331170
**[Test build #129540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129540/testReport)** for PR 29973 at commit [`2bad37c`](https://github.com/apache/spark/commit/2bad37cdb3c1ffda4b01b168820b70a5e6e30846).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29973:
URL: https://github.com/apache/spark/pull/29973#discussion_r501439874
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala
##########
@@ -64,185 +64,7 @@ import org.apache.spark.sql.types._
*/
private[orc] object OrcFilters extends Logging {
- private def findMethod(klass: Class[_], name: String, args: Class[_]*): Method = {
- val method = klass.getMethod(name, args: _*)
- method.setAccessible(true)
- method
- }
-
def createFilter(schema: StructType, filters: Array[Filter]): Option[SearchArgument] = {
- DatasourceOrcFilters.createFilter(schema, filters).asInstanceOf[Option[SearchArgument]]
- }
-
- def convertibleFilters(
- schema: StructType,
- dataTypeMap: Map[String, DataType],
- filters: Seq[Filter]): Seq[Filter] = {
- import org.apache.spark.sql.sources._
-
- def convertibleFiltersHelper(
- filter: Filter,
- canPartialPushDown: Boolean): Option[Filter] = filter match {
- // At here, it is not safe to just convert one side and remove the other side
- // if we do not understand what the parent filters are.
- //
- // Here is an example used to explain the reason.
- // Let's say we have NOT(a = 2 AND b in ('1')) and we do not understand how to
- // convert b in ('1'). If we only convert a = 2, we will end up with a filter
- // NOT(a = 2), which will generate wrong results.
- //
- // Pushing one side of AND down is only safe to do at the top level or in the child
- // AND before hitting NOT or OR conditions, and in this case, the unsupported predicate
- // can be safely removed.
- case And(left, right) =>
- val leftResultOptional = convertibleFiltersHelper(left, canPartialPushDown)
- val rightResultOptional = convertibleFiltersHelper(right, canPartialPushDown)
- (leftResultOptional, rightResultOptional) match {
- case (Some(leftResult), Some(rightResult)) => Some(And(leftResult, rightResult))
- case (Some(leftResult), None) if canPartialPushDown => Some(leftResult)
- case (None, Some(rightResult)) if canPartialPushDown => Some(rightResult)
- case _ => None
- }
-
- // The Or predicate is convertible when both of its children can be pushed down.
- // That is to say, if one/both of the children can be partially pushed down, the Or
- // predicate can be partially pushed down as well.
- //
- // Here is an example used to explain the reason.
- // Let's say we have
- // (a1 AND a2) OR (b1 AND b2),
- // a1 and b1 is convertible, while a2 and b2 is not.
- // The predicate can be converted as
- // (a1 OR b1) AND (a1 OR b2) AND (a2 OR b1) AND (a2 OR b2)
- // As per the logical in And predicate, we can push down (a1 OR b1).
- case Or(left, right) =>
- for {
- lhs <- convertibleFiltersHelper(left, canPartialPushDown)
- rhs <- convertibleFiltersHelper(right, canPartialPushDown)
- } yield Or(lhs, rhs)
- case Not(pred) =>
- val childResultOptional = convertibleFiltersHelper(pred, canPartialPushDown = false)
- childResultOptional.map(Not)
- case other =>
- for (_ <- buildLeafSearchArgument(dataTypeMap, other, newBuilder())) yield other
- }
- filters.flatMap { filter =>
- convertibleFiltersHelper(filter, true)
- }
- }
-
- /**
- * Build a SearchArgument and return the builder so far.
- *
- * @param dataTypeMap a map from the attribute name to its data type.
- * @param expression the input predicates, which should be fully convertible to SearchArgument.
- * @param builder the input SearchArgument.Builder.
- * @return the builder so far.
- */
- private def buildSearchArgument(
- dataTypeMap: Map[String, DataType],
- expression: Filter,
- builder: Builder): Builder = {
- expression match {
- case And(left, right) =>
- val lhs = buildSearchArgument(dataTypeMap, left, builder.startAnd())
- val rhs = buildSearchArgument(dataTypeMap, right, lhs)
- rhs.end()
-
- case Or(left, right) =>
- val lhs = buildSearchArgument(dataTypeMap, left, builder.startOr())
- val rhs = buildSearchArgument(dataTypeMap, right, lhs)
- rhs.end()
-
- case Not(child) =>
- buildSearchArgument(dataTypeMap, child, builder.startNot()).end()
-
- case other =>
- buildLeafSearchArgument(dataTypeMap, other, builder).getOrElse {
- throw new SparkException(
- "The input filter of OrcFilters.buildSearchArgument should be fully convertible.")
- }
- }
- }
-
- /**
- * Build a SearchArgument for a leaf predicate and return the builder so far.
- *
- * @param dataTypeMap a map from the attribute name to its data type.
- * @param expression the input filter predicates.
- * @param builder the input SearchArgument.Builder.
- * @return the builder so far.
- */
- private def buildLeafSearchArgument(
- dataTypeMap: Map[String, DataType],
- expression: Filter,
- builder: Builder): Option[Builder] = {
- def isSearchableType(dataType: DataType): Boolean = dataType match {
- // Only the values in the Spark types below can be recognized by
- // the `SearchArgumentImpl.BuilderImpl.boxLiteral()` method.
- case ByteType | ShortType | FloatType | DoubleType => true
- case IntegerType | LongType | StringType | BooleanType => true
- case TimestampType | _: DecimalType => true
- case _ => false
- }
-
- import org.apache.spark.sql.sources._
-
- // NOTE: For all case branches dealing with leaf predicates below, the additional `startAnd()`
- // call is mandatory. ORC `SearchArgument` builder requires that all leaf predicates must be
- // wrapped by a "parent" predicate (`And`, `Or`, or `Not`).
- expression match {
- // NOTE: For all case branches dealing with leaf predicates below, the additional `startAnd()`
- // call is mandatory. ORC `SearchArgument` builder requires that all leaf predicates must be
- // wrapped by a "parent" predicate (`And`, `Or`, or `Not`).
-
- case EqualTo(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "equals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case EqualNullSafe(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "nullSafeEquals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case LessThan(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "lessThan", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case LessThanOrEqual(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "lessThanEquals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case GreaterThan(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startNot()
- val method = findMethod(bd.getClass, "lessThanEquals", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case GreaterThanOrEqual(attribute, value) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startNot()
- val method = findMethod(bd.getClass, "lessThan", classOf[String], classOf[Object])
- Some(method.invoke(bd, attribute, value.asInstanceOf[AnyRef]).asInstanceOf[Builder].end())
-
- case IsNull(attribute) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "isNull", classOf[String])
- Some(method.invoke(bd, attribute).asInstanceOf[Builder].end())
-
- case IsNotNull(attribute) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startNot()
- val method = findMethod(bd.getClass, "isNull", classOf[String])
- Some(method.invoke(bd, attribute).asInstanceOf[Builder].end())
-
- case In(attribute, values) if isSearchableType(dataTypeMap(attribute)) =>
- val bd = builder.startAnd()
- val method = findMethod(bd.getClass, "in", classOf[String], classOf[Array[Object]])
- Some(method.invoke(bd, attribute, values.map(_.asInstanceOf[AnyRef]))
- .asInstanceOf[Builder].end())
-
- case _ => None
- }
+ DatasourceOrcFilters.createFilter(schema, filters)
Review comment:
Could we remove this file? We can use `org.apache.spark.sql.execution.datasources.orc.OrcFilters` directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705376576
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705432934
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705345394
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34146/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-706008368
Oh, nice catch, @wangyum !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705521238
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705331412
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705432934
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-706091961
Thanks @dongjoon-hyun!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-706040751
Nice, thanks @wangyum. Will update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705436291
**[Test build #129553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129553/testReport)** for PR 29973 at commit [`2bad37c`](https://github.com/apache/spark/commit/2bad37cdb3c1ffda4b01b168820b70a5e6e30846).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29973:
URL: https://github.com/apache/spark/pull/29973#discussion_r501420256
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
##########
@@ -296,7 +296,7 @@ private[orc] class OrcOutputWriter(
override def close(): Unit = {
if (recordWriterInstantiated) {
- // Hive 1.2.1 ORC initializes its private `writer` field at the first write.
+ // Hive ORC initializes its private `writer` field at the first write.
Review comment:
This is still same in the master of Hive.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705316417
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34141/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705331412
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705329860
**[Test build #129539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129539/testReport)** for PR 29973 at commit [`66ec0ff`](https://github.com/apache/spark/commit/66ec0ffa48fb6df4fb7603112e4bdcf6aca37b0b).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705299873
cc @dongjoon-hyun FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705376590
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129540/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705472646
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705316407
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34141/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705329882
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129539/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29973: [SPARK-33082][SPARK-20202][BUILD][SQL][FOLLOW-UP] Remove Hive 1.2 workarounds and Hive 1.2 profile in Jenkins script
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29973:
URL: https://github.com/apache/spark/pull/29973#issuecomment-705329875
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org