You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by davies <gi...@git.apache.org> on 2016/08/24 21:20:42 UTC
[GitHub] spark pull request #14797: [SPARK-17230] [SQL] Should not pass optimized que...
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/14797
[SPARK-17230] [SQL] Should not pass optimized query into QueryExecution in DataFrameWriter
## What changes were proposed in this pull request?
Some analyzer rules have assumptions on logical plans, optimizer may break these assumption, we should not pass an optimized query plan into QueryExecution (will be analyzed again), otherwise we may some weird bugs.
For example, we have a rule for decimal calculation to promote the precision before binary operations, use PromotePrecision as placeholder to indicate that this rule should not apply twice. But a Optimizer rule will remove this placeholder, that break the assumption, then the rule applied twice, cause wrong result.
Ideally, we should make all the analyzer rules all idempotent, that may require lots of effort to double checking them one by one (may be not easy).
An easier approach could be never feed a optimized plan into Analyzer, this PR fix the case for RunnableComand, they will be optimized, during execution, the passed `query` will also be passed into QueryExecution again. This PR make these `query` not part of the children, so they will not be optimized and analyzed again.
Right now, we did not know a logical plan is optimized or not, we could introduce a flag for that, and make sure a optimized logical plan will not be analyzed again.
## How was this patch tested?
Added regression tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark fix_writer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14797.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14797
----
commit 9e1c555ebbab62c73f0b5a39617e5cd284720851
Author: Davies Liu <da...@databricks.com>
Date: 2016-08-24T21:06:17Z
fix DataFrameWriter
commit 25c451a67aeb912c0d215548e49c1a661d308b14
Author: Davies Liu <da...@databricks.com>
Date: 2016-08-24T21:07:00Z
Merge branch 'master' of github.com:apache/spark into fix_writer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14797
Just created a JIRA: https://issues.apache.org/jira/browse/SPARK-17409
FYI, Spark 2.0.1 has been fixed by this PR, but Spark 2.1.0 has the issue due to the related codes have been changed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14797
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14797
**[Test build #64682 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64682/consoleFull)** for PR 14797 at commit [`7771c6a`](https://github.com/apache/spark/commit/7771c6af2ca6a32141e2cf8851212e9f302485e1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14797
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/14797
Thanks! Is there a jira?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14797: [SPARK-17230] [SQL] Should not pass optimized que...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/14797
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14797
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64791/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14797
**[Test build #64377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64377/consoleFull)** for PR 14797 at commit [`25c451a`](https://github.com/apache/spark/commit/25c451a67aeb912c0d215548e49c1a661d308b14).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14797
**[Test build #64791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64791/consoleFull)** for PR 14797 at commit [`02b2896`](https://github.com/apache/spark/commit/02b289673654c2572dc8087e973a714f7c637d36).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the issue:
https://github.com/apache/spark/pull/14797
cc @yhuai @JoshRosen @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14797
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64377/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14797
**[Test build #64377 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64377/consoleFull)** for PR 14797 at commit [`25c451a`](https://github.com/apache/spark/commit/25c451a67aeb912c0d215548e49c1a661d308b14).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` logWarning(s\"All labels belong to a single class and fitIntercept=false. It's a \" +`
* ` * Set thresholds in multiclass (or binary) classification to adjust the probability of`
* `class MultinomialLogisticRegression @Since(\"2.1.0\") (`
* ` logWarning(s\"All labels belong to a single class and fitIntercept=false. It's\" +`
* ` /** Margin (rawPrediction) for each class label. */`
* ` /** Score (probability) for each class label. */`
* ` class MultinomialLogisticRegressionModelWriter(instance: MultinomialLogisticRegressionModel)`
* ` class ALSWrapperWriter(instance: ALSWrapper) extends MLWriter `
* ` class ALSWrapperReader extends MLReader[ALSWrapper] `
* ` class GaussianMixtureWrapperWriter(instance: GaussianMixtureWrapper) extends MLWriter `
* ` class GaussianMixtureWrapperReader extends MLReader[GaussianMixtureWrapper] `
* ` class IsotonicRegressionWrapperWriter(instance: IsotonicRegressionWrapper) extends MLWriter `
* ` class IsotonicRegressionWrapperReader extends MLReader[IsotonicRegressionWrapper] `
* ` class LDAWrapperWriter(instance: LDAWrapper) extends MLWriter `
* ` class LDAWrapperReader extends MLReader[LDAWrapper] `
* ` class MultilayerPerceptronClassifierWrapperWriter(instance: MultilayerPerceptronClassifierWrapper)`
* `class JavaClassificationModel(JavaPredictionModel):`
* `class LogisticRegressionModel(JavaModel, JavaClassificationModel, JavaMLWritable, JavaMLReadable):`
* `class DecisionTreeClassificationModel(DecisionTreeModel, JavaClassificationModel, JavaMLWritable,`
* `class RandomForestClassificationModel(TreeEnsembleModel, JavaClassificationModel, JavaMLWritable,`
* `class GBTClassificationModel(TreeEnsembleModel, JavaPredictionModel, JavaMLWritable,`
* `class NaiveBayesModel(JavaModel, JavaClassificationModel, JavaMLWritable, JavaMLReadable):`
* `class MultilayerPerceptronClassificationModel(JavaModel, JavaPredictionModel, JavaMLWritable,`
* `class LinearRegressionModel(JavaModel, JavaPredictionModel, JavaMLWritable, JavaMLReadable):`
* `class DecisionTreeModel(JavaModel, JavaPredictionModel):`
* `class RandomForestRegressionModel(TreeEnsembleModel, JavaPredictionModel, JavaMLWritable,`
* `class GBTRegressionModel(TreeEnsembleModel, JavaPredictionModel, JavaMLWritable, JavaMLReadable):`
* `class GeneralizedLinearRegressionModel(JavaModel, JavaPredictionModel, JavaMLWritable,`
* `class JavaPredictionModel():`
* `class SubstituteUnresolvedOrdinals(conf: CatalystConf) extends Rule[LogicalPlan] `
* `case class UnresolvedInlineTable(`
* `case class UnresolvedTableValuedFunction(functionName: String, functionArgs: Seq[Expression])`
* `case class UnresolvedOrdinal(ordinal: Int)`
* `case class Literal (value: Any, dataType: DataType) extends LeafExpression with CodegenFallback `
* `abstract class PlanExpression[T <: QueryPlan[_]] extends Expression `
* `abstract class SubqueryExpression extends PlanExpression[LogicalPlan] `
* `case class ListQuery(plan: LogicalPlan, exprId: ExprId = NamedExpression.newExprId)`
* `case class Exists(plan: LogicalPlan, exprId: ExprId = NamedExpression.newExprId)`
* `case class SubqueryAlias(`
* `class QuantileSummaries(`
* ` case class Stats(value: Double, g: Int, delta: Int)`
* `class JDBCOptions(`
* `abstract class ExecSubqueryExpression extends PlanExpression[SubqueryExec] `
* `class OrcFileFormat extends FileFormat with DataSourceRegister with Serializable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the issue:
https://github.com/apache/spark/pull/14797
Merged this into master and 2.0 branch, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/14797
@gatorsmile want to put the regression tests at here? Or, you have already have a pr?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14797
The test failure in Spark 2.0 has been fixed in https://github.com/apache/spark/pull/14951. However, after this fix, we still optimize the query in CTAS twice, as reported in this PR. Thus, I plan to fix it soon. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14797
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by sameeragarwal <gi...@git.apache.org>.
Github user sameeragarwal commented on the issue:
https://github.com/apache/spark/pull/14797
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14797
**[Test build #64682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64682/consoleFull)** for PR 14797 at commit [`7771c6a`](https://github.com/apache/spark/commit/7771c6af2ca6a32141e2cf8851212e9f302485e1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14797
**[Test build #64791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64791/consoleFull)** for PR 14797 at commit [`02b2896`](https://github.com/apache/spark/commit/02b289673654c2572dc8087e973a714f7c637d36).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14797: [SPARK-17230] [SQL] Should not pass optimized que...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/14797#discussion_r77394975
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -479,13 +480,23 @@ case class DataSource(
}
}
+ // SPARK-17230: Resolve the partition columns so InsertIntoHadoopFsRelationCommand does
+ // not need to have the query as child, to avoid to analyze an optimized query,
+ // because InsertIntoHadoopFsRelationCommand will be optimized first.
+ val columns = partitionColumns.map { name =>
--- End diff --
This is only for write(), it does not have `val partitionSchema =` (others have).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14797
This does not catch all the cases. In CTAS, we still optimize the query. By following the way in this PR, I can try to fix that case. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14797: [SPARK-17230] [SQL] Should not pass optimized que...
Posted by srinathshankar <gi...@git.apache.org>.
Github user srinathshankar commented on a diff in the pull request:
https://github.com/apache/spark/pull/14797#discussion_r77280523
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -479,13 +480,23 @@ case class DataSource(
}
}
+ // SPARK-17230: Resolve the partition columns so InsertIntoHadoopFsRelationCommand does
+ // not need to have the query as child, to avoid to analyze an optimized query,
+ // because InsertIntoHadoopFsRelationCommand will be optimized first.
+ val columns = partitionColumns.map { name =>
--- End diff --
Can this be done before
val partitionSchema = ...
above ? Looks like we're trying to analyze the columns there as well...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14797
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64682/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...
Posted by srinathshankar <gi...@git.apache.org>.
Github user srinathshankar commented on the issue:
https://github.com/apache/spark/pull/14797
Looks fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org