You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rxin <gi...@git.apache.org> on 2016/08/09 04:15:36 UTC
[GitHub] spark pull request #14554: [SPARK-16964][SQL] Remove private[sql] and privat...
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/14554
[SPARK-16964][SQL] Remove private[sql] and private[spark] from sql.execution package
## What changes were proposed in this pull request?
This package is meant to be internal, and as a result it does not make sense to mark things as private[sql] or private[spark]. It simply makes debugging harder when Spark developers need to inspect the plans at runtime.
This patch removes all private[sql] and private[spark] visibility modifiers in org.apache.spark.sql.execution.
## How was this patch tested?
N/A - just visibility changes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark remote-private
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14554.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14554
----
commit 78d1381e3bf2aa1b21a3afc16b64a4ab1125b3fb
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-08-09T04:13:56Z
[SPARK-16964][SQL] Remove private[sql] and private[spark] from sql.execution
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14554: [SPARK-16964][SQL] Remove private[sql] and private[spark...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14554
**[Test build #63409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63409/consoleFull)** for PR 14554 at commit [`78d1381`](https://github.com/apache/spark/commit/78d1381e3bf2aa1b21a3afc16b64a4ab1125b3fb).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class CachedData(plan: LogicalPlan, cachedRepresentation: InMemoryRelation)`
* `class CacheManager extends Logging `
* `trait DataSourceScanExec extends LeafExecNode with CodegenSupport `
* `case class RowDataSourceScanExec(`
* `case class FileSourceScanExec(`
* `case class ExternalRDD[T](`
* `case class ExternalRDDScanExec[T](`
* `case class LogicalRDD(`
* `case class RDDScanExec(`
* `trait FileRelation `
* `case class LocalTableScanExec(`
* `abstract class RowIterator `
* `trait LeafExecNode extends SparkPlan `
* `trait UnaryExecNode extends SparkPlan `
* `trait BinaryExecNode extends SparkPlan `
* `case class PlanLater(plan: LogicalPlan) extends LeafExecNode `
* `abstract class SparkStrategies extends QueryPlanner[SparkPlan] `
* `class UnsafeRowSerializer(`
* `case class ScalaUDAF(`
* `case class InMemoryRelation(`
* `case class InMemoryTableScanExec(`
* `trait RunnableCommand extends LogicalPlan with logical.Command `
* `case class ExecutedCommandExec(cmd: RunnableCommand) extends SparkPlan `
* `case class DataSourceAnalysis(conf: CatalystConf) extends Rule[LogicalPlan] `
* `class FindDataSourceTable(sparkSession: SparkSession) extends Rule[LogicalPlan] `
* `case class InsertIntoDataSourceCommand(`
* `case class InsertIntoHadoopFsRelationCommand(`
* `case class PartitionDirectory(values: InternalRow, path: Path)`
* `case class PartitionSpec(`
* `case class JDBCPartition(whereClause: String, idx: Int) extends Partition `
* `class ResolveDataSource(sparkSession: SparkSession) extends Rule[LogicalPlan] `
* `case class PreprocessTableInsertion(conf: SQLConf) extends Rule[LogicalPlan] `
* `case class PreWriteCheck(conf: SQLConf, catalog: SessionCatalog)`
* ` case class DebugExec(child: SparkPlan) extends UnaryExecNode with CodegenSupport `
* `class ExchangeCoordinator(`
* `case class MapPartitionsRWrapper(`
* `class IncrementalExecution(`
* `class ExecutionPage(parent: SQLTab) extends WebUIPage(\"execution\") with Logging `
* `class SQLHistoryListenerFactory extends SparkHistoryListenerFactory `
* `class SQLListener(conf: SparkConf) extends SparkListener with Logging `
* `class SQLHistoryListener(conf: SparkConf, sparkUI: SparkUI)`
* `class SQLTab(val listener: SQLListener, sparkUI: SparkUI)`
* `case class SparkPlanGraph(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14554: [SPARK-16964][SQL] Remove private[sql] and private[spark...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14554
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14554: [SPARK-16964][SQL] Remove private[sql] and private[spark...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14554
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63415/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14554: [SPARK-16964][SQL] Remove private[sql] and private[spark...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14554
**[Test build #63415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63415/consoleFull)** for PR 14554 at commit [`d9ba88f`](https://github.com/apache/spark/commit/d9ba88f465898e78a79db8390213dc831f841ae2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14554: [SPARK-16964][SQL] Remove private[sql] and private[spark...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14554
**[Test build #63415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63415/consoleFull)** for PR 14554 at commit [`d9ba88f`](https://github.com/apache/spark/commit/d9ba88f465898e78a79db8390213dc831f841ae2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14554: [SPARK-16964][SQL] Remove private[sql] and privat...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14554#discussion_r74000537
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/udaf.scala ---
@@ -204,7 +204,7 @@ sealed trait BufferSetterGetterUtils {
/**
* A Mutable [[Row]] representing a mutable aggregation buffer.
*/
-private[sql] class MutableAggregationBufferImpl (
+private[aggregate] class MutableAggregationBufferImpl(
--- End diff --
here too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14554: [SPARK-16964][SQL] Remove private[sql] and private[spark...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14554
**[Test build #63409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63409/consoleFull)** for PR 14554 at commit [`78d1381`](https://github.com/apache/spark/commit/78d1381e3bf2aa1b21a3afc16b64a4ab1125b3fb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14554: [SPARK-16964][SQL] Remove private[sql] and private[spark...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14554
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14554: [SPARK-16964][SQL] Remove private[sql] and privat...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14554#discussion_r74000577
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala ---
@@ -41,22 +42,23 @@ object PartitionDirectory {
* Holds a directory in a partitioned collection of files as well as as the partition values
* in the form of a Row. Before scanning, the files at `path` need to be enumerated.
*/
-private[sql] case class PartitionDirectory(values: InternalRow, path: Path)
+case class PartitionDirectory(values: InternalRow, path: Path)
-private[sql] case class PartitionSpec(
+case class PartitionSpec(
partitionColumns: StructType,
partitions: Seq[PartitionDirectory])
-private[sql] object PartitionSpec {
+object PartitionSpec {
val emptySpec = PartitionSpec(StructType(Seq.empty[StructField]), Seq.empty[PartitionDirectory])
}
-private[sql] object PartitioningUtils {
+object PartitioningUtils {
// This duplicates default value of Hive `ConfVars.DEFAULTPARTITIONNAME`, since sql/core doesn't
// depend on Hive.
- private[sql] val DEFAULT_PARTITION_NAME = "__HIVE_DEFAULT_PARTITION__"
+ val DEFAULT_PARTITION_NAME = "__HIVE_DEFAULT_PARTITION__"
- private[sql] case class PartitionValues(columnNames: Seq[String], literals: Seq[Literal]) {
+ private[datasources] case class PartitionValues(columnNames: Seq[String], literals: Seq[Literal])
--- End diff --
here too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14554: [SPARK-16964][SQL] Remove private[sql] and private[spark...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14554
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63409/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14554: [SPARK-16964][SQL] Remove private[sql] and private[spark...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14554
LGTM, merging to master!
this conflicts with 2.0, so I didn't backport
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14554: [SPARK-16964][SQL] Remove private[sql] and privat...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/14554
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14554: [SPARK-16964][SQL] Remove private[sql] and privat...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14554#discussion_r74000462
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanInfo.scala ---
@@ -47,7 +47,7 @@ class SparkPlanInfo(
}
}
-private[sql] object SparkPlanInfo {
+private[execution] object SparkPlanInfo {
--- End diff --
why keep this private?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14554: [SPARK-16964][SQL] Remove private[sql] and private[spark...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14554
cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14554: [SPARK-16964][SQL] Remove private[sql] and privat...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/14554#discussion_r74007849
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanInfo.scala ---
@@ -47,7 +47,7 @@ class SparkPlanInfo(
}
}
-private[sql] object SparkPlanInfo {
+private[execution] object SparkPlanInfo {
--- End diff --
If it is something internal to the implementation (i.e. not part of some public field for expressions or commands or query plan), I'm keeping them private for tighten visibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14554: [SPARK-16964][SQL] Remove private[sql] and privat...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14554#discussion_r74000543
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/udaf.scala ---
@@ -266,7 +266,7 @@ private[sql] class MutableAggregationBufferImpl (
/**
* A [[Row]] representing an immutable aggregation buffer.
*/
-private[sql] class InputAggregationBuffer private[sql] (
+private[aggregate] class InputAggregationBuffer(
--- End diff --
here too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org