You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2017/05/16 10:26:27 UTC
[GitHub] spark pull request #17998: [SPARK-20703][SQL][WIP] Add an operator for writi...
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/17998
[SPARK-20703][SQL][WIP] Add an operator for writing data out
## What changes were proposed in this pull request?
Right now in the explain plan / UI, we have no way to tell whether a query is writing data out, and also there is no way to associate metrics with data writes. We should add an operator for writing data out. This operator can be used to track writing data out and related metrics.
### Approach
We have several paths for writing data out through some `RunnableCommand` classes.
#### File-based relation: `InsertIntoHadoopFsRelationCommand`, `InsertIntoHiveTable`
Those commands use `FileFormatWriter` to write out data files. We can record some metrics in `FileFormatWriter` and update it later. `FileFormatWriter` accepts a `QueryExecution`. We can track the execution plan of `QueryExecution`.
This patch adds a new operator `WriteDataFileOutExec`. It is simply used to track the metrics of writing data file out for file-based relations. Currently we track some metrics:
* number of written files
* number of dynamic partitions
* bytes of written files
* number of output rows
* writing data out time (ms)
#### Other datasources: `InsertIntoDataSourceCommand`, `SaveIntoDataSourceCommand`
For other datasource relations, the logic of writing data out is delegated to the datasource implementations, e.g., `InsertableRelation.insert`, `CreatableRelationProvider.createRelation`. The APIs basically take a `DataFrame` for the data to write. Those APIs can possibly create new `DataFrame` based on the give one. So we can't easily track its execution, and theoretically we don't know the details of the API implementation in those datasources. So we can't obtain enough metrics.
## How was this patch tested?
Jenkins tests.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 SPARK-20703
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17998.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17998
----
commit f60e2801ba79eceefcc5c1dc2a9e1a3fbd026534
Author: Liang-Chi Hsieh <vi...@gmail.com>
Date: 2017-05-14T11:11:40Z
Add an operator for writing data out.
commit eb8553e105f197c46e3e99745b8aa17dac8f1105
Author: Liang-Chi Hsieh <vi...@gmail.com>
Date: 2017-05-15T08:56:22Z
Set metrics for file-based relation.
commit afb6e752922b6b2270b1e598f86f20ca2a858e51
Author: Liang-Chi Hsieh <vi...@gmail.com>
Date: 2017-05-16T09:57:13Z
Add tests for writing data metrics.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17998
cc @rxin Do you think the current approach makes sense to you? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17998: [SPARK-20703][SQL][WIP] Add an operator for writi...
Posted by viirya <gi...@git.apache.org>.
Github user viirya closed the pull request at:
https://github.com/apache/spark/pull/17998
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17998
**[Test build #76960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76960/testReport)** for PR 17998 at commit [`6e50181`](https://github.com/apache/spark/commit/6e50181b769445b928ac0bacbd5ba134277f2550).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17998
**[Test build #76960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76960/testReport)** for PR 17998 at commit [`6e50181`](https://github.com/apache/spark/commit/6e50181b769445b928ac0bacbd5ba134277f2550).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/17998
@shaneknapp is amplap jenkins down?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17998
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76960/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17998
`CreateDataSourceTableAsSelectCommand` works similarly as `SaveIntoDataSourceCommand`. Depending on the type of datasource, it calls `CreatableRelationProvider.createRelation` or invokes `InsertIntoHadoopFsRelationCommand` to write data.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17998
#18064 is merged. Since it changes relative classes and code paths a lot. And seems an alternative approach for showing the metrics of writing data out is better. I'd close this and create a new PR for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17998
How about `CreateDataSourceTableAsSelectCommand`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/17998
yep. it was wedged, so i kicked it and it's back now.
On Tue, May 16, 2017 at 7:34 AM, Herman van Hovell <notifications@github.com
> wrote:
> @shaneknapp <https://github.com/shaneknapp> is amplap jenkins down?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/17998#issuecomment-301801788>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABiDrG901lIJIaajUix8U4MfQLM1KAfUks5r6bP2gaJpZM4NcSg8>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17998
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17998: [SPARK-20703][SQL][WIP] Add an operator for writing data...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17998
Btw, `CreateHiveTableAsSelectCommand` inserts data by invoking `InsertIntoHiveTable`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org