You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by paragpc <gi...@git.apache.org> on 2016/03/15 19:36:39 UTC
[GitHub] spark pull request: [SPARK-13914] [Scheduler] Add functionality to...
GitHub user paragpc opened a pull request:
https://github.com/apache/spark/pull/11736
[SPARK-13914] [Scheduler] Add functionality to optionally configure a backup path for the Spark event logs
## What changes were proposed in this pull request?
Added "spark.eventLog.backup.enabled" and "spark.eventLog.backup.dir" options for spark event logs to allow real time back up of spark event logs.
Added functionality to backup event logs in back up directory as event happens.
Added corresponding documentation changes.
Users can backup the event logs to a cloud storage service directly and keep them there forever or in cloud environment, users may want to store the event logs on local disks and back them up to a cloud storage service from time to time using cloud backup agents.
## How was this patch tested?
Existing tests passed.
Added unit tests for back up functionality.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/paragpc/spark master_event_log_backup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11736.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11736
----
commit ec9999fcb4aabefd9b26462cb0b3c0914d6a9ea9
Author: Parag Chaudhari <pa...@users.noreply.github.com>
Date: 2016-03-15T17:25:46Z
Merge pull request #1 from apache/master
Update from original repository
commit 5af1f8a4abc4ee48e862c052a406407f184e7161
Author: Parag Chaudhari <pa...@amazon.com>
Date: 2016-03-14T17:54:17Z
Added functionality to optionally configure a backup path for the Spark event log.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13914] [Scheduler] Add functionality to...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/11736#issuecomment-196970794
Why does this functionality belong as part of Spark itself?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13914] [Scheduler] Add functionality to...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11736#issuecomment-197217669
Several committers have indicated they don't support this. Thank you but would you close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13914] [Scheduler] Add functionality to...
Posted by paragpc <gi...@git.apache.org>.
Github user paragpc commented on the pull request:
https://github.com/apache/spark/pull/11736#issuecomment-196983711
The only available alternative which I know is manually copy files to desired back up location. Manual copy of files doesn't provide real-time events, especially in case of spark streaming jobs which can run for a long period of time. Copying files after job completion does not make sense if someone wants real time backup as the events happen.
In cloud environment, this patch provides common way for all users to configure backup of event logs using existing cloud back up agents by configuring the back up directory to local disks(from where these back up agents would pick up these files). At the same time, it also allows users to configure main event log directory of their own choice. What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13914] [Scheduler] Add functionality to...
Posted by paragpc <gi...@git.apache.org>.
Github user paragpc closed the pull request at:
https://github.com/apache/spark/pull/11736
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13914] [Scheduler] Add functionality to...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11736#issuecomment-196966188
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org