You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by manishgupta88 <gi...@git.apache.org> on 2018/08/01 14:15:35 UTC
[GitHub] carbondata pull request #2595: [Documentation] [Unsafe Configuration] Added ...
GitHub user manishgupta88 opened a pull request:
https://github.com/apache/carbondata/pull/2595
[Documentation] [Unsafe Configuration] Added carbon.unsafe.driver.working.memory.in.mb parameter to differentiate between driver and executor unsafe memory
Added carbon.unsafe.driver.working.memory.in.mb parameter to differentiate between driver and executor unsafe memory
- [ ] Any interfaces changed?
No
- [ ] Any backward compatibility impacted?
No
- [ ] Document update required?
Yes. Updated
- [ ] Testing done
Verified manually
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/manishgupta88/carbondata unsafe_driver_property
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2595.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2595
----
commit 8196be0a50d9d0f38452cfdf5fbe0b0485973e87
Author: manishgupta88 <to...@...>
Date: 2018-08-01T14:08:30Z
Added carbon.unsafe.driver.working.memory.in.mb parameter to differentiate between driver and executor unsafe memory
----
---
[GitHub] carbondata issue #2595: [Documentation] [Unsafe Configuration] Added carbon....
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2595
LGTM
---
[GitHub] carbondata issue #2595: [Documentation] [Unsafe Configuration] Added carbon....
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2595
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6436/
---
[GitHub] carbondata issue #2595: [Documentation] [Unsafe Configuration] Added carbon....
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2595
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7710/
---
[GitHub] carbondata pull request #2595: [Documentation] [Unsafe Configuration] Added ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2595
---
[GitHub] carbondata issue #2595: [Documentation] [Unsafe Configuration] Added carbon....
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2595
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6103/
---
[GitHub] carbondata issue #2595: [Documentation] [Unsafe Configuration] Added carbon....
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2595
Why driver needs this unsafe memory?
---
[GitHub] carbondata issue #2595: [Documentation] [Unsafe Configuration] Added carbon....
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/2595
@xuchuanyin ....Usually in production scenarios driver memory will be less than the executor memory. Now we are using unsafe for caching block/blocklet dataMap in driver. Current unsafe memory configured fo executor is getting used for driver also which is not a good idea.
Therefore it is required to separate out driver and executor unsafe memory.
You can observe the same in spark configuration also that spark has given different parameters for configuring driver and executor memory overhead to control the unsafe memory usage.
spark.yarn.driver.memoryOverhead and spark.yarn.executor.memoryOverhead
---