You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xuchuanyin <gi...@git.apache.org> on 2017/12/08 06:20:48 UTC
[GitHub] carbondata pull request #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsaf...
GitHub user xuchuanyin opened a pull request:
https://github.com/apache/carbondata/pull/1633
[CARBONDATA-1878] [DataMap] Fix bugs in unsafe datamap store
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [X] Any interfaces changed?
`NO`
- [X] Any backward compatibility impacted?
`NO`
- [X] Document update required?
`NO`
- [X] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
`ADDED TESTS`
- How it is tested? Please attach test report.
`TEST IN LOCAL MACHINE`
- Is it a performance related change? Please attach the performance test report.
`NO, ONLY BUG FIXED`
- Any additional information to help reviewers in testing this change.
`NO`
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
`NOT RELATED`
COPY FROM JIRA:
===
# SCENARIO
Recently I have fixed some issues in Carbondata. To perform a full test to cover all the code that has been modified by me, I performed some iteration of the whole test case. Each iteration is started with different key configurations that will affect the flow in the code.
After I set `enable.offheap.sort=false` (default value is true) in the configuration, running tests will always end up with JVM crash error. The error messages are shown as below:
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f346b207ff1, pid=144619, tid=0x00007f346c2fc700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0xa90ff1] Unsafe_SetNativeShort+0x51
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/xu/ws/carbondata/hs_err_pid144619.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
```
# STEPS TOREPRODUCE
The error can be easily reproduced in different ways. Here I will provide a simple way to reproduce it:
1. Find the test case `DateDataTypeDirectDictionaryTest`.
2. Add the following code in the method `beforeAll`.
```
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.ENABLE_OFFHEAP_SORT, "false")
```
3. Run this test case.
4. You will find the test failed with the above error.
5. Replace the code in Step2 with the following code:
```
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.ENABLE_OFFHEAP_SORT, "true")
```
6. Run this test case.
7. The test is success without error.
# ANALYZE & RESOLVE
I have reproduced this error and analyzed the core dump file. The final stack message in core dump looks like below:
```
Thread 73303: (state = IN_VM)
- sun.misc.Unsafe.putShort(long, short) @bci=0 (Interpreted frame)
- org.apache.carbondata.core.indexstore.UnsafeMemoryDMStore.addToUnsafe(org.apache.carbondata.core.indexstore.schema.CarbonRowSchema, org.apache.carbondata.core.indexstore.row.DataMapRow, int) @bci=781, line=150 (Interpreted frame)
- org.apache.carbondata.core.indexstore.UnsafeMemoryDMStore.addIndexRowToUnsafe(org.apache.carbondata.core.indexstore.row.DataMapRow) @bci=59, line=99 (Interpreted frame)
...
```
After inspecting the code, I found there lies bug in `UnsafeMemoryDMStore line=150` while writing length to unsafe memory -- It writes with wrong base object.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuchuanyin/carbondata bug_unsafe_dm_store
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1633.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1633
----
commit 23030ee08de44af5da469678eb7bad009e7b3bd0
Author: xuchuanyin <xu...@hust.edu.cn>
Date: 2017-12-08T04:03:09Z
Fix bugs in unsafe datamap store
----
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1633
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2209/
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1633
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/652/
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1633
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/614/
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1633
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/696/
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1633
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1883/
---
[GitHub] carbondata pull request #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsaf...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1633
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1633
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1872/
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1633
retest this please
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1633
retest this please
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1633
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1842/
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1633
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/643/
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1633
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1925/
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/1633
retest this please
---
[GitHub] carbondata issue #1633: [CARBONDATA-1878] [DataMap] Fix bugs in unsafe datam...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1633
LGTM, Thank you for fixing the issue.
---