You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tedyu <gi...@git.apache.org> on 2016/03/24 21:39:41 UTC
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
GitHub user tedyu opened a pull request:
https://github.com/apache/spark/pull/11943
[SPARK-12181] Check Cached unaligned-access capability before using Unsafe
## What changes were proposed in this pull request?
For MemoryMode.OFF_HEAP, Unsafe.getInt etc. are used with no restriction.
However, the Oracle implementation uses these methods only if the class variable unaligned (commented as "Cached unaligned-access capability") is true, which seems to be calculated whether the architecture is i386, x86, amd64, or x86_64.
I think we should perform similar check for the use of Unsafe.
## How was this patch tested?
Unit test suite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tedyu/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11943.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11943
----
commit ed7064690de566270ec7dbc02870a3e08dd17f2d
Author: tedyu <yu...@gmail.com>
Date: 2016-03-24T20:37:37Z
[SPARK-12181] Check Cached unaligned-access capability before using Unsafe
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-202407877
@JnyJny
Did you have a chance to try out the change ?
Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11943
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by JnyJny <gi...@git.apache.org>.
Github user JnyJny commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-202458241
I have tested this change on a host which does not support misaligned data access with a conf/spark-defaults.conf configured to request off heap memory. The host was correctly identified as not supporting misaligned data access and the
```
$ ./bin/run-example SparkPi
2016-03-28 08:31:50,558 INFO [main] spark.SparkContext (Logging.scala:logInfo(54)) - Running Spark version 2.0.0-SNAPSHOT
2016-03-28 08:31:50,779 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-03-28 08:31:50,897 INFO [main] spark.SecurityManager (Logging.scala:logInfo(54)) - Changing view acls to: eoshaugh
2016-03-28 08:31:50,898 INFO [main] spark.SecurityManager (Logging.scala:logInfo(54)) - Changing modify acls to: eoshaugh
2016-03-28 08:31:50,899 INFO [main] spark.SecurityManager (Logging.scala:logInfo(54)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(eoshaugh); users with modify permissions: Set(eoshaugh)
2016-03-28 08:31:51,380 INFO [main] util.Utils (Logging.scala:logInfo(54)) - Successfully started service 'sparkDriver' on port 50828.
2016-03-28 08:31:51,396 INFO [main] spark.SparkEnv (Logging.scala:logInfo(54)) - Registering MapOutputTracker
2016-03-28 08:31:51,413 ERROR [main] spark.SparkContext (Logging.scala:logError(91)) - Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: No support for unaligned Unsafe. Set spark.memory.offHeap.enabled to false.
```
Removing conf/spark-defaults.conf and running on the same host with no other requested configuration changes resulted in the SparkPI demo to finish successfully.
I am concerned about SQL's unfettered use of Unsafe, but this fix is still valuable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203127866
LGTM pending tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201013695
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201013678
**[Test build #54082 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54082/consoleFull)** for PR 11943 at commit [`ed70646`](https://github.com/apache/spark/commit/ed7064690de566270ec7dbc02870a3e08dd17f2d).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201035925
That's right, Ryan.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203124345
> Adding a space would make the line longer than 100 chars.
> I suggest keeping the current form.
You can just break it to two lines like this:
```
Class<?> bitsClass =
Class.forName("java.nio.Bits", false, ClassLoader.getSystemClassLoader());
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201073235
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54087/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201039160
Just realize SQL always uses unsafe. However, I think it's still worth to have this method.
ping @davies @JoshRosen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203053830
Adding a space would make the line longer than 100 chars.
I suggest keeping the current form.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203162628
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201073233
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203162629
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54464/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11943#discussion_r57385602
--- Diff: core/src/main/scala/org/apache/spark/memory/MemoryManager.scala ---
@@ -176,6 +177,7 @@ private[spark] abstract class MemoryManager(
if (conf.getBoolean("spark.memory.offHeap.enabled", false)) {
require(conf.getSizeAsBytes("spark.memory.offHeap.size", 0) > 0,
"spark.memory.offHeap.size must be > 0 when spark.memory.offHeap.enabled == true")
+ require(Platform.unaligned(), "No support for unaligned Unsafe")
--- End diff --
Could you add a better instruction for the user? E.g., tell them to set spark.memory.offHeap.enabled to false.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by JnyJny <gi...@git.apache.org>.
Github user JnyJny commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201022926
Just "found" it, will pull the fix and re-start. Thanks :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201020821
@JnyJny
Compilation error is fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201011646
**[Test build #54082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54082/consoleFull)** for PR 11943 at commit [`ed70646`](https://github.com/apache/spark/commit/ed7064690de566270ec7dbc02870a3e08dd17f2d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201010303
@zsxwing
Created this updated PR according to your suggestion on #10181.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201073044
**[Test build #54087 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54087/consoleFull)** for PR 11943 at commit [`8bc7374`](https://github.com/apache/spark/commit/8bc737444df124a87bd705481886f00ab1cfe3a2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11943#discussion_r57385229
--- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java ---
@@ -37,6 +38,32 @@
public static final int DOUBLE_ARRAY_OFFSET;
+ private static final boolean unaligned;
+ static {
+ boolean _unaligned;
+ // use reflection to access unaligned field
+ try {
+ Class<?> bitsClass = Class.forName("java.nio.Bits", false,ClassLoader.getSystemClassLoader());
--- End diff --
nit: please add space after `,`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by JnyJny <gi...@git.apache.org>.
Github user JnyJny commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201018261
@tedyu I am building this pull request on SPARCv9 to test, should have an initial result in an hour or so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203127079
**[Test build #54464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54464/consoleFull)** for PR 11943 at commit [`9707e0a`](https://github.com/apache/spark/commit/9707e0a23c1abdc658b2bffb3562624db0e40f66).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203162210
**[Test build #54464 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54464/consoleFull)** for PR 11943 at commit [`9707e0a`](https://github.com/apache/spark/commit/9707e0a23c1abdc658b2bffb3562624db0e40f66).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201013701
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54082/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203163519
Merging to master. Thanks, @tedyu
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201015873
@tedyu could you add where the codes come from to the PR description?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201033556
@tedyu Just to confirm that I understand this issue. For platforms that don't support `unaligned memory access`, only `Unsafe.getByte` is safe to use since accessing unaligned memory address is undefined in these platforms.
In future, to support these platforms, we should fallback to `java.nio.Bits` (It's not public, we may need to copy these codes) like `DirectByteBuffer`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203020773
@zsxwing
Is there anything that needs to be done for this PR ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203050335
@tedyu could you address the small nit? Otherwise LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by JnyJny <gi...@git.apache.org>.
Github user JnyJny commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201048847
I have successfully built the pull request, however I won't be able to test it until tomorrow ( 25 Mar ).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-202466140
@davies @JoshRosen
Do you have any comment ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201019698
> @tedyu I am building this pull request on SPARCv9 to test, should have an initial result in an hour or so.
It has a compiler error now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-203032958
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by JnyJny <gi...@git.apache.org>.
Github user JnyJny commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-202439655
@tedyu Sorry, I haven't had the opportunity to test the change. I am hoping to get that done today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11943#discussion_r57385114
--- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java ---
@@ -37,6 +38,32 @@
public static final int DOUBLE_ARRAY_OFFSET;
+ private static final boolean unaligned;
+ static {
+ boolean _unaligned;
+ // use reflection to access unaligned field
+ try {
+ Class<?> bitsClass = Class.forName("java.nio.Bits", false,ClassLoader.getSystemClassLoader());
+ Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
+ unalignedMethod.setAccessible(true);
+ _unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
+ } catch (Throwable t) {
+ // We at least know x86 and x64 support unaligned access.
+ String arch = SystemPropertyUtil.get("os.arch", "");
--- End diff --
SystemPropertyUtil is a netty class. It's fine to just use the Java `System.getProperty`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12181] Check Cached unaligned-access ca...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11943#issuecomment-201021314
**[Test build #54087 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54087/consoleFull)** for PR 11943 at commit [`8bc7374`](https://github.com/apache/spark/commit/8bc737444df124a87bd705481886f00ab1cfe3a2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org